ITADN

wrong content in model-parallelism?

#122Opencyr0930 创建于 2025-09-11
C
cyr0930commented
In here (https://github.com/stas00/ml-engineering/tree/master/training/model-parallelism#parallelism-network-collectives), there's a sentence `ZeRO-DP ZeRO-1/ZeRO-2: 1x all-gather for optimizer states plus 1x reduce-scatter for gradients - total volume: 2x model params comms` but according to paper (https://arxiv.org/pdf/1910.02054) I think all-gather is used for updated parameters as same as in DP.
4 条评论