wrong content in model-parallelism?
In here (https://github.com/stas00/ml-engineering/tree/master/training/model-parallelism#parallelism-network-collectives), there's a sentence
`ZeRO-DP ZeRO-1/ZeRO-2: 1x all-gather for optimizer states plus 1x reduce-scatter for gradients - total volume: 2x model params comms`
but according to paper (https://arxiv.org/pdf/1910.02054) I think all-gather is used for updated parameters as same as in DP.
4 条评论