Enhancements

Thanks for awesome content. I am enjoying it so far. I was wondering if you could help me understand the following In the Overcoming the huge batch size section, why is large training size discouraged or not desirable? My naive assumption was higher batch size faster convergence and less training time. > If you use, say, 1024 accelerators, you’ll have tiny shards per accelerator and a ton of free memory for micro-batch-size (MBS), let’s say you can fit MBS=32 - you end up with GBS=32k - most likely not what you want. So you either need to deploy Tensor Parallelism which is non-trivial to implement, or often it’s much simpler to deploy Sequence Parallelism. Also, can we not use gradient accumulation to control the batch sizes as well?