ITADN

ncclAllReduce may require additional barriers

#1587Opensuranap 创建于 2025-01-27
bug
S
suranapcommented
https://github.com/flexflow/flexflow-train/blob/de7fa32511e77fc34cb9a9f09df12dba826a84ab/lib/kernels/src/cuda/optimizer_kernel.cu#L83 A thread on Zulip mentioned some additional care required for NCCL within a Legion task. Rohan [spotted a problem](https://legion.zulipchat.com/#narrow/channel/187787-general/topic/stalled.20dmas.20on.20slingshot.20network/near/496212104) in FlexFlow's use of `ncclAllReduce`. You may need to add `concurrent_task_barrier` before and after the call, and call `set_concurrent_barrier` on the task. More info is in the comment for that barrier.
0 条评论