[Bug] Multi-node TP rank-divergence deadlock: one rank wedges in NCCL proxy append (logits all-gather), peer idles at request broadcast — DeepSeek-V4 + DSpark on 2× DGX Spark (GB10)

## Summary Running DeepSeek-V4-Flash-0731 with `--speculative-algorithm DSPARK` across **2 nodes (TP=2)**, the server intermittently deadlocks minutes to hours into normal OpenAI-compatible traffic. Captured live, the two scheduler ranks sit in a permanent split: - **Rank A (busy)**: spinning in the NCCL host enqueue path (`sched_yield` in `ncclLocalOpAppend` / `SaveProxy`) while posting an **all-gather of logits** — the proxy thread no longer consumes ops. - **Rank B (peer)**: idle, parked at `_broadcast_reqs_across_ranks` — it has exited the batch/step loop and will never join that collective. The unmatched collective wedges the NCCL proxy, the GPU stream never completes, and the scheduler watchdog eventually fires and kills the server. The failure shape matches the family described in #27221 (per-rank divergence in batch composition → mismatched collectives → NCCL deadlock), but with request timeouts **disabled** (both `SGLANG_REQ_WAITING_TIMEOUT` and `SGLANG_REQ_RUNNING_TIMEOUT` at default `-1`), so the divergence trigger here is a different, still-unidentified per-rank decision — all captured incidents have the DSpark verify path in the wedged rank's stack. ## Environment - **SGLang**: v0.5.16 (`lmsysorg/sglang:latest`, pulled 2026-08-02, digest `sha256:7b6a35df…`) - **Model**: `deepseek-ai/DeepSeek-V4-Flash-0731` (FP4 MoE + FP8 dense, bundled DSpark draft head) - **Hardware**: 2× NVIDIA DGX Spark (GB10, SM121, aarch64, 128 GB unified memory each), TP=2 across nodes - **Interconnect**: ConnectX-7 RoCEv2 between the nodes (12.9 GB/s measured NCCL all-reduce, zero error counters) - **NCCL**: 2.28.9 (same in the reference vLLM stack, which is stable on identical hardware) - **Key flags**: `--speculative-algorithm DSPARK --moe-runner-backend flashinfer_mxfp4 --kv-cache-dtype fp8_e4m3 --mem-fraction-static 0.85 --cuda-graph-backend-decode=disabled --watchdog-timeout 120` - Local patches (bind-mounted, needed for DSpark on SM120/121): topk-192→512 index padding in `flash_mla_sm120.py` (the SM120 sparse-MLA kernel lacks a topk=192 instantiation — see flashinfer#3828) and a referenced-page bitmap for the pbs=256→64 page-split copy. Both wedges below also reproduce **without** these patches. ## Observed behavior 1. Server runs fine for minutes–hours, then a request stops making progress mid-decode. 2. No decode batches in the log; no error; one rank's scheduler pegged, the other idle. 3. Watchdog fires (`Scheduler watchdog timeout`), `kill_process_tree`, containers exit(0). 4. Repeatable across **five independent incidents**, under decode CUDA graphs both **on and off**. ### Incident 1 — worker (TP1) wedged, head (TP0) idle TP1 py-spy (native, active): ``` sched_yield ncclLocalOpAppend (proxy.cc:499) SaveProxy (proxy.cc:580) ncclProxySaveOp (proxy.cc:597) uploadProxyOps (enqueue.cc:1293) ncclLaunchKernelAfter_NoCuda (enqueue.cc:1670) doLaunches / groupLaunch / ncclGroupEndInternal (group.cc) ncclAllGather (collectives.cc:91) c10d::ProcessGroupNCCL::_allgather_base all_gather_into_tensor (distributed_c10d.py:4186) tensor_model_parallel_all_gather (distributed/communication_op.py:47) __call__ (triton_symm_mem_ag.py:515) _get_logits (logits_processor.py:710) forward (logits_processor.py:450) forward (deepseek_v4.py:2530) _execute_extend (runner/eager_runner.py:332) ... _forward_prepared_verify (dspark_components/dspark_verify.py:271) _forward_decode (dspark_components/dspark_worker_v2.py:590) run_batch (scheduler.py:3351) ``` TP0 py-spy (idle): ``` broadcast (torch/distributed/distributed_c10d.py:2974) broadcast_pyobj (utils/common.py:2175) _broadcast_reqs_across_ranks (scheduler_components/request_receiver.py:198) recv_requests (scheduler_components/request_receiver.py:90) event_loop_overlap (scheduler.py:1570) ``` ### Incident 2 — mirrored: head (TP0) wedged, worker (TP1) idle Identical frames, roles swapped: TP0 active in `ncclLocalOpAppend` during `ncclAllGather` via `_get_logits` → `_execute_extend` → DSpark verify; TP1 idle at `_broadcast_reqs_across_ranks`. Earlier incidents (before decode graphs were disabled) showed the downstream projection of the same wedge: the scheduler parked in `process_batch_result_decode → torch.cuda.synchronize` waiting for a stream that never completes. ## What we ruled out experimentally - **NCCL fabric / config**: all-reduce microbenchmarks clean (12.9 GB/s, 14 µs at 2 KB, no stalls); zero RoCE error/retransmit counters on both nodes; per-node RoCEv2 GID auto-resolved correctly; env matched to a known-good vLLM stack on the same pair (`CROSS_NIC=1`, RoCEv2, no plugins). Same NCCL 2.28.9 in both engines. `NCCL_PROTO=Simple` and `NCCL_LAUNCH_MODE=PARALLEL` — wedge still recurs. - **CUDA graphs**: recurs with `--cuda-graph-backend-decode=disabled` (prefill graphs are auto-disabled for this model anyway). - **Attention backend / kernels**: recurs on `dsv4` (flashinfer SM120 sparse-MLA) and on the triton SM120 fallback. - **DSpark ragged/confidence scheduling**: `SGLANG_RAGGED_VERIFY_MODE` at default `static` (budget planner not created); `SGLANG_SIMULATE_ACC_LEN` unset. - **Request timeout aborts (#27221's trigger)**: both env timeouts at default `-1`. - **Client aborts**: 35 streaming requests cancelled mid-prefill and mid-decode in a loop — no wedge. Normal completed requests also pass for hours under synthetic load. - **Host memory**: recurs at `mem-fraction-static 0.85` with tens of GB free on both nodes. The remaining common factor in all live captures: **multi-node TP=2 + DSpark verify path** in the wedged rank's stack, and the peer rank having already moved on to the inter-step request broadcast — i.e., a one-step divergence in batch composition between the per-rank schedulers. ## Expected behavior Both scheduler ranks agree on batch composition every step; no unmatched collectives; no proxy wedge. ## Notes - A reference vLLM stack (same two machines, same checkpoint, DSpark) is stable under the same client traffic for days, consistent with vLLM's driver-broadcast step table not having independent per-rank scheduler loops. - Happy to provide full container logs (NCCL `PROXY,NET,INIT` INFO level from both nodes), the watchdog scheduler-debug dumps, and to test patches. If there is a debug flag that makes the two ranks log their batch composition per step (`cur_batch_for_debug`-style), a run with that enabled should catch the divergent step in the act.