[Performance] PR #29223 regresses Llama 3.3 FP8 TP8 H100 throughput by 3.2%

[by Codex] ## Checklist - [x] I searched related issues but did not find an existing report for this regression. - [x] The regression appears after PR #29223 / merge commit `da802ddcafe55e25b3e1db86b1e0444afc3e05bc`. - [x] This report uses a public model, public tools, and public SGLang entry points only. ## Describe the bug PR #29223, `(perf): Shard Kimi-K2.5 Eagle3 draft fc + symm-mem AG`, causes an approximately 3.2% decode-throughput regression for `nvidia/Llama-3.3-70B-Instruct-FP8` on one 8x H100 node with TP8. The source boundary was reproduced and bisected repeatedly: the parent of merge commit `da802ddcafe55e25b3e1db86b1e0444afc3e05bc` remains at approximately 128.8 output tok/s, while that merge commit measures approximately 124.8 output tok/s. Because this workload does not use Kimi-K2.5 EAGLE3, the likely area to debug is the shared `LogitsProcessor` change in PR #29223. It replaces `tensor_model_parallel_all_gather(logits)` with `MultimemAllGatherer(..., skip_entry_sync=True)` for the normal tensor-parallel logits-gather path. This is only a suspected mechanism; the bisect proves the PR boundary, not the specific code path within the PR. cc @kpham-sgl ## Reference performance Metric: AIPerf `Output Token Throughput (tokens/sec)` for the completions endpoint, streaming enabled, concurrency 1, fixed ISL 1000, and fixed OSL 1000. | Workload | System / shape | Before #29223 | After #29223 | Drop | | --- | --- | ---: | ---: | ---: | | `nvidia/Llama-3.3-70B-Instruct-FP8` | 1x 8-GPU H100 node, TP8, ModelOpt FP8 | 128.856 tok/s | 124.677 tok/s | 3.24% | Additional fixed-source bisect samples were consistent with this boundary: the last good point measured 128.762-128.859 tok/s and the first bad point measured 124.795-124.930 tok/s. ## Reproduction Compare `da802ddcafe55e25b3e1db86b1e0444afc3e05bc^` against `da802ddcafe55e25b3e1db86b1e0444afc3e05bc` in otherwise identical environments. The measurements above used a CUDA 13.0-era stack, PyTorch 2.11.0-era build, FlashInfer 0.6.12, and Triton 3.6.0. ### 1. Launch the server ```bash export MODEL=nvidia/Llama-3.3-70B-Instruct-FP8 export SERVED_MODEL_NAME=llama3.3-70b export PORT=30000 export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 # Prevent prefix reuse while keeping the optimized radix-cache code path. export SGLANG_RADIX_FORCE_MISS=1 python3 -m sglang.launch_server \ --model-path "${MODEL}" \ --served-model-name "${SERVED_MODEL_NAME}" \ --host 0.0.0.0 \ --port "${PORT}" \ --tp 8 \ --quantization modelopt \ --context-length 2048 \ --piecewise-cuda-graph-max-tokens 2048 \ --mem-fraction-static 0.79 ``` Wait for `http://localhost:${PORT}/health` to become ready. ### 2. Run AIPerf ```bash export ISL=1000 export OSL=1000 export CONCURRENCY=1 export REQUEST_COUNT=5 aiperf profile \ --url "http://localhost:${PORT}" \ --model "${SERVED_MODEL_NAME}" \ --tokenizer "${MODEL}" \ --tokenizer-trust-remote-code \ --endpoint-type completions \ --streaming \ --ui simple \ --extra-inputs "min_tokens:${OSL}" \ --extra-inputs "max_tokens:${OSL}" \ --extra-inputs "ignore_eos:true" \ --concurrency "${CONCURRENCY}" \ --synthetic-input-tokens-mean "${ISL}" \ --synthetic-input-tokens-stddev 0 \ --output-tokens-mean "${OSL}" \ --output-tokens-stddev 0 \ --request-count "${REQUEST_COUNT}" \ --warmup-request-count 1 \ --artifact-dir "aiperf_llama33_fp8_isl${ISL}_osl${OSL}_con${CONCURRENCY}" ``` Compare AIPerf's `Output Token Throughput (tokens/sec)` between the two source revisions. ## Useful debug checks - Revert only the `python/sglang/srt/layers/logits_processor.py` hunk from #29223 and repeat the benchmark. This isolates the shared logits gather from the Kimi-specific FC changes. - Compare the new `MultimemAllGatherer` path with the previous `tensor_model_parallel_all_gather` path on TP8 H100 for a single-token decode workload. - Check the `skip_entry_sync=True` assumption and the lazy symmetric-memory buffer path for this non-speculative Llama workload. - Profile the logits gather latency and any synchronization gap before and after the merge commit. ## Environment - Hardware: 8x NVIDIA H100 in one node - Parallelism: TP8 - Model: `nvidia/Llama-3.3-70B-Instruct-FP8` - Quantization loader: ModelOpt FP8 - Workload: concurrency 1, ISL 1000, OSL 1000, five measured requests, one warmup request - Culprit PR: #29223 - First bad commit: `da802ddcafe55e25b3e1db86b1e0444afc3e05bc`