[Bug] DeepSeek-V4 + hierarchical cache: deterministic SWA KV position corruption (kv-canary TAIL_K_SWA write_position), downstream NaN sampling crash

### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Describe the bug Running DeepSeek-V4-Flash with `--enable-hierarchical-cache` on the v0.5.16 image, we hit a production crash where the sampling probability tensor contains NaN (`Assertion 'probability tensor contains either inf, nan or element < 0' failed`), killing all TP ranks. Load at crash time was light (1 running request, ~24K context), so it is not a long-context or concurrency issue. To investigate, we enabled `--kv-canary=log` and caught deterministic KV corruption in the **SWA component**: ``` kv_canary violation: launch_tag=TAIL_K_SWA fail_reason=write_position slot_idx=53504 position=512 stored_token=1299 expected_token=1299 stored_chain_hash=0xe11e13af98c23f7f expected_aux=0x0000000000002100 KV cache canary violation detected (kernel_kind=TAIL_K_SWA, slot_idx=53504, position=512) canary_kind: per_forward_tail_k_swa fail_reasons: write_position actual: token_id=1299 position=512 prev_hash=0xe11e13af98c23f7f expected: token_id=1299 position=8448 total_violations=516 ring_overflow=False step_when_pumped=6358 ``` Key observations: - All violations are in `TAIL_K_SWA` (`per_forward_tail_k_swa`); the FULL component is clean. - **This is not random bit corruption**: `token_id` and `prev_hash` match expectations exactly, only the **position** is wrong (written at position 512, expected 8448). It looks like KV written to the wrong SWA slot — a position/translation bookkeeping bug, not data damage. - All 8 TP ranks report the **identical** violation count (516) — deterministic and synchronized, ruling out hardware. - The violation burst coincided with a prefill that had a large prefix-cache hit (`#cached-token: 184832`), pointing at the cached-prefix restore path. - Downstream effect matches the earlier crash: corrupted SWA K → garbage logits → NaN in sampling probs → device-side assert on all ranks. Mitigation: after disabling `--enable-hierarchical-cache` (GPU-only radix cache, kv-canary still on) we have not observed further violations so far; will update after a longer observation window. This strongly resembles #25889 (stale full→SWA translation indices after HiCache rebuild causing wrong-position writes), but: - We do **not** set `SGLANG_OPT_CACHE_SWA_TRANSLATION` (default off), and - v0.5.16 should already contain that fix. So this looks like a **regression or sibling bug** in the newer UnifiedRadixTree-based hicache path for DSv4 (`Init Unified RadixTree with components (FULL, SWA)`, `build_deepseek_v4_hicache_stack`), possibly introduced around #30468 (UnifiedRadixTree as default cache for SWA models). ### Reproduction Single node, 8× H20 (96GB), TP=8. ```bash python3 -m sglang.launch_server \ --model-path /models/DeepSeek-V4-Flash-0731 \ --tp-size=8 \ --trust-remote-code \ --moe-runner-backend=flashinfer_mxfp4 \ --tool-call-parser=deepseekv4 \ --reasoning-parser=deepseek-v4 \ --mem-fraction-static=0.85 \ --chunked-prefill-size=4096 \ --context-length=524288 \ --enable-hierarchical-cache \ --hicache-ratio=1.5 \ --speculative-algorithm=DSPARK \ --speculative-num-steps=1 \ --speculative-num-draft-tokens=6 \ --max-running-requests=128 \ --cuda-graph-max-bs-decode=128 \ --kv-canary=log ``` Environment variable: `SGLANG_DSV4_FP4_EXPERTS=1` (mixed-quant checkpoint: fp8 attention/dense + MXFP4 routed experts). Workload: multi-turn agent/coding traffic with high prefix reuse (radix hit rate ~86%). Violations appear after the GPU radix cache starts evicting to the host tier and a later request restores a large cached prefix. Under this workload it reproduces within hours. NaN crash stack (async report, actual fault is upstream of this line): ``` File ".../speculative/dspark_components/dspark_worker_v2.py", line 406, in _forward_prefill ctx_lens = torch.tensor(batch.extend_lens, dtype=torch.int32, device=device) torch.AcceleratorError: CUDA error: device-side assert triggered /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:109: Assertion `probability tensor contains either `inf`, `nan` or element < 0` failed. ``` ### Environment - SGLang: official Docker image `lmsysorg/sglang:v0.5.16-cu130` - GPU: 8× NVIDIA H20 (96GB), single node, pure TP (no DP attention) - Model: DeepSeek-V4-Flash-0731 (fp8 + MXFP4 experts mixed quantization) - CUDA: 13.0.1 (from image), Python 3.12 - Host: 1TB RAM; hicache host pool ≈ 550GB at ratio 1.5