[Bug][ROCm/gfx942]: DeepSeek-V4-Flash silent retrieval corruption for prompts ≥ ~4-5k tokens (AITER sparse indexer)
rocm
## Environment
- 8× AMD Instinct MI325X (gfx942), ROCm 7.14.0, amdgpu 6.19.14
- Image: `vllm/vllm-openai-rocm:nightly` (2026-08-12, v0.26.1rc1.dev668+g3ee2df303)
- plus local backports of #51821 (merged 08-13) and #52058 / #51252 (open) — reproduces identically without the two open-PR backports
- Model: `deepseek-ai/DeepSeek-V4-Flash-0731`, TP=8, `--kv-cache-dtype fp8_ds_mla`,
`--max-num-batched-tokens 16384`, attention backend `DEEPSEEK_SPARSE_SWA`
## 🐛 Describe the bug
Needle-in-haystack retrieval works perfectly for short prompts and collapses to 0/3
somewhere between **3,611 tokens (3/3 retrieved)** and **5,294 tokens (0/3)**. Above the
threshold the model claims the needles do not exist and output quality degrades broadly
(often rambling to `max_tokens`). No crash, no error — **healthy server, silently wrong
results**. GSM8K-style short-prompt correctness is unaffected (facts probes pass).
Measured (3 needles at 10%/50%/90% depth, `temperature=0`):
| prompt tokens | retrieved |
|---|---|
| 1,913 | 3/3 |
| 3,611 | 3/3 |
| 5,294 | 0/3 |
| 7,003 / 10,411 / 12,061 / 27k / 84k / 253k / 506k | 0/3 (10,411 once gave 2/3 with a corrupted digit string) |
## Ruled out experimentally
- `max_model_len` (fails identically at 131,072 and 1,048,576)
- chunked-prefill boundary (`max_num_batched_tokens=16384`; 12k prompts are single-chunk and still fail)
- #51252 (sparse-indexer prefill buffer budget) — backported, no change on this path
- AITER `fp8_mqa_logits` split-KV heuristic (`seq_len_kv >= 4096` → splits>1):
forced `num_splits=1`, no change
Looks like the gfx942 sibling of #40018 (gfx950 `ROCM_AITER_MLA_SPARSE` garbage for
prompt_len > ~20K), with a lower threshold (~4-5k).
## Repro
Self-contained script (generates haystack, inserts 3 needles, checks retrieval):
expand below. Run against an OpenAI-compatible endpoint:
`python3 needle_test.py <api-key> 6000` → RETRIEVED 0/3; `... 4000` → 3/3.
<details><summary>needle_test.py</summary>
```python
#!/usr/bin/env python3
"""長上下文 needle 測試:產生約 targetTok 的 haystack,插 3 根 needle(10%/50%/90% 深度),要求模型取回。"""
import json, time, random, urllib.request, sys
KEY = sys.argv[1]
TARGET_TOK = int(sys.argv[2]) if len(sys.argv) > 2 else 600_000
MODEL = sys.argv[3] if len(sys.argv) > 3 else "deepseek-v4-flash"
random.seed(42)
subjects = ["The quarterly report", "A municipal committee", "The research station", "An old lighthouse",
"The logistics team", "A regional archive", "The observatory", "A harbor authority",
"The botanical survey", "An engineering guild"]
verbs = ["documented", "reviewed", "misplaced", "catalogued", "audited", "transferred",
"digitized", "inspected", "renovated", "commissioned"]
objs = ["seventeen ledgers", "a set of brass instruments", "the annual rainfall data",
"three shipping manifests", "the訪客紀錄", "a collection of maps", "the maintenance schedule",
"several personnel files", "the calibration records", "an inventory of spare parts"]
years = list(range(1951, 2026))
NEEDLES = {
"ZEPHYR-CODE": "738291",
"MARLIN-KEY": "460517",
"ONYX-TOKEN": "092384",
}
def sentence():
return f"{random.choice(subjects)} {random.choice(verbs)} {random.choice(objs)} in {random.choice(years)}. "
# 估 1 token ≈ 4 chars(英文混雜),目標字元數:
target_chars = TARGET_TOK * 4
parts, n = [], 0
while n < target_chars:
s = sentence()
parts.append(s)
n += len(s)
keys = list(NEEDLES.items())
for frac, (k, v) in zip((0.10, 0.50, 0.90), keys):
idx = int(len(parts) * frac)
parts.insert(idx, f"[IMPORTANT] The secret value of {k} is {v}. Remember it. ")
haystack = "".join(parts)
prompt = (haystack
+ "\n\n---\nFrom the document above, report the secret values of ZEPHYR-CODE, MARLIN-KEY, and ONYX-TOKEN. "
"Answer in exactly this format:\nZEPHYR-CODE=<value>\nMARLIN-KEY=<value>\nONYX-TOKEN=<value>")
print(f"haystack chars: {len(haystack):,}", flush=True)
body = {"model": MODEL,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 4000, "temperature": 0}
req = urllib.request.Request("http://127.0.0.1:8000/v1/chat/completions",
json.dumps(body).encode(),
{"Authorization": f"Bearer {KEY}", "content-type": "application/json"})
t0 = time.time()
with urllib.request.urlopen(req, timeout=3600) as r:
d = json.load(r)
dt = time.time() - t0
u = d["usage"]
content = d["choices"][0]["message"].get("content") or ""
print(f"prompt_tokens={u['prompt_tokens']:,} completion_tokens={u['completion_tokens']} time={dt:.0f}s", flush=True)
print("--- model answer ---")
print(content[-500:])
ok = sum(1 for k, v in NEEDLES.items() if v in content)
print(f"RETRIEVED {ok}/3")
```
</details>
2 条评论