[Bug]: online FP8 (--quantization fp8) produces corrupted, non-EOS-terminating output on Qwen2.5-1.5B-Instruct

### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04.3 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 Clang version : Could not collect CMake version : version 3.28.3 Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.10.0+cu128 Is debug build : False CUDA used to build PyTorch : 12.8 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] (64-bit runtime) Python platform : Linux-6.8.0-79-generic-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : Could not collect GPU models and configuration : GPU 0: NVIDIA H200 Nvidia driver version : 570.124.06 HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.6 [pip3] numpy==2.1.2 [pip3] torch==2.10.0 [pip3] transformers==5.14.1 [pip3] triton==3.6.0 [conda] Could not collect ============================== vLLM Info ============================== vLLM Version : 0.19.1 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled ``` (Full output, including the complete `pip freeze` and NIC topology dump, omitted here for length -- available on request. GPU/CUDA/vLLM-relevant fields above are the complete, unedited values from the script.) </details> ### 🐛 Describe the bug `vllm serve <dense bf16 checkpoint> --quantization fp8` (online/on-the-fly W8A8, checkpoint not pre-quantized) produces incoherent, mixed-script output from the *first* generated token and essentially never reaches EOS -- not degraded-but-readable text, and not the kind of subtle numerical drift normal quantization noise causes. **Repro:** ```bash vllm serve Qwen/Qwen2.5-1.5B-Instruct --quantization fp8 ``` ```bash curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model": "Qwen/Qwen2.5-1.5B-Instruct", "messages": [ {"role": "system", "content": "You are a warm, emotionally present conversational companion in a live voice call. Reply in one or two short spoken sentences. Never use lists or markdown."}, {"role": "user", "content": "I have been thinking about something all week and I cannot shake it."} ], "max_tokens": 80, "temperature": 0.7, "stream": false, "seed": 0 }' ``` **Expected** (and what the identical request against the same checkpoint with no `--quantization` flag returns): ``` "content": "It's okay to spend time on thoughts that matter to you. What is the thought or idea you've been pondering? Sometimes talking through what's on your mind can help clear things up." "finish_reason": "stop" ``` **Actual**, `--quantization fp8`, same prompt/seed, re-confirmed live immediately before filing this issue: ``` "content": "袅 解.Resolve\";} yabǃ qualidade感じるolving (...ñas andaLab yab standardized zweolving � lastnameolics\tresolvefeatures ectñas learlab尼亚resolve yab.resolveresolve clock benchmarks anda ǃ yab！ijase第八ourcem原文地址olv巬 elimination eliminatedany세resolution qi!\n\n\n /></.setPosition等活动olvLab_gradientsledジョited巬.setPositionoler Labs(arguments moleLETEiteDOUBLE eliminated yö anda!(\"⭕\tresolve.setPosition ạ yabǃ" "finish_reason": "length" ``` 100% hitting `max_tokens` across every sample tested (n=76 in a live-harness check, n=160 in a separate calibration checkpoint at the same `max_tokens=80`) -- 0 natural `stop` in either sample, against the unquantized checkpoint's ~99.5% natural-stop rate under otherwise identical config. The unquantized checkpoint's mean response length across the same workload is ~24 tokens; the FP8 arm's is ~78 (every sample landing at or within a couple tokens of the `max_tokens` cap, not a wider but still-bounded distribution). Garbled from the first token in every sample checked, including at `max_tokens=8`, ruling out late-sequence drift. **Resolved config** (read from the live `Fp8Config` object, not inferred from the flag name): `activation_scheme: dynamic`, `is_checkpoint_fp8_serialized: False` -- `Fp8OnlineLinearMethod` in 0.19.1, W8A8 with dynamic per-forward-pass activation scaling. **Ruled out before concluding this is a vLLM-side bug, not a config/harness issue:** - Chat template: both the unquantized and `--quantization fp8` runs load the same checkpoint, so tokenizer/template are the same files by construction -- confirmed identical startup log line (`Detected the chat template content format to be 'string'`), not just assumed. - EOS / sampling config resolution: both startup logs show the identical `generation_config.json`-derived defaults (`repetition_penalty=1.1, temperature=0.7, top_k=20, top_p=0.8`), same source file, same values. - A static-activation-scale override via `--hf-overrides` was tried as a quick workaround and did **not** fix it (still garbled, still 100% `length`) -- not a one-flag fix. - Harness-level cause: the identical harness, against the identical checkpoint, with no `--quantization` flag, produces clean output at the same load level, same code path, same request shape. **What I have *not* been able to test:** whether this reproduces on a newer vLLM release. The GPU driver on the only hardware I have access to (`570.124.06`) caps out at CUDA 12.8, and every vLLM release after `0.19.1` resolves a torch build that requires CUDA 13 by default, so I can't install a newer vLLM on this machine to check directly. From reading the current `main` branch source (not tested): `Fp8OnlineLinearMethod` itself was removed in #45463, but the on-the-fly/unserialized-checkpoint path still exists today under `Fp8PerTensorOnlineLinearMethod` (`fp8.py`, `get_quant_method`, `not self.is_checkpoint_fp8_serialized` branch) -- so this doesn't look like something that was resolved by that removal, just renamed. #41022 (open) describes a plausible, related root cause -- `ops.scaled_fp8_quant` applied to an entire fused `MergedColumnParallelLinear` weight (gate_up_proj in a SwiGLU MLP, which Qwen2.5's architecture is) producing one shared scale across shards that should each have their own -- and explicitly lists "SwiGLU models (Llama, Mistral, Qwen, Gemma, etc.)" as affected, though its own repro is a different model (`Qwen3.5-35B-A3B`, MoE) and its documented symptom (precision loss, "!!!!!!!!!!!" on GDN models) doesn't obviously match ours (mixed-script token soup on a small dense model, never reaching EOS). Flagging as possibly the same underlying mechanism, not confirmed as a duplicate -- filing separately since our repro is a plain dense 1.5B model, smallest/simplest case I could find of this shape, and the exact symptom differs enough that I didn't want to assume it's the same bug without someone who knows this code path confirming it. Happy to run any repro against a version-pinned build if pointed at one, or to share the full `collect_env.py` output / additional samples on request.