[Bug]: FP8 on RDNA3/gfx1100 exceeds the 600 s engine-ready timeout on first start — no AMD_Radeon_Graphics tuned configs shipped

### TL;DR - The official FP8 checkpoint (Qwen/Qwen3.8-27B-FP8) **loads and serves fine on a consumer RDNA3 card** (gfx1100, Radeon W7900 48 GB, `vllm==0.27.1+rocm723` on ROCm). There is no architecture refusal — community trackers currently report FP8 as "does not run on RDNA3", but it does. - The out-of-the-box failure is a **startup timeout**: for `device_name=AMD_Radeon_Graphics` the wheel ships **zero** tuned W8A8 block-FP8 configs (of 218 device-keyed configs it ships 76 for AMD Instinct and 142 for NVIDIA devices), so the default config is used and warmup **JIT-compiles the Triton GEMM kernels shape by shape**. First engine init measured **1474.26 s** — 2.5x the 600 s default of `VLLM_ENGINE_READY_TIMEOUT_S` — and the APIServer aborts. - The abort message says the timeout is "**often caused by slow weight loading**". Our weights loaded in **16.33 s**; the 23+ minutes between kernel warnings were CPU-side Triton compilation with the GPU idle. - `VLLM_ENGINE_READY_TIMEOUT_S=1800` makes it start. Once the on-disk Triton cache is warm, engine init is **49 s** and every later start is ~2 min — the cost is one-time per host (until `~/.triton/cache` is cleared). - **Asks:** (1) ship tuned W8A8 configs for `AMD_Radeon_Graphics` (mirroring the MI300X/MI325 sets); (2) cheap: mention cold-start JIT/config warmup as a cause in the engine-ready timeout error or the env-var docs; (3) secondary observation: the **first inference** triggers a further ~43 min of JIT staging that `/health` does not reflect. - **I have the hardware** and am happy to run whatever tuning/profiling procedure you point at and PR the resulting `AMD_Radeon_Graphics` JSONs, and to test any RC on gfx1100. ### Your current environment <details> <summary>Output of <code>python -m vllm.collect_env</code> (venv used for all runs below)</summary> ```text Collecting environment information... uv is set ============================== System Info ============================== OS : Ubuntu 24.04.4 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version : Could not collect CMake version : version 3.28.3 Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.11.0+gitd0c8b1f Is debug build : False CUDA used to build PyTorch : N/A ROCM used to build PyTorch : 7.2.53211 XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.3 (main, Mar 3 2026, 12:15:18) [GCC 13.3.0] (64-bit runtime) Python platform : Linux-6.8.0-79-generic-x86_64-with-glibc-2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : Could not collect CUDA_MODULE_LOADING set to : GPU models and configuration : AMD Radeon Graphics (gfx1100) Nvidia driver version : Could not collect cuDNN version : Could not collect HIP runtime version : 7.2.53211 MIOpen runtime version : 3.5.1 Is XNNPACK available : True ============================== vLLM Info ============================== ROCM Version : 7.2.53211-e1a6bc5663 vLLM Version : 0.27.1 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled GPU Topology: single GPU, NUMA node 0 ============================== Environment Variables ============================== PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root ``` Install (per the ROCm install docs): `uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/` → `vllm==0.27.1+rocm723`; `transformers==5.15.0`; host ROCm userspace 7.2.1. CPU: 2x AMD EPYC 9334 (full `lscpu` flags and the rocm-smi topology table omitted for brevity; available on request). </details> ### 🐛 Describe the bug **Expected:** `vllm serve` (or `api_server`) with an official FP8 checkpoint on gfx1100 either starts within the default engine-ready budget, or fails with a message pointing at the real cost. **Actual (measured on Radeon W7900D, gfx1100, 48 GB):** | Run | `VLLM_ENGINE_READY_TIMEOUT_S` | Triton cache | Result | |---|---|---|---| | cold serve | 600 (default) | cold | `TimeoutError` at 600 s mid-warmup; weights had loaded in 16.33 s | | cold serve | 1800 | cold | engine init **1474.26 s**, then `/health` 200, serving OK | | warm re-run | 1800 | warm | engine init **49.01 s** | | later starts (x3) | 1800 | warm | 49.10 / 49.11 / 49.19 s, server up in ~2 min | After unlock, serving works end-to-end (105/105 benchmark requests OK) — at untuned-config performance; see measured results below. #### Reproduce without hardware (2 minutes) Anyone with the wheel installed can verify the config gap that drives the cold-start cost: ```bash CFG=$(python -c "import vllm.model_executor.layers.quantization.utils.fp8_utils as m, os; print(os.path.join(os.path.dirname(m.__file__),'configs'))") ls "$CFG" | wc -l # 219 (218 device-keyed JSONs + README.md) ls "$CFG" | grep -c MI300X # 28 (+ 24 MI325X + 24 MI325_OAM = 76 Instinct) ls "$CFG" | grep -ci radeon # 0 ls "$CFG" | grep -c 'N=16384,K=5120' # 0 — not shipped for ANY device ``` The engine-side lookup is a plain file-exists check in `vllm/model_executor/layers/quantization/utils/fp8_utils.py` (0.27.1, ~line 830): the filename is built from `get_device_name_as_file_name()` — which on RDNA3 consumer cards is literally the torch device name `AMD Radeon Graphics` → `device_name=AMD_Radeon_Graphics` — and when the JSON is missing the code logs the warning below and **returns `None` (default config)**. Nothing is ever written back to that directory, so the cost recurs for every fresh host/venv. #### Reproduce on gfx1100 (~35 min, needs a 24 GB+ RDNA3 card) ```bash # fresh venv — a warm ~/.triton/cache hides the bug (init drops to ~49 s) uv venv /root/venv-fp8 && VIRTUAL_ENV=/root/venv-fp8 uv pip install vllm \ --extra-index-url https://wheels.vllm.ai/rocm/ # 0.27.1+rocm723 # model: official FP8 checkpoint, ~28.7 GiB # HF: huggingface-cli download Qwen/Qwen3.8-27B-FP8 # (we used ModelScope: modelscope download Qwen/Qwen3.8-27B-FP8) SNAP=/path/to/Qwen3.8-27B-FP8 # 1) default timeout → aborts at 600 s: /root/venv-fp8/bin/python -m vllm.entrypoints.openai.api_server \ --model "$SNAP" --served-model-name Qwen/Qwen3.8-27B-FP8 \ --max-model-len 8192 --gpu-memory-utilization 0.90 --port 8199 # 2) unlocked: VLLM_ENGINE_READY_TIMEOUT_S=1800 /root/venv-fp8/bin/python -m vllm.entrypoints.openai.api_server \ --model "$SNAP" --served-model-name Qwen/Qwen3.8-27B-FP8 \ --max-model-len 8192 --gpu-memory-utilization 0.90 --port 8199 ``` (All measurements above were taken with `--enforce-eager` added to skip torch.compile/CUDA-graph capture; the cold abort also reproduces without it, with `torch.compile took 67.53 s` interleaved.) #### Verbatim evidence <details> <summary>1) Cold run, default timeout — aborts while weights were long done</summary> ```text (EngineCore pid=280186) INFO 08-17 01:42:46 [default_loader.py:430] Loading weights took 16.33 seconds (EngineCore pid=280186) INFO 08-17 01:42:46 [gpu_model_runner.py:5405] Model loading took 29.38 GiB memory and 17.353099 seconds (EngineCore pid=280186) INFO 08-17 01:44:23 [monitor.py:53] torch.compile took 67.53 s in total (EngineCore pid=280186) WARNING 08-17 01:44:23 [fp8_utils.py:851] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /root/venv-fp8probe/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/utils/configs/N=16384,K=5120,device_name=AMD_Radeon_Graphics,dtype=fp8_w8a8,block_shape=[128,128].json (EngineCore pid=280186) WARNING 08-17 02:07:49 [fp8_utils.py:851] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /root/venv-fp8probe/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/utils/configs/N=5120,K=6144,device_name=AMD_Radeon_Graphics,dtype=fp8_w8a8,block_shape=[128,128].json (APIServer pid=279547) TimeoutError: Timed out waiting for engine core processes to start. This is often caused by slow weight loading for large models. Waited 600s (configured by VLLM_ENGINE_READY_TIMEOUT_S). To increase the timeout, set the environment variable: VLLM_ENGINE_READY_TIMEOUT_S=<seconds> ``` Note the **23 m 26 s gap between the two fp8_utils warnings** — that is one Triton GEMM kernel compiling (single CPU core pinned, GPU idle), not weight loading. </details> <details> <summary>2) Same wheel, timeout raised — starts, and the cost is visible</summary> ```text (EngineCore pid=371418) WARNING 08-17 08:07:37 [fp8_utils.py:851] Using default W8A8 Block FP8 kernel config. ... N=16384,K=5120,device_name=AMD_Radeon_Graphics,... (EngineCore pid=371418) WARNING 08-17 08:31:17 [fp8_utils.py:851] Using default W8A8 Block FP8 kernel config. ... N=34816,K=5120,device_name=AMD_Radeon_Graphics,... (EngineCore pid=371418) INFO 08-17 08:31:42 [core.py:355] init engine (profile, create kv cache, warmup model) took 1474.26 s (APIServer pid=371094) INFO: 127.0.0.1:33878 - "GET /health HTTP/1.1" 200 OK ``` The warmup pass walks five GEMM shapes for this model (N=16384/K=5120, N=5120/K=6144, N=34816/K=5120, N=5120/K=17408, N=14336/K=5120); none has a shipped config. </details> <details> <summary>3) Warm Triton cache — the whole cliff disappears</summary> ```text (EngineCore pid=378832) INFO 08-17 08:38:28 [core.py:355] init engine (profile, create kv cache, warmup model) took 49.01 s (APIServer pid=378511) INFO 08-17 08:38:45 [api_server.py:682] Starting vLLM server on http://0.0.0.0:8199 (APIServer pid=378511) INFO: Application startup complete. (APIServer pid=378511) INFO: 127.0.0.1:46230 - "GET /health HTTP/1.1" 200 OK ``` </details> <details> <summary>4) Secondary: first-INFERENCE JIT staging (~43 min), invisible to /health</summary> Even after a healthy start, the first completion request triggers more JIT compiles; the server answers `/health` 200 the whole time and a default-timeout client (ours: 300 s) just times out. Verbatim from the first served request on a cache-warm-start server: ```text (EngineCore pid=434612) WARNING 08-17 13:34:18 [chunked_prefill_paged_decode.py:419] Cannot use ROCm custom paged attention kernel, falling back to Triton implementation. (EngineCore pid=434612) WARNING 08-17 13:55:59 [jit_monitor.py:135] Triton kernel JIT compilation during inference: _causal_conv1d_update_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (EngineCore pid=434612) WARNING 08-17 13:55:59 [jit_monitor.py:135] Triton kernel JIT compilation during inference: fused_recurrent_gated_delta_rule_packed_decode_kernel. ... (EngineCore pid=434612) WARNING 08-17 13:55:59 [jit_monitor.py:135] Triton kernel JIT compilation during inference: layer_norm_fwd_kernel. ... ``` The very first request (13:34:18) outlived the client's 300 s ceiling; the first fully-served response landed at ~14:17:41 — **43 min after the first request was sent** (the request that completed was issued 14:07:52 and took `total_s=608.536757`). Subsequent traffic is normal: a 16-request warmup burst after this returned all 200 in ~26–28 s. </details> ### Root cause (as far as the code shows) 1. `fp8_utils.py` builds the tuned-config filename from the torch device name — on RDNA3 consumer cards that is `AMD Radeon Graphics` → `device_name=AMD_Radeon_Graphics` — and the wheel ships **no** JSONs for that name (the entire AMD set is Instinct-only). Missing file → warning → `return None` → default W8A8 config. 2. With the default config, `w8a8_triton_block_scaled_mm` JIT-compiles its Triton kernel for each distinct (N, K) shape during engine warmup. On this host each big compile takes ~20+ min of single-threaded CPU (GPU idle); the shape-by-shape walk (first warning 08:07:37 → init done 08:31:42) accounts for essentially all of the 1474.26 s engine init. 3. The APIServer's engine-ready deadline (default 600 s, `VLLM_ENGINE_READY_TIMEOUT_S`, consumed in the v1 engine core client) expires mid-warmup, and the error names "slow weight loading" as the likely cause. 4. The Triton disk cache (`~/.triton/cache`) makes every subsequent start fast, and nothing writes tuned JSONs back — so the cliff is a **per-host one-time cost that looks like a hard failure** the first time. ### Proposed fixes (ranked) 1. **Ship tuned W8A8 block-FP8 configs for `AMD_Radeon_Graphics`** (gfx1100 at minimum), mirroring the shipped MI300X/MI325 sets. This is what the runtime warning already asks for, it removes the "sub-optimal default" path entirely, and it is the lever for the performance gap below. I have a W7900 and am glad to run the config-generation/benchmark procedure on it and open a PR with the resulting JSONs if you can point me at the tooling. 2. **Cheap message/docs improvement:** `VLLM_ENGINE_READY_TIMEOUT_S` is documented, but the timeout error's cause list ("often caused by slow weight loading") sends users down the wrong path when the real cost is JIT/config warmup on an untuned device (weights: 16.33 s here). One extra sentence in the error string or in the env-var docs — "first-start JIT compilation of untuned kernels can take tens of minutes on newly-supported platforms; raise the timeout or pre-warm" — would have saved us hours. 3. **Secondary observation** (same platform gap, different surface): the first inference after a cold start triggers ~43 min more of Triton JIT that `/health` does not reflect (details block 4). If preferred I'll split this into its own issue — keeping it here because it shares the root cause (no RDNA3 tuning artifacts shipped). ### Measured results after unlock (context for prioritization, not a complaint) `--enforce-eager`, temp 0, fixed 100-token prompt, 128-token generations, 5 reps per cell, non-streaming: | concurrency | aggregate decode tok/s | requests OK | p50 request wall | |---|---:|---|---:| | 1 | 0.612 | 5/5 | 209.2 s | | 4 | 2.301 | 20/20 | 222.4 s | | 16 | 8.730 | 80/80 | 234.5 s | For scale on the same GPU, llama.cpp with a Q4_K_M GGUF of the same model does 28.6 tok/s single-stream and 164.4 tok/s aggregate at c=32 — so FP8-on-RDNA3 today is ~47x slower per stream, which the tuned-config gap plausibly explains a large part of. Capacity is not the issue: 29.38 GiB of FP8 weights on the 48 GB card with ~19 GiB headroom. ### Related issues (searched; not duplicates) - #51136 — AITER not enabled on RDNA3 (gfx1100): same GPU class, different mechanism (INT8/Quark + AITER gating vs FP8 W8A8 Triton path here); complementary data point. - #35547, #32116 — same engine-ready timeout symptom class on NVIDIA (slow weight loading / DeepGEMM warmup). This issue adds the RDNA3/FP8 instance where the dominant cost is per-shape Triton JIT under a missing tuned config, plus the configs gap itself. ### Before submitting a new issue... - [x] I searched existing issues (linked above) and found no report of FP8-on-RDNA3 failing startup via untuned-config JIT with this mechanism. - [x] `collect_env` output attached. - [x] Repro is self-contained; the configs-gap part needs no hardware. Happy to contribute: tuning runs on gfx1100 → `AMD_Radeon_Graphics` config PR, and RC testing on this card.