ITADN

[Bug]: Gemma4 multimodal serving — two startup crashes (vision tp_size>1, xgrammar guided decoding)

#15613OpenThachnh 创建于 2026-06-25
PytorchMultimodal
T
Thachnhcommented
## System Info - GPU: NVIDIA B200 (observed serving Gemma4 multimodal); the failures are config-triggered and not GPU-specific. - TensorRT-LLM: `main` @ `9ed7ce468b` (also present on v1.3.0rc16 / rc18). - Backend: PyTorch. - Model: Gemma4 multimodal — `Gemma4ForConditionalGeneration` (text + vision). ## Information - [x] My own modified scripts (serving via `trtllm-serve`) ## Reproduction Two independent startup crashes when serving Gemma4 multimodal on the PyTorch backend. Each is isolated to Gemma4 and triggered by a specific config. **(1) Vision tower crashes at `tp_size > 1`.** Serve the model with `--tp_size 2`: ``` AssertionError: Unexpected qkv shape (1152, 1152) ... (expected first dim = 8*72) ``` Root cause: `Gemma4VisionModel._pad_attention_head_dim` zero-pads the **full, unsharded** vision q/k/v weights (rounding `head_dim` 72 → 80 for FMHA) before the tensor-parallel split, but sizes the reshape from `first_attn.num_heads` — which `Attention.__init__` has already divided by `tp_size`. At `tp_size=2` the expected row count is half the weight's actual rows, so the shape assert fires. Works at `tp_size=1` only because `16 // 1 == 16`. **(2) xgrammar guided decoding crashes at startup.** Serve with `guided_decoding_backend=xgrammar`: ``` 'Gemma4ForConditionalGeneration' object has no attribute 'vocab_size_padded' ``` Root cause: `py_executor_creator` reads `model.vocab_size_padded` to build the xgrammar token bitmask. Text-only models inherit this property from `DecoderModelForCausalLM`, but the multimodal wrapper holds the language model as `self.llm` and does not expose it (other multimodal wrappers such as `modeling_gemma3vl` delegate it). ## Expected behavior Gemma4 multimodal serves at `tp_size > 1` and with `guided_decoding_backend=xgrammar` without crashing at startup. Fix proposed in PR #15566.
1 条评论