[Bug]: Gemma4 multimodal serving — two startup crashes (vision tp_size>1, xgrammar guided decoding)
PytorchMultimodal
## System Info
- GPU: NVIDIA B200 (observed serving Gemma4 multimodal); the failures are config-triggered and not GPU-specific.
- TensorRT-LLM: `main` @ `9ed7ce468b` (also present on v1.3.0rc16 / rc18).
- Backend: PyTorch.
- Model: Gemma4 multimodal — `Gemma4ForConditionalGeneration` (text + vision).
## Information
- [x] My own modified scripts (serving via `trtllm-serve`)
## Reproduction
Two independent startup crashes when serving Gemma4 multimodal on the PyTorch backend. Each is isolated to Gemma4 and triggered by a specific config.
**(1) Vision tower crashes at `tp_size > 1`.** Serve the model with `--tp_size 2`:
```
AssertionError: Unexpected qkv shape (1152, 1152) ... (expected first dim = 8*72)
```
Root cause: `Gemma4VisionModel._pad_attention_head_dim` zero-pads the **full, unsharded** vision q/k/v weights (rounding `head_dim` 72 → 80 for FMHA) before the tensor-parallel split, but sizes the reshape from `first_attn.num_heads` — which `Attention.__init__` has already divided by `tp_size`. At `tp_size=2` the expected row count is half the weight's actual rows, so the shape assert fires. Works at `tp_size=1` only because `16 // 1 == 16`.
**(2) xgrammar guided decoding crashes at startup.** Serve with `guided_decoding_backend=xgrammar`:
```
'Gemma4ForConditionalGeneration' object has no attribute 'vocab_size_padded'
```
Root cause: `py_executor_creator` reads `model.vocab_size_padded` to build the xgrammar token bitmask. Text-only models inherit this property from `DecoderModelForCausalLM`, but the multimodal wrapper holds the language model as `self.llm` and does not expose it (other multimodal wrappers such as `modeling_gemma3vl` delegate it).
## Expected behavior
Gemma4 multimodal serves at `tp_size > 1` and with `guided_decoding_backend=xgrammar` without crashing at startup.
Fix proposed in PR #15566.
1 条评论