[Gemma 4] Gemma4UnifiedForConditionalGeneration text-only inference produces degenerate output (token repetition collapse)

- `transformers`: latest `main` (`5.10.0.dev0`, tested 2026-06-08) - `torch`: 2.7.0 - Platform: macOS, Apple M4 Pro (reproduced on **both MPS and CPU**) - Python: 3.12 The collapse is **device-, dtype-, and attention-implementation-independent** — it reproduces on CPU with `float32` and `attn_implementation="eager"`, so it is not an MPS or a fused-kernel artifact. ### Who can help? @ArthurZucker @molbap (Gemma) ### Reproduction ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "google/gemma-4-12B-it" tok = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="cpu" ).eval() prompt = "The capital of France is" ids = tok(prompt, return_tensors="pt") # 1) generation collapses to a repeated token out = model.generate(**ids, max_new_tokens=12, do_sample=False) print(tok.decode(out[0], skip_special_tokens=True)) # -> "The capital of France is111111111111" # 2) prompt cross-entropy is far too high for a coherent 12B with torch.no_grad(): logits = model(**ids).logits loss = torch.nn.functional.cross_entropy( logits[0, :-1], ids["input_ids"][0, 1:] ) print(float(loss)) # ~17.9 (a coherent 12B sits at ~2-3) ``` **Expected:** coherent continuation (e.g. "… Paris."), prompt loss in the ~2–3 range. **Actual:** the model emits a fixed repeated token (`"1111…"`); prompt cross-entropy ≈ **17.9**; greedy top-5 next-token candidates are `['1', '-', '.', '0', '_']`. ### The weights are good — the fault is in the `transformers` forward The **same weights**, converted to a `Q4_K_M` GGUF, generate coherent text under `llama.cpp` (build `9430`). So this is not a checkpoint problem; the `Gemma4Unified` forward path in `transformers` is producing corrupted hidden states for **text-only** input. ### Diagnostic findings We ran an in-process diagnosis to localize the fault. Summary: 1. **Multimodal context partially rescues the text path.** With a *real* image in the input, the text-token cross-entropy drops from **17.9 → 5.95**. Dummy / zero-embedding prepends do **not** rescue it (best 13.7). Setting `mm_token_type_ids` to all-zeros has **no effect**. This points to a fault on the **text-only path** that the image-content branch happens to avoid. 2. **`language_model.forward` receives well-formed inputs in both cases.** Per-type attention masks, sequential `position_ids`, and correctly scaled input embeddings are all structurally identical between text-only and with-image runs at the `language_model` input boundary — so this is **not** a missing-input-state problem at that boundary; the corruption is inside the decoder stack. 3. **Eleven components individually ruled out** as the sole cause (each toggled/normalized and re-measured, no recovery): per-type RoPE dispatch, per-type attention-mask dispatch, attention scaling (two variants), `embed_scale`, `final_logit_softcapping`, the RMSNorm `+1` convention, `layer_scalar` (three variants), and `v_norm`. 4. **Attention weights look healthy** (max attention probability 0.28–0.71, entropy 0.9–1.9 nats across layers) — no attention-collapse signature. 5. **Hidden-state norms stay alive and varied across all 48 layers** — magnitude is preserved while *content* is corrupted, consistent with a subtle per-layer transform/dispatch error rather than a NaN/overflow blowup. ### Possibly related - #45200 — `mm_token_type_ids` handling for fine-tuning. **Tested; no effect on this inference bug.** - google-deepmind/gemma#622 — reported repetition on the 31B / 26B Gemma 4 variants (same family; possibly a shared root cause). ### Affected - `google/gemma-4-12B-it` **text-only** inference (`Gemma4UnifiedForConditionalGeneration`). - Multimodal (with-image) inference is **partially** functional (loss ~5.95). --- *Draft prepared 2026-06-08. Probe numbers from `transformers` `5.10.0.dev0` on M4 Pro.*