TritonGPURemoveLayoutConversions dominance failure compiling vLLM fused_moe_kernel on sm87

### Describe the bug I hit a `PassManager::run failed` while tuning vLLM's fused MoE kernel on a Jetson AGX Orin / NVIDIA T234-class device (`sm87`). The failure is in Triton's NVIDIA backend while running `TritonGPURemoveLayoutConversions` and produces an SSA dominance verifier error: ```text error: operand #0 does not dominate this use c_mask = token_mask[:, None] & (offs_cn[None, :] < N) ^ note: operand defined here (op in a child region) mask=token_mask[:, None] & (offs_k[None, :] < K - k * BLOCK_SIZE_K), ^ RuntimeError: PassManager::run failed ``` This looks related to the family of dominance issues in `TritonGPURemoveLayoutConversions`, especially: - #6733 - #9809 - #9928 - possibly #7786 - and the fix in #9020 However, I am filing this as a separate report because this is a concrete vLLM fused MoE tuning case on `sm87` / Jetson AGX Orin. The closed vLLM MoE report in #9809 was on GH200 (`sm90`) and was closed as fixed on trunk. I have not yet verified whether Triton `main` also fixes this `sm87` case. ### Reproduction context The failure occurs while running vLLM's fused MoE tuner: ```bash cd ~/vllm/benchmarks/kernels python benchmark_moe.py \ --model /home/rm01/models/dev/llm/Qwen3.6-35B-A3B-AWQ \ --tp-size 1 \ --dtype auto \ --batch-size 1 2 4 8 16 \ --tune \ --save-dir ~/fused_moe \ --trust-remote-code ``` Important observed behavior: - `batch_size=1` completed. - `batch_size=2` completed. - `batch_size=4` completed. - `batch_size=8` completed. - `batch_size=16` failed during autotuning, around `1.63k/1.92k` configurations. The failure is not a gradual memory/capacity failure: smaller batches complete, and batch 16 fails at a specific autotune candidate during Triton compilation. ### Earlier cache-related failure Before the compiler failure, I first hit a Triton cache metadata error: ```text json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) File ".../triton/runtime/cache.py", line 79, in get_group grp_data = json.load(f) ``` After moving the old Triton cache away and forcing a clean cache directory: ```bash ray stop --force 2>/dev/null || true mv ~/.triton/cache ~/.triton/cache.bad.$(date +%Y%m%d-%H%M%S) 2>/dev/null || true export TRITON_HOME=/home/rm01/.triton-vllm0202-moe export TRITON_CACHE_DIR=$TRITON_HOME/cache rm -rf "$TRITON_CACHE_DIR" mkdir -p "$TRITON_CACHE_DIR" ``` the JSON/cache issue disappeared. The later `PassManager::run failed` below is a separate compiler failure. ### Failing autotune candidate From the emitted MLIR shapes and the vLLM tuning search order, the failing candidate appears to be: ```text BLOCK_SIZE_M = 256 BLOCK_SIZE_N = 64 BLOCK_SIZE_K = 64 GROUP_SIZE_M = 1 num_warps = 4 num_stages = 2 ``` This matches the progress position around configuration 1633 of 1920 in vLLM's search space. ### Stack trace excerpt ```text ray.exceptions.RayTaskError(RuntimeError): ray::BenchmarkWorker.tune() File "/home/rm01/vllm/benchmarks/kernels/benchmark_moe.py", line 635, in tune kernel_time = benchmark_config( File "/home/rm01/vllm/benchmarks/kernels/benchmark_moe.py", line 306, in benchmark_config run() File "/home/rm01/vllm/benchmarks/kernels/benchmark_moe.py", line 295, in run return fused_experts( File ".../vllm/model_executor/layers/fused_moe/fused_moe.py", line 1606, in fused_experts return dispatch_fused_experts_func(inplace)( File ".../vllm/model_executor/layers/fused_moe/fused_moe.py", line 1527, in torch_vllm_outplace_fused_experts return torch.ops.vllm.outplace_fused_experts(**kwargs) File ".../vllm/model_executor/layers/fused_moe/fused_moe.py", line 1455, in outplace_fused_experts return fused_experts_impl( File ".../vllm/model_executor/layers/fused_moe/fused_moe.py", line 1801, in fused_experts_impl dispatch_fused_moe_kernel( File ".../vllm/model_executor/layers/fused_moe/fused_moe.py", line 898, in dispatch_fused_moe_kernel invoke_fused_moe_triton_kernel( File ".../vllm/model_executor/layers/fused_moe/fused_moe.py", line 774, in invoke_fused_moe_triton_kernel fused_moe_kernel[grid]( File ".../triton/runtime/jit.py", line 720, in run kernel = self._do_compile(key, signature, device, constexprs, options, attrs, warmup) File ".../triton/runtime/jit.py", line 849, in _do_compile kernel = self.compile(src, target=target, options=options.__dict__) File ".../triton/compiler/compiler.py", line 324, in compile next_module = compile_ir(module, metadata) File ".../triton/backends/nvidia/compiler.py", line 541, in <lambda> stages["ttgir"] = lambda src, metadata: self.make_ttgir(src, metadata, options, capability) File ".../triton/backends/nvidia/compiler.py", line 316, in make_ttgir pm.run(mod, 'make_ttgir') RuntimeError: PassManager::run failed ``` ### Verifier error excerpt ```text /home/rm01/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/fused_moe.py:554:24: error: operand #0 does not dominate this use c_mask = token_mask[:, None] & (offs_cn[None, :] < N) ^ /home/rm01/miniconda3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/fused_moe.py:488:28: note: operand defined here (op in a child region) mask=token_mask[:, None] & (offs_k[None, :] < K - k * BLOCK_SIZE_K), ^ ``` The emitted reproducer footer says: ```text Pipeline failed while executing [`TritonGPURemoveLayoutConversions` on 'builtin.module' operation]: reproducer generated at `std::errs, please share the reproducer above with Triton project.` ``` The relevant pipeline in the emitted reproducer targets `cuda:87`: ```text builtin.module( convert-triton-to-tritongpu{enable-source-remat=false num-ctas=1 num-warps=4 target=cuda:87 threads-per-warp=32}, tritongpu-coalesce, tritongpu-F32DotTC{emu-tf32=true}, triton-nvidia-gpu-plan-cta, tritongpu-remove-layout-conversions, ... nvgpu-warp-specialization{dump-intermediate-steps=false num-stages=2}, tritongpu-assign-latencies{num-stages=2}, tritongpu-schedule-loops, tritongpu-pipeline{dump-intermediate-steps=false num-stages=2}, ... tritongpu-remove-layout-conversions, ... triton-nvidia-gpu-fence-insertion{compute-capability=87}, triton-nvidia-mma-lowering, sccp, cse, canonicalize{...} ) ``` ### Why this looks like a Triton compiler issue The vLLM kernel compiles and runs for several smaller batch sizes. The failure is raised by Triton's MLIR verifier after a compiler pass, not by a vLLM shape assertion or runtime CUDA error. The source pattern is the vLLM fused MoE kernel reusing `token_mask`: - inside the `for k in range(...)` loop as part of the `tl.load` mask - later outside the loop for the final output store mask Triton's pass appears to rematerialize or move a value such that the final use sees a definition in a child region, producing invalid SSA dominance. ### Environment details Host/device: ```text Device: Jetson AGX Orin / NVIDIA T234-class GPU Compute capability: sm87 / cuda:87 OS/arch: Linux aarch64 ``` Python environment: ```text Python: 3.12 Triton: 3.6.0 PyTorch: version not collected in this run vLLM: local checkout, running benchmarks/kernels/benchmark_moe.py Ray: local Ray instance started by benchmark_moe.py ``` I can provide a full MLIR reproducer if needed. The pasted stderr contains the module, but it is long, so I kept the issue body to the stack trace, verifier error, pipeline, and failing autotune config. ### Expected behavior Triton should either compile the candidate or reject it with a valid compiler diagnostic. It should not generate invalid MLIR that fails SSA dominance verification inside `TritonGPURemoveLayoutConversions`. ### Actual behavior Compilation aborts with: ```text RuntimeError: PassManager::run failed Pipeline failed while executing [`TritonGPURemoveLayoutConversions` on 'builtin.module' operation] error: operand #0 does not dominate this use ``` This stops vLLM MoE tuning for `batch_size=16`, even though `batch_size=1/2/4/8` complete successfully.