resnet50-v1-12 / resnet50-v2-7 produce all-NaN output on hip/gfx1100 when batch dim is dynamic
codegen/rocm
### Summary
Compiling and running `resnet50-v1-12.onnx` (and `resnet50-v2-7.onnx`) for `--iree-hal-target-device=hip --iree-rocm-target=gfx1100` produces an output tensor that is entirely NaN. The reference output from `onnxruntime` is fine.
The bug is sensitive to whether the **batch dimension is static or dynamic** at MLIR-import time. With static batch (`[1,3,224,224]`) the output matches ORT to ~5e-6. With dynamic batch (`[?,3,224,224]`), even when the dim is resolved to 1 at runtime, every output element is NaN.
### Repro
Requires `iree-base-compiler` and `iree-base-runtime` (any recent nightly):
`pip install --find-links https://iree.dev/pip-release-links.html --pre --upgrade iree-base-compiler iree-base-runtime`.
```bash
mkdir -p /tmp/resnet50-nan && cd /tmp/resnet50-nan
# 1. Download the original ONNX (dynamic batch dim).
wget -q https://huggingface.co/onnxmodelzoo/legacy_models/resolve/main/validated/vision/classification/resnet/model/resnet50-v1-12.onnx
# 2. Deterministic input (matches the iree-test-suites fixture).
python3 -c "import numpy as np; np.random.default_rng(0).random((1,3,224,224), dtype=np.float32).tofile('input.bin')"
# 3. Import + compile + run.
iree-import-onnx resnet50-v1-12.onnx --opset-version 17 -o resnet50.mlir
iree-compile resnet50.mlir --iree-hal-target-device=hip --iree-rocm-target=gfx1100 -o resnet50.vmfb
iree-run-module --module=resnet50.vmfb --device=hip \
--input=1x3x224x224xf32=@input.bin --output=@out.bin
# 4. Check the output.
python3 -c "import numpy as np; y=np.fromfile('out.bin', dtype=np.float32); print(f'NaN: {np.isnan(y).sum()}/{y.size}')"
```
Expected: `NaN: 0/1000`.
Actual: `NaN: 1000/1000`.
The imported MLIR's entry signature is:
```mlir
func.func @mxnet_converted_model(
%arg0: !torch.vtensor<[?,3,224,224],f32>) -> !torch.vtensor<[?,1000],f32>
```
If you statically pin the ONNX batch dim to 1 *before* `iree-import-onnx` (so the MLIR becomes `[1,3,224,224]`), the same compile + run produces a correct output (`NaN: 0 / 1000`, diff ~5e-6).
The same behavior reproduces with `resnet50-v2-7.onnx` from the same HuggingFace path.
### Environment
- Hardware: AMD Radeon PRO W7900 (gfx1100, RDNA3)
- ROCm: 7.x
- IREE commit: `b7ee744f` (head of PR #24394's CI build)
### Originally observed in
CI job `Test ONNX / test_onnx_models :: amdgpu_hip_rdna3` in workflow run [25530086833](https://github.com/iree-org/iree/actions/runs/25530086833). Confirmed independent of that PR — its only changes are to the iree-run-module float-comparison logic, which cannot affect the model's actual output.
### Triage hints
- Same input bytes, same ORT reference, same hardware, same compiler/runtime binaries — only the input MLIR differs (static vs dynamic batch). The divergence is in the lowering/codegen path that takes the dynamic batch dim.
- All 1000 output elements being NaN suggests garbage produced very early in the network (first conv or batchnorm) and then propagated. `iree-compile --iree-hal-dump-executable-sources-to=...` plus isolating the first few dispatches should narrow it.
- Worth checking other model-zoo models with dynamic batch on gfx1100 for the same symptom.
关闭于 2026-05-11 0 条评论