ITADN

resnet50-v1-12 / resnet50-v2-7 produce all-NaN output on hip/gfx1100 when batch dim is dynamic

#24414ClosedhanhanW 创建于 2026-05-08
codegen/rocm
H
hanhanWcommented
### Summary Compiling and running `resnet50-v1-12.onnx` (and `resnet50-v2-7.onnx`) for `--iree-hal-target-device=hip --iree-rocm-target=gfx1100` produces an output tensor that is entirely NaN. The reference output from `onnxruntime` is fine. The bug is sensitive to whether the **batch dimension is static or dynamic** at MLIR-import time. With static batch (`[1,3,224,224]`) the output matches ORT to ~5e-6. With dynamic batch (`[?,3,224,224]`), even when the dim is resolved to 1 at runtime, every output element is NaN. ### Repro Requires `iree-base-compiler` and `iree-base-runtime` (any recent nightly): `pip install --find-links https://iree.dev/pip-release-links.html --pre --upgrade iree-base-compiler iree-base-runtime`. ```bash mkdir -p /tmp/resnet50-nan && cd /tmp/resnet50-nan # 1. Download the original ONNX (dynamic batch dim). wget -q https://huggingface.co/onnxmodelzoo/legacy_models/resolve/main/validated/vision/classification/resnet/model/resnet50-v1-12.onnx # 2. Deterministic input (matches the iree-test-suites fixture). python3 -c "import numpy as np; np.random.default_rng(0).random((1,3,224,224), dtype=np.float32).tofile('input.bin')" # 3. Import + compile + run. iree-import-onnx resnet50-v1-12.onnx --opset-version 17 -o resnet50.mlir iree-compile resnet50.mlir --iree-hal-target-device=hip --iree-rocm-target=gfx1100 -o resnet50.vmfb iree-run-module --module=resnet50.vmfb --device=hip \ --input=1x3x224x224xf32=@input.bin --output=@out.bin # 4. Check the output. python3 -c "import numpy as np; y=np.fromfile('out.bin', dtype=np.float32); print(f'NaN: {np.isnan(y).sum()}/{y.size}')" ``` Expected: `NaN: 0/1000`. Actual: `NaN: 1000/1000`. The imported MLIR's entry signature is: ```mlir func.func @mxnet_converted_model( %arg0: !torch.vtensor<[?,3,224,224],f32>) -> !torch.vtensor<[?,1000],f32> ``` If you statically pin the ONNX batch dim to 1 *before* `iree-import-onnx` (so the MLIR becomes `[1,3,224,224]`), the same compile + run produces a correct output (`NaN: 0 / 1000`, diff ~5e-6). The same behavior reproduces with `resnet50-v2-7.onnx` from the same HuggingFace path. ### Environment - Hardware: AMD Radeon PRO W7900 (gfx1100, RDNA3) - ROCm: 7.x - IREE commit: `b7ee744f` (head of PR #24394's CI build) ### Originally observed in CI job `Test ONNX / test_onnx_models :: amdgpu_hip_rdna3` in workflow run [25530086833](https://github.com/iree-org/iree/actions/runs/25530086833). Confirmed independent of that PR — its only changes are to the iree-run-module float-comparison logic, which cannot affect the model's actual output. ### Triage hints - Same input bytes, same ORT reference, same hardware, same compiler/runtime binaries — only the input MLIR differs (static vs dynamic batch). The divergence is in the lowering/codegen path that takes the dynamic batch dim. - All 1000 output elements being NaN suggests garbage produced very early in the network (first conv or batchnorm) and then propagated. `iree-compile --iree-hal-dump-executable-sources-to=...` plus isolating the first few dispatches should narrow it. - Worth checking other model-zoo models with dynamic batch on gfx1100 for the same symptom.
关闭于 2026-05-11 0 条评论