CI: 8 GPU Integration Test on H100 ROCm leg red on main (blocked on pytorch/test-infra#8152)
module: rocm
## Summary
The `8 GPU Integration Test on H100` workflow's ROCm leg has been red on every `main` run since [b598831d (#3464)](https://github.com/pytorch/torchtitan/pull/3464) merged on 2026-06-01, failing at container init with:
```
Error response from daemon: Head "https://308535385114.dkr.ecr.us-east-1.amazonaws.com/v2/torchtitan/torchtitan-rocm-ubuntu-22.04-clang12/manifests/<sha>": no basic auth credentials
Error: Docker pull failed with exit code 1
```
The root cause is upstream in `pytorch/test-infra`'s `linux_job_v3.yml` reusable workflow (which #3464 switched this workflow to). v3 uses GitHub Actions' native `container:` directive, which fires `Initialize containers` as step 2 — *before* `Configure AWS credentials` (step 5) runs — so private-ECR pulls have no credentials and fail.
Other torchtitan workflows (`8 GPU Feature Tests`, `8 GPU Model Tests`) still call `linux_job_v2.yml` and pull the same image on the same runner pool successfully.
We have filed a big on the pytorch/test-infra side. If it seems like a longer scale issue, we'd prefer moving back to version 2 so that our CI can be ok.
## Tracking
Upstream issue: **pytorch/test-infra#8152**
This issue exists so the failure is findable from inside torchtitan. No torchtitan-side fix is planned — once test-infra#8152 lands, the next push to `main` will pick it up automatically (we reference `linux_job_v3.yml@main`).
## Sample failing runs
- https://github.com/pytorch/torchtitan/actions/runs/27033360312
- https://github.com/pytorch/torchtitan/actions/runs/27020331040
- https://github.com/pytorch/torchtitan/actions/runs/27016364309
## Notes
- The CUDA leg of the same workflow (on the OSDC pool `mt-l-bx86iamx-176-1800-h100-8`) is unaffected and passes.
- Pre-#3464, this workflow's ROCm leg was also red — but for a different, downstream reason (FSDP symmetric-memory init in the test script). Once #8152 is fixed, that older failure may resurface; we have made a fix for it, but not sure it works yet till this bug is fixed.
2 条评论