ITADN

CI: 8 GPU Integration Test on H100 ROCm leg red on main (blocked on pytorch/test-infra#8152)

#3552Openrishisinhanj 创建于 2026-06-05
module: rocm
R
rishisinhanjcommented
## Summary The `8 GPU Integration Test on H100` workflow's ROCm leg has been red on every `main` run since [b598831d (#3464)](https://github.com/pytorch/torchtitan/pull/3464) merged on 2026-06-01, failing at container init with: ``` Error response from daemon: Head "https://308535385114.dkr.ecr.us-east-1.amazonaws.com/v2/torchtitan/torchtitan-rocm-ubuntu-22.04-clang12/manifests/<sha>": no basic auth credentials Error: Docker pull failed with exit code 1 ``` The root cause is upstream in `pytorch/test-infra`'s `linux_job_v3.yml` reusable workflow (which #3464 switched this workflow to). v3 uses GitHub Actions' native `container:` directive, which fires `Initialize containers` as step 2 — *before* `Configure AWS credentials` (step 5) runs — so private-ECR pulls have no credentials and fail. Other torchtitan workflows (`8 GPU Feature Tests`, `8 GPU Model Tests`) still call `linux_job_v2.yml` and pull the same image on the same runner pool successfully. We have filed a big on the pytorch/test-infra side. If it seems like a longer scale issue, we'd prefer moving back to version 2 so that our CI can be ok. ## Tracking Upstream issue: **pytorch/test-infra#8152** This issue exists so the failure is findable from inside torchtitan. No torchtitan-side fix is planned — once test-infra#8152 lands, the next push to `main` will pick it up automatically (we reference `linux_job_v3.yml@main`). ## Sample failing runs - https://github.com/pytorch/torchtitan/actions/runs/27033360312 - https://github.com/pytorch/torchtitan/actions/runs/27020331040 - https://github.com/pytorch/torchtitan/actions/runs/27016364309 ## Notes - The CUDA leg of the same workflow (on the OSDC pool `mt-l-bx86iamx-176-1800-h100-8`) is unaffected and passes. - Pre-#3464, this workflow's ROCm leg was also red — but for a different, downstream reason (FSDP symmetric-memory init in the test script). Once #8152 is fixed, that older failure may resurface; we have made a fix for it, but not sure it works yet till this bug is fixed.
2 条评论