ITADN

linux_job_v3.yml fails to pull private ECR images: "no basic auth credentials"

#8152Openrishisinhanj 创建于 2026-06-05
R
rishisinhanjcommented
## Summary `linux_job_v3.yml@main` (added in #8129) fails to pull private-ECR docker images, surfacing as: ``` We noticed the 8 GPU Integration Test on H100 has been failing for the last 3 or 4 days with a issue with the docker container Initialization. We have put in the error below. Error response from daemon: Head "https://308535385114.dkr.ecr.us-east-1.amazonaws.com/v2/torchtitan/torchtitan-rocm-ubuntu-22.04-clang12/manifests/<sha>": no basic auth credentials Error: Docker pull failed with exit code 1 ``` The pull is attempted by the runner's `Initialize containers` step (step 2 of the job), which fires *before* `Configure AWS credentials` (step 5). When step 2 fails, step 5 is skipped, so ECR auth never runs. `linux_job_v2.yml@main` on the same runner, same image, same registry works correctly — its `aws-actions/configure-aws-credentials` + `amazon-ecr-login` steps run *before* a manual `docker pull`. ## Repro Affects the ROCm leg of `pytorch/torchtitan`'s `8 GPU Integration Test on H100` workflow, which switched from v2 → v3 in [pytorch/torchtitan#3464](https://github.com/pytorch/torchtitan/pull/3464) (merged 2026-06-01 19:42 UTC). | | Workflow | Reusable | Result | |---|---|---|---| | Pre-#3464 | `integration_test_8gpu_h100.yaml` | `linux_job_v2.yml@main` | Pull docker image succeeds | | Post-#3464 | `integration_test_8gpu_h100.yaml` | `linux_job_v3.yml@main` | Initialize containers → `no basic auth credentials` | | Today (control) | `integration_test_8gpu_features.yaml` | `linux_job_v2.yml@main` | Pulls the same ROCm image on the same runner pool | Sample failing runs (all ROCm leg, all same error): - https://github.com/pytorch/torchtitan/actions/runs/27033360312 - https://github.com/pytorch/torchtitan/actions/runs/27020331040 - https://github.com/pytorch/torchtitan/actions/runs/27016364309 Sample passing run on v2 with the identical image + runner: - https://github.com/pytorch/torchtitan/actions/runs/27020331040 (Features workflow leg, same SHA) ## Step-ordering evidence **v3 (failing):** ``` 1. Set up job ok 2. Initialize containers FAIL <- docker pull, no ECR auth yet 3. Clean workspace skipped 4. Checkout test-infra skipped 5. Configure AWS credentials skipped <- never runs ... ``` **v2 (passing) for the same image/registry:** ``` 1. Set up job ok 2. Clean workspace ok ... Download aws-actions/configure-aws-credentials ok Download aws-actions/amazon-ecr-login ok Run pytorch/pytorch/.github/actions/ecr-login@main ok Authenticated as AROAUPVRELQNLLCOPFEJR:gha-ecr-login ok docker pull <ECR image> ok ``` ## Suggested direction (not prescriptive) Either: 1. Move ECR-login (the same `aws-actions/configure-aws-credentials` + `amazon-ecr-login` pair that v2 uses) into v3 *before* the `container:` directive is evaluated — likely needs to happen in a setup job whose output is then consumed, or via the runner's pre-job hook. 2. Document that v3 only supports public images / GHCR until ECR auth is wired in, so callers migrating from v2 know to keep v2 for private-ECR workloads. Happy to test any fix against the torchtitan workflow above. cc @huydhn (author of #8129 and the torchtitan migration in pytorch/torchtitan#3464)
1 条评论