linux_job_v3.yml fails to pull private ECR images: "no basic auth credentials"
## Summary
`linux_job_v3.yml@main` (added in #8129) fails to pull private-ECR docker images, surfacing as:
```
We noticed the 8 GPU Integration Test on H100 has been failing for the last 3 or 4 days with a issue with the docker container Initialization. We have put in the error below.
Error response from daemon: Head "https://308535385114.dkr.ecr.us-east-1.amazonaws.com/v2/torchtitan/torchtitan-rocm-ubuntu-22.04-clang12/manifests/<sha>": no basic auth credentials
Error: Docker pull failed with exit code 1
```
The pull is attempted by the runner's `Initialize containers` step (step 2 of the job), which fires *before* `Configure AWS credentials` (step 5). When step 2 fails, step 5 is skipped, so ECR auth never runs.
`linux_job_v2.yml@main` on the same runner, same image, same registry works correctly — its `aws-actions/configure-aws-credentials` + `amazon-ecr-login` steps run *before* a manual `docker pull`.
## Repro
Affects the ROCm leg of `pytorch/torchtitan`'s `8 GPU Integration Test on H100` workflow, which switched from v2 → v3 in [pytorch/torchtitan#3464](https://github.com/pytorch/torchtitan/pull/3464) (merged 2026-06-01 19:42 UTC).
| | Workflow | Reusable | Result |
|---|---|---|---|
| Pre-#3464 | `integration_test_8gpu_h100.yaml` | `linux_job_v2.yml@main` | Pull docker image succeeds |
| Post-#3464 | `integration_test_8gpu_h100.yaml` | `linux_job_v3.yml@main` | Initialize containers → `no basic auth credentials` |
| Today (control) | `integration_test_8gpu_features.yaml` | `linux_job_v2.yml@main` | Pulls the same ROCm image on the same runner pool |
Sample failing runs (all ROCm leg, all same error):
- https://github.com/pytorch/torchtitan/actions/runs/27033360312
- https://github.com/pytorch/torchtitan/actions/runs/27020331040
- https://github.com/pytorch/torchtitan/actions/runs/27016364309
Sample passing run on v2 with the identical image + runner:
- https://github.com/pytorch/torchtitan/actions/runs/27020331040 (Features workflow leg, same SHA)
## Step-ordering evidence
**v3 (failing):**
```
1. Set up job ok
2. Initialize containers FAIL <- docker pull, no ECR auth yet
3. Clean workspace skipped
4. Checkout test-infra skipped
5. Configure AWS credentials skipped <- never runs
...
```
**v2 (passing) for the same image/registry:**
```
1. Set up job ok
2. Clean workspace ok
...
Download aws-actions/configure-aws-credentials ok
Download aws-actions/amazon-ecr-login ok
Run pytorch/pytorch/.github/actions/ecr-login@main ok
Authenticated as AROAUPVRELQNLLCOPFEJR:gha-ecr-login ok
docker pull <ECR image> ok
```
## Suggested direction (not prescriptive)
Either:
1. Move ECR-login (the same `aws-actions/configure-aws-credentials` + `amazon-ecr-login` pair that v2 uses) into v3 *before* the `container:` directive is evaluated — likely needs to happen in a setup job whose output is then consumed, or via the runner's pre-job hook.
2. Document that v3 only supports public images / GHCR until ECR auth is wired in, so callers migrating from v2 know to keep v2 for private-ECR workloads.
Happy to test any fix against the torchtitan workflow above.
cc @huydhn (author of #8129 and the torchtitan migration in pytorch/torchtitan#3464)
1 条评论