pr-bench-history `analyze` job fails on download-artifact 404 after a partial job re-run

[Copilot speaking] ## Summary The `PR Benchmark history` workflow's `analyze` job can fail deterministically at the **Download collected machine keys** step with a `(404) Not Found: workflow run not found` error from `actions/download-artifact@v4`, whenever the run has had a *subset* of its jobs re-run (e.g. after a flaky `collect` leg is retried). This is a GitHub tooling limitation, not a benchmark or code regression, but it leaves an unrecoverable red X on the PR that only a **fresh full workflow run** can clear. Filing for follow-up; it is safe to ignore per-incident because `analyze` is advisory (non-gating). ## Where it was observed - Run: `PR Benchmark history` run `30054989132` on PR #420 (branch `sandersaares-bench-regression-diagnosis`). - Failing job: `analyze` (e.g. job `89437480313`), step **Download collected machine keys**. ## Exact error ``` Preparing to download the following artifacts: - pr-bench-history-machine-key-windows-latest (ID: 8590219085, ...) - pr-bench-history-machine-key-ubuntu-latest (ID: 8585531731, ...) Starting download of artifact to: .../machine-keys/pr-bench-history-machine-key-windows-latest ##[error]Unable to download artifact(s): Failed to GetSignedArtifactURL: Received non-retryable error: Failed request: (404) Not Found: workflow run not found SHA256 digest of downloaded artifact is 2b95aacf... Artifact download completed successfully. ``` Note the self-contradiction in the log (a hard `##[error]` immediately followed by `Artifact download completed successfully.`): the action iterates artifacts, one resolves fine and one 404s, and the step still exits non-zero. ## How we got here 1. `collect (windows-latest)` first failed with *"The hosted runner lost communication with the server"* — an unrelated transient hosted-runner death after ~4h of a whole-workspace collection. 2. `gh run rerun --failed` retried just the windows `collect` leg, which then succeeded. Both `collect` legs are green. 3. But `analyze` (which `needs: [delta, collect]`) then re-ran against the same run-id and hit the 404 above. Re-running `analyze` again reproduces it **every time** — it is deterministic once the run has mixed job attempts. ## Root cause `actions/download-artifact@v4` invoked with an explicit `run-id:` resolves each artifact's signed URL through its backing workflow-run-attempt reference. After a partial re-run, the machine-key artifacts belong to a different run *attempt* than the one `analyze` now executes under, so the signed-URL lookup 404s ("workflow run not found"). This is a known class of `download-artifact@v4` + partial-re-run breakage, not specific to this repo's data. ## Impact - `analyze` is **advisory / non-gating**, so an affected PR is still `MERGEABLE`. - However, the workflow is *designed* to tolerate a partially-failed collect (`analyze` uses `if: !cancelled()` precisely so one dead leg still analyzes whatever landed). That resilience is defeated here: the moment any single `collect` leg needs a retry — which is common for a multi-hour, whole-workspace collection exposed to hosted-runner deaths — `analyze` becomes impossible to make green without re-running the *entire* matrix (~4h) from scratch. ## Suggested follow-up (options, not yet decided) - Make the machine-key hand-off robust to partial re-runs. Candidates: - Have `analyze` tolerate a machine-key artifact that fails to download (skip that platform's key, mirroring the existing "a failed leg uploaded nothing" handling) instead of hard-failing the whole step. - Add a retry/`continue-on-error` + fallback around the `download-artifact` step, or resolve keys via the API rather than pinning `run-id` in a way that breaks across attempts. - Pin `download-artifact` to the specific successful attempt, or upload machine keys to the shared scratch/Azure store the collect step already uses, instead of run-scoped Actions artifacts. - Separately, consider whether the whole-workspace scope (triggered when a PR touches workspace-global files such as root `Cargo.toml` / `constants.env`) should cap or shard the collect matrix so a single leg's runtime — and thus its exposure to transient runner deaths — is bounded. ## Workaround for now Trigger a fresh full `PR Benchmark history` run (new run-id, all artifacts under one attempt) if a green `analyze` is required; otherwise merge, since the job is advisory.