# Design: Deterministic devcontainer image digests

**Issue:** [xmtp/libxmtp#3461](https://github.com/xmtp/libxmtp/issues/3461)

## Context

PR #3457 introduced `.github/workflows/test-devcontainer.yml`, which builds
the devcontainer image, pins `devcontainer.json` to the freshly-built digest,
and asserts `git diff --quiet -- .devcontainer/devcontainer.json`. The
workflow passed on the PR but immediately failed on the subsequent push to
`main` — the two builds of the same commit produced different manifest
digests:

- PR build (committed):  `sha256:f8c7e41f2d51e2c6974131085f7028d1a8a309d3ca5ddbd3a76c2aa9ccac9688`
- Main build (expected): `sha256:3cd92ccc9a182a542056d99c5f02892ab8b01be0257cf64f03e5eb2d475291a2`

The workflow was *designed* to be deterministic (`SOURCE_DATE_EPOCH: "0"`,
`provenance: false`, no explicit `labels:` passed to `build-push-action`).
The comment in the workflow even mentions "dropping volatile labels
(metadata-action's default set includes `created` and `revision`)". Yet the
digests still differ.

## Root cause

Inspection of the failing run's logs (job 70870725839) reveals that
`docker/metadata-action@v6` sets two environment variables which
`docker/build-push-action@v7` picks up automatically via `@docker/actions-toolkit`:

```text
DOCKER_METADATA_OUTPUT_LABELS: org.opencontainers.image.created=2026-04-10T23:45:36.826Z
                               org.opencontainers.image.version=main
                               ...
DOCKER_METADATA_OUTPUT_ANNOTATIONS: manifest:org.opencontainers.image.created=2026-04-10T23:45:36.826Z
                                    manifest:org.opencontainers.image.version=main
                                    ...
```

Because `build-push-action` reads `DOCKER_METADATA_OUTPUT_LABELS` /
`DOCKER_METADATA_OUTPUT_ANNOTATIONS` as defaults for its `labels` /
`annotations` inputs (even though the workflow never passes them
explicitly), the image config **and** the OCI manifest each acquire two
volatile fields:

1. **`org.opencontainers.image.created`** — set to the wall-clock
   millisecond when the metadata-action step ran. This changes on every
   single CI run, so every build's manifest digest is different. This
   is **100% non-deterministic** regardless of anything else.

2. **`org.opencontainers.image.version`** — derived from the tag type that
   matched. On a PR build this is `pr-<number>`; on a push to `main` this
   is `main`. So **the PR build's manifest can never match the main build's
   manifest**, even if every other field is identical. The PR → merge →
   main handoff is structurally broken.

`SOURCE_DATE_EPOCH: "0"` only affects the image config's own `created`
timestamp and layer-file mtimes (BuildKit 0.12+). It does not touch
metadata-action's labels or annotations — those are injected *after*
buildkit, as part of the push pipeline.

There is a secondary, latent source of non-determinism: the Dockerfile
uses `FROM mcr.microsoft.com/devcontainers/base:ubuntu-24.04`, a mutable
tag. Microsoft rebuilds and re-tags this image periodically (for security
patches). Between a PR build and a later main build, the tag may resolve
to a different digest, changing all subsequent layers and the final
manifest. This didn't fire in the specific failing run (the log shows the
same base digest `d94c97dd9cacf183d0a6fd12a8e87b526e9e928307674ae9c94139139c0c6eae`
resolved for both), but it's a bomb waiting to go off as soon as
Microsoft rebuilds.

## Non-goals

- Making apt/curl/nix-env calls in the Dockerfile fully deterministic
  (pinning every transitive package version). This is a larger refactor
  and is out of scope for this fix. The current mitigation is that all
  these network calls happen in `RUN` layers that are layer-cached, so
  rebuilding the same Dockerfile within minutes reliably produces the
  same layer content.
- Switching the devcontainer build to Nix-based image construction.

## Requirements (EARS)

- **R1** — When `docker/metadata-action@v6` runs in
  `.github/workflows/test-devcontainer.yml`, the resulting
  `org.opencontainers.image.created` label and annotation SHALL be a
  deterministic value that does not vary with wall-clock time.
- **R2** — When `docker/metadata-action@v6` runs in
  `.github/workflows/test-devcontainer.yml`, the resulting
  `org.opencontainers.image.version` label and annotation SHALL be a
  value that is identical for any build of the same source commit,
  regardless of whether the build was triggered by `pull_request` or
  `push` to `main`.
- **R3** — When `.devcontainer/Dockerfile` references its base image, it
  SHALL reference it by content digest, not by a mutable tag.
- **R4** — When the `test-devcontainer` job runs on two separate CI runs
  against the same commit with the same upstream base image digest
  (and the same package mirror state), the pushed image SHALL have the
  same manifest digest on both runs.
- **R5** — When the fix is applied, the documented "author workflow" for
  updating the devcontainer (edit Dockerfile → open PR → run the
  `set-devcontainer-image` one-liner from the PR comment → push) SHALL
  continue to work without modification.

## Design

### Change 1: Override volatile labels/annotations in metadata-action

Override `org.opencontainers.image.created` and
`org.opencontainers.image.version` in the `docker/metadata-action@v6`
step. metadata-action's `labels:` and `annotations:` inputs append to
its defaults, with later keys overriding earlier ones in the final
`dedupAndSortLabels()` pass. So explicitly specifying these two keys
replaces the volatile defaults.

```yaml
- name: Extract metadata (tags, labels)
  id: meta
  uses: docker/metadata-action@v6
  with:
    images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
    tags: |
      type=ref,event=branch
      type=ref,event=pr
    labels: |
      org.opencontainers.image.created=1970-01-01T00:00:00.000Z
      org.opencontainers.image.version=libxmtp-devcontainer
    annotations: |
      manifest:org.opencontainers.image.created=1970-01-01T00:00:00.000Z
      manifest:org.opencontainers.image.version=libxmtp-devcontainer
```

**Why `1970-01-01T00:00:00.000Z` for `created`?** — It matches the
`SOURCE_DATE_EPOCH: "0"` already set on the build step (Unix epoch 0),
keeping the config blob's `created` and the labels/annotations'
`created` consistent.

**Why `libxmtp-devcontainer` for `version`?** — The metadata-action's
`version` default is derived from whichever tag rule matched
(`type=ref,event=pr` → `pr-123`, `type=ref,event=branch` → `main`),
which by construction differs between PR and main builds. We need a
value that is identical for both. A fixed string keyed to the project
is deterministic, self-describing, and requires no template logic.
Using `github.sha` would *not* work: the PR event's `github.sha` is
the virtual merge commit, while the push-to-main event's `github.sha`
is the commit that actually landed — these are different SHAs for the
same logical change.

Note that `revision`, `source`, `title`, `url`, `description`, and
`licenses` (the other defaults) are already deterministic for a given
commit, so they don't need overrides.

### Change 2: Pin the base image by digest

Replace the mutable tag in `.devcontainer/Dockerfile`:

```dockerfile
FROM mcr.microsoft.com/devcontainers/base:ubuntu-24.04@sha256:d94c97dd9cacf183d0a6fd12a8e87b526e9e928307674ae9c94139139c0c6eae
```

The specific digest `d94c97dd9cacf183d0a6fd12a8e87b526e9e928307674ae9c94139139c0c6eae`
is the one that was resolved for `ubuntu-24.04` in the failing run's
logs, so pinning to it preserves the current behavior while making
future rebuilds of the same Dockerfile bit-identical regardless of
whether Microsoft has re-tagged the upstream image.

This also makes devcontainer builds resilient to a common failure mode:
the base image being updated with a breaking change (e.g., new
`useradd` defaults, new base package versions) that silently propagates
to everyone rebuilding the image. Pinning by digest means security
updates are explicit, via a dependabot-style bump.

## Testing & Validation

**Local testability is limited.** Fork PRs are gated out of
`test-devcontainer` (it requires `head.repo.full_name == github.repository`
for access to GHCR push credentials), so we cannot run the workflow
end-to-end from a fork PR. Validation strategy:

1. **Static review of the logs.** The failing run's logs (job
   70870725839) show the exact env vars being auto-propagated and the
   exact labels/annotations being embedded. The fix directly targets
   those two fields. A careful diff of the metadata-action step output
   before and after the change confirms that `created` is now static
   and `version` is now static.

2. **Metadata-action source audit.** `dedupAndSortLabels()` in
   `docker/metadata-action@v6` (`src/meta.ts`) iterates defaults first,
   then user-provided labels, storing into a `Record<string, string>`.
   Later writes clobber earlier writes, so user overrides always win.
   Same logic applies to annotations. This confirms the override
   mechanism is correct.

3. **First main-branch build after merge.** The first run of
   `test-devcontainer` after this PR lands on `main` will itself
   produce a new digest (because the labels/annotations embedded in
   every prior image now differ), so the sync check will fail once
   more with a *single* clearly-identified new digest. The maintainer
   then runs the documented `set-devcontainer-image` one-liner once,
   commits, and every subsequent rebuild on the same commit produces
   the same digest. This is a one-time migration cost equivalent to
   any other Dockerfile edit.

4. **Long-term validation.** Once the new digest is committed, any
   future push (PR → main) on an unchanged devcontainer should sail
   through the sync check without manual intervention. If it doesn't,
   we've missed another source of non-determinism and the investigation
   continues.

## Open questions / caveats

- **Residual non-determinism from apt/curl/nix-env.** If
  `mcr.microsoft.com/devcontainers/base` is pinned and metadata labels
  are stabilized, the remaining risk is network-fetched content inside
  `RUN` steps: apt package versions, the nodesource setup script, the
  Nix install script, and the nixpkgs channel resolution. Within the
  minutes-long gap between a PR merging and the main-branch build
  firing, these are effectively stable. Across a longer gap (days), any
  of them could change and re-break the check. Fixing this properly
  would require pinning exact apt versions and using a pinned nixpkgs
  tarball — a worthwhile follow-up but too invasive for this fix.
- **Buildkit version drift.** `docker/setup-buildx-action@v4` does not
  pin a specific BuildKit version; a version bump in buildx itself
  could change manifest serialization details (field ordering, label
  encoding). In practice this is rare and out of our control.
