# Artifact.from_id + ArtifactRegistry

_Why are we doing this? What's the benefit?_

Today an artifact's identity is its GCS path. `Artifact.from_path(...)` resolves the metadata sidecar under that path, which is fine until the path needs to change — and we can't easily restructure GCS. Downstream code can't refer to a dataset or model checkpoint by a stable name; it has to know the exact computed path that produced it.

We want a path-independent way to refer to artifacts: `Artifact.from_id("datasets/fineweb-resiliparse", "2026.05.29", NormalizedData)` should resolve to the same payload regardless of where the bytes live, today or after a re-org. This requires a small registry that maps `(id, version) → path + metadata`, with a default backed by GCS / local filesystem and a user-supplied override hook.

## Background

Current state, prior art, and pitfalls survey: see [`research.md`](./research.md). Short version:

- `Artifact` is a JSON metadata envelope (`artifact.json` sidecar) over Pydantic payloads like `PathMetadata`, `TokenizedAttrData`. `from_path(base, type)` requires `type` to subclass `BaseModel`. ([`lib/marin/src/marin/execution/artifact.py:32`](https://github.com/marin-community/marin/blob/f07122bbd20376b97e8951ac75f0eb8a7ebee7e3/lib/marin/src/marin/execution/artifact.py#L32))
- Marin already has *content-addressed* path hashes from the executor ([`step_spec.py:90`](https://github.com/marin-community/marin/blob/f07122bbd20376b97e8951ac75f0eb8a7ebee7e3/lib/marin/src/marin/execution/step_spec.py#L90)) and *schema* versions inside payloads. This proposal adds a third axis: a **logical name + user-supplied version** that is independent of both.
- No existing registry in `marin/execution/`. Prior art (MLflow, W&B, DVC, HF Hub) converges on `(name, version) → metadata + pointer`, with mutable aliases layered over immutable versions. A manifest-file-on-storage registry is firmly in the DVC / HF-as-git camp.

## Challenges

_What's hard?_

The core mechanics are easy; the contentious bits are policy choices.

1. **Concurrent writes on object storage.** GCS has no native locking. We avoid this at v1 with a layout where concurrent registrations of distinct `(id, version)` pairs never touch the same key; same-pair races are last-writer-wins on a backend that allows overwrite (on the canonical 99-year-retention bucket, overwrites are rejected, so the second writer errors instead). CAS (`if-generation-match`) is a clean follow-up.
2. **Version flavor is a one-way door.** Once callers pin a version string, changing the version model costs a migration. We pick **CalVer** — `YYYY.MM.DD` with an optional `-<modifier>` suffix (`2026.10.01-fall-hero`) — validated by regex *and* an actual date parse. The modifier alphabet is restricted to `[A-Za-z0-9._-]`; no `/`, since the whole version is a single path segment. This buys lexical-equals-chronological ordering for free, gives provenance (the date the artifact was minted), and leaves room for hero / RC suffixes; see Decisions for what we considered instead.

## Costs / Risks

- New surface area in `lib/marin/src/marin/execution/` that downstream code will start depending on.
- Two ways to load the same artifact — `from_path` (legacy) and `from_id` (new). `from_id` delegates to `from_path` so the typed-loader behavior stays in one place; the duplication is API-only.
- Users must *register* artifacts before `from_id` works. Existing pipelines keep working unchanged; only consumers who want id-based lookup pay the one-line `registry.register(...)` cost.

## Design

_How are we doing this?_

Two new pieces in `lib/marin/src/marin/execution/`:

**`ArtifactRegistry`** — an interface plus a default `FilesystemArtifactRegistry` backed by GCS or local FS via `rigging.filesystem` / `url_to_fs` (the abstractions `artifact.py` already uses, [artifact.py:9](https://github.com/marin-community/marin/blob/f07122bbd20376b97e8951ac75f0eb8a7ebee7e3/lib/marin/src/marin/execution/artifact.py#L9)). Two methods:

- `register(id, version, uri) -> ArtifactEntry` — fails loudly if `(id, version)` already exists. No overwrite, no upsert at v1.
- `lookup(id, version) -> ArtifactEntry` — returns the stored entry or raises `ArtifactNotFoundError`.

`ArtifactEntry` is a minimal Pydantic record: `id`, `version`, `uri`. It serves both as the persisted file schema and as the return type of `lookup` / `register`. Extra fields (`created_at`, `metadata`, `artifact_type`) are deferred until there's a consumer for them — easier to add later than to deprecate.

**Manifest layout** is one file per `(id, version)`:

```
<registry-root>/<namespace>/<name>/<version>.json
```

Example: `gs://marin-artifact-registry/datasets/fineweb-resiliparse/2026.05.29.json`. The hot path is id+version → known key (O(1) GET); directory size doesn't matter for reads. Concurrent writers of distinct `(id, version)` pairs never contend; same-pair races are last-writer-wins on an overwrite-allowing backend (rejected by the canonical retention bucket; no CAS at v1).

**Immutability is registry-level only.** A registered `(id, version)` always points at the same `uri`, but the bytes at that `uri` are protected by GCS IAM, not by this design. Anyone with write access can delete or repoint the underlying artifact; the open content-hash field would let `from_id` detect this.

**Decisions / considered alternatives.**

- *Sharded prefix* (`<name[-2:]>/<name>/<version[-2:]>/<version>.json`) — rejected: reads go straight to a known key, and "list all versions" is a debugging operation. Adding sharding later is a one-time migration.
- *Single global `registry.json`* — rejected (worst concurrency, worst diff-ability).
- *SQL-backed service* — out of scope; the manifest-file shape parallels DVC and HF-as-git, the right reference class for a library component.
- *Content-addressable storage (content-hash as identity)* — **considered and rejected.** Marin already content-addresses one layer down ([`step_spec.py:90`](https://github.com/marin-community/marin/blob/f07122bbd20376b97e8951ac75f0eb8a7ebee7e3/lib/marin/src/marin/execution/step_spec.py#L90)), so the registry hash would mostly duplicate that. More importantly, humans say "give me `fineweb-resiliparse` from 2026-05-29", not "`sha256:e3b0c44…`" — content-hash naming reintroduces the path-coupling problem we're trying to leave. Callers pick the version string; `register` fails on collision. A content-hash field for verification (not identity) remains an Open Question.
- *Free-form version strings* (the original draft: `[A-Za-z0-9_.-]+`) — **rejected** in favor of CalVer. Free-form gives no ordering, no provenance, no shared vocabulary; pinning structure now is cheaper than retrofitting it later.
- *Semver via `packaging.version()` (3-tuple, ordered)* — **considered.** Strict semver maps badly to datasets and trained checkpoints — "what's a breaking change for `fineweb-resiliparse`?" doesn't have an obvious answer, and the major/minor/patch discipline is overhead without payoff for this domain. CalVer doubles as a creation date and the zero-padded prefix gives lexical=chronological sort for free.
- *Mutable aliases (`latest`, `prod`, …)* — **intentionally not supported, non-goal not deferral.** The registry is `(id, version) → uri` and nothing else. Callers needing a "latest" pointer maintain it themselves. Rationale: aliases drift implicitly on every registration, breaking reproducibility of old run logs without anyone touching them — exactly the failure mode the registry exists to prevent.

**`Artifact.from_id`** delegates to existing machinery after resolving:

```python
@classmethod
def from_id(
    cls,
    artifact_id: str,                  # "<namespace>/<name>", e.g. "datasets/fineweb-resiliparse"
    version: str,                      # CalVer YYYY.MM.DD[-suffix], e.g. "2026.05.29-fall-hero"
    artifact_type: type[T] | None = None,  # Pydantic schema, same role as in from_path
    *,
    registry: ArtifactRegistry | None = None,
) -> T | PathMetadata | dict[str, Any]:
    reg = registry or get_default_registry()
    entry = reg.lookup(artifact_id, version)
    return cls.from_path(entry.uri, artifact_type)
```

`get_default_registry()` returns a module-level singleton built from `MARIN_ARTIFACT_REGISTRY`, falling back to the canonical `gs://marin-artifact-registry` (one global location so an id resolves the same from every region/cloud). The dedicated bucket has a 99-year retention policy, so manifests can't be deleted or overwritten — the append-only contract is enforced by storage and no separate backup is required. Tests must never touch that root: an autouse fixture in `tests/conftest.py` redirects the default to a temp dir and resets the cached singleton around each test. Tests and notebook users can also pass `registry=` to override per call.

**IDs** are strict `<namespace>/<name>` (one slash, both segments non-empty, restricted character set). `register` and `lookup` validate this up-front; ambiguous inputs raise immediately.

**Out of scope** at v1: explicit overwrite / upsert, content hashing in entries, multi-writer transactional semantics, the auto-registration shortcut from step outputs, deletion / garbage collection. Each is a clean follow-up; see [`spec.md`](./spec.md) for the explicit out-of-scope list. Mutable aliases (`latest`, `prod`, …) are a non-goal, not a deferral — see Decisions.

## Testing

_Agents make mistakes — how do we catch them?_

Integration tests against the local-FS backend cover the load-bearing behaviors:

- Round-trip: `register(id, "v1", uri, MyModel)` then `from_id(id, "v1", MyModel)` returns the same payload via the existing `from_path` deserialization.
- Immutability: second `register(id, "v1", ...)` raises a typed error; the original entry is untouched.
- ID and version validation: malformed ids (`"foo"`, `"foo/"`, `"a/b/c"`) and non-CalVer versions (`"v3"`, `"2026.5.29"`, `"2026.02.30"`) raise `InvalidArtifactIdError` before any FS access.
- Manifest layout invariant: a registered entry lands at the expected path. (Pins the on-disk contract.)
- Override: `from_id(..., registry=other)` ignores the default singleton.

We will also run one smoke test against a GCS sandbox bucket to confirm the same suite passes with a `gs://` registry root — this is the only test that exercises the real object-store semantics we care about.

## Open Questions

- **Content hash on `ArtifactEntry` for verification (not identity)?** Adding the field later is cheap (`extra="ignore"` on the model), but writing it only on new registrations leaves the registry mixed-mode forever. If reviewers want it, we add it at v1.
- **Per-region registries vs. one canonical region?** Marin runs in multiple regions and we don't want cross-region reads on the read path. Leaning per-region. Interacts with `uri` portability: a `us-east1` registry recording a `gs://marin-us-central2/...` uri is a cross-region pointer that may want to be a registration-time error.
- **Should `Artifact.save` auto-publish?** Tempting but couples the registry to every step output. Leaning *no* — explicit `registry.register(...)` — but reviewers may have a better instinct.
