# Experiments Agent Notes

Start with `/AGENTS.md`; this file adds experiment-specific guidance.

## Dataset Catalogs

All dataset definitions live under `experiments/datasets/`, one leaf module per
dataset (or dataset family), built with the lazy data builders in
`marin.experiment.data` (`tokenized`, `hf_download`, `pretokenized`, ...).

- A single-corpus dataset exposes `<name>_dataset() -> ArtifactStep[TokenizedCache]`.
- A keyed family exposes `<name>_datasets() -> dict[str, ArtifactStep[TokenizedCache]]`,
  keyed by subset; a `<name>_dataset(key)` builder is added only when the keys are a
  natural registry (e.g. a Paloma subset).
- Mixture weights are policy, not catalog: keep them in a separate
  `<NAME>_MIXTURE_WEIGHTS` constant keyed the same way, and let the experiment pick
  weights and assemble them with `mixture()`.
- A module that builds tokenized-cache handles is runnable: end it with
  `if __name__ == "__main__": dataset_main(<family-dict>)`. `python -m
  experiments.datasets.<name>` prints the build plan; `--run` (optionally `--only
  <keys>`) builds it. Modules of a different shape — `eval` (builds `EvalDataset`s)
  and the `instruction`/`preference` HF-id registries — are exceptions with no CLI.
  Never change a handle's `name=`/`version=`/`pin=`/path strings — those are
  artifact identities.
- `__init__.py` stays docstring-only (no re-exports); import catalogs directly, e.g.
  `from experiments.datasets.nemotron import nemotron_datasets`.

## Artifact Path Rules

- Do not hardcode `gs://...` paths in experiment configs unless there is no viable alternative.
- Prefer artifact dependencies resolved at run time via `ctx.path(dep)` inside a recipe's `build_config`, so paths stay region-aware and reproducible.
- To reference a pre-existing external output, use `adopt(name, version, source=...)` (see `marin.execution.lazy`) instead of a raw `gs://` URL.
- If a referenced artifact may come from another region, point its source at a `mirror://...` path so reads are local-first and copied once when needed.

## Cross-Region Large Artifacts

- For `mirror://` copies, up to 10 GB can be copied without explicit human permission.
- If a `mirror://` copy would exceed 10 GB, get explicit human permission first, regardless of previous instructions.
- After approval, mirror it once into the local region and reference that mirrored copy thereafter.

## Mirror FS (Quick Reference)

- `mirror://` is an fsspec filesystem for transparent cross-region reads.
- Read behavior: check local `marin_prefix()` first; if missing, scan other Marin regional buckets, copy to local once, then serve locally.
- Copying uses a distributed lock so concurrent workers do not duplicate the same transfer.
- Write behavior: writes go directly to the local prefix.
- Both `mirror://` copies and direct cross-region GCS reads share a single process-global `TransferBudget` (default 10 GB) that raises `TransferBudgetExceeded` when exhausted.

Implementation references: `lib/iris/src/iris/marin_fs.py` and `rigging.filesystem` (the `TransferBudget`).
