# Federation: what crosses a cluster boundary

A cluster with `peers:` in its config may hand whole jobs to another cluster. `marin` (GCP:
TPU and CPU) peers with `cw-rno2a` and `cw-us-east-02a` (CoreWeave: H100) and
`cw-us-east-08a` (CoreWeave: GB200), so a user submits every job to `marin` and GPU work
lands on CoreWeave.

This page is the job model. For the auth, networking, and DNS that carry a handoff, see
[`coreweave.md`](coreweave.md).

## Routing: classify at submit, place on the tick

Peer placement is not decided at submit. Submit only *classifies* a job; the controller's
single scheduling tick decides which peer a queued job lands on, alongside every local
scheduling decision. This keeps one thread of control for all placement and lets a job wait
for a peer to report free capacity instead of piling onto the first peer that merely *could*
host it.

`PeerRouter.classify` (`cluster/federation/router.py`) runs once per submission and returns one
of three dispositions:

1. **`QUEUE` on a `cluster=<peer>` pin.** `iris job run --target-cluster <peer>` sets it. The
   job queues for that peer even when it could run locally.
2. **`LOCAL` if the shape is locally feasible.** `job_feasibility` asks whether any local
   scaling group could in principle host the job.
3. **`QUEUE` if some reachable peer could host it** — its last capability heartbeat shows a
   backend whose advertised attributes satisfy **every routing constraint** on the job.
4. **`REJECT` otherwise**, so the job fails fast as unschedulable rather than wedging.

A `QUEUE` disposition parks the job in the parent's `federated_jobs` table in the
`QUEUED_HANDOFF` state. It is *not* yet assigned to a peer.

## Placement: the control tick drains the queue against availability

Each control tick, `FederationManager.plan_federation` (`cluster/federation/manager.py`) runs
the pure pass in `cluster/federation/availability.py` over the queued jobs and the peers' most
recent availability, and emits `(job → peer, backend)` promotions. A promotion is applied as a
conditional CAS (`promote_queued_handoff`): it advances the job to a pending handoff only if the
job is still queued, not cancelled, and non-terminal — so a cancel or terminalize racing the
tick can never be overtaken by a promotion. A confirmed promotion then delivers over the same
handoff machinery as before.

Placement translates a job's device request into an **availability gate**: N replicas of an
8×`h100` job becomes `ge(available:h100, N*8)`, and a peer backend hosts the job only when its
advertised free capacity meets the gate. This mirrors the `availability:<variant>` *EXISTS*
constraints the scheduler already uses for reservations, but the `available:<token>` metric is
*numeric* (a count), not a boolean.

A peer reports the resources available per band: alongside `amounts` it reports `held_by_band`,
what its *admitted* work holds at each priority band. A candidate's effective capacity on a
backend is that backend's free amount plus everything held **below** its own band (a
numerically higher band is lower priority) — the work the peer's scheduler would preempt to
admit the job. Placement spends idle capacity first, reclaims from the lowest-priority band
upward, and prefers a peer that needs no preemption at all. A peer that reports no band split
(a worker-daemon backend, or one predating the field) reclaims nothing and is gated on its free
amount alone.

Three properties keep placement honest without pretending to be exact:

- **Never summed across backends.** A job pins to one backend, so 6 free on one backend plus 4
  on another does not host an 8-GPU job. Availability is evaluated per backend.
- **Queued work at the peer does not suppress its numbers.** The free count subtracts only what
  *admitted* work holds — on Kubernetes, work whose Kueue scheduling gate is released, bound to
  a node or not. A pod still waiting in the peer's queue holds no chips, so a long queue there
  never makes the peer look full to the parent.
- **A reservation ledger bounds cross-tick over-assignment.** The tick runs on every submit
  wake — far more often than the 30s heartbeat — so a naive per-tick read would re-spend the
  same advertised number every tick. The ledger records capacity already promoted against a
  peer backend *since its last heartbeat* (keyed on the heartbeat's `observation_epoch_ms`), and
  effective availability is `advertised − reserved`. A strictly newer heartbeat — whose number
  already reflects the delivered jobs — resets the ledger. Over-assignment is thus bounded to a
  peer's advertised free capacity per observation, which the issue explicitly tolerates; the
  peer's own scheduler (and a requeue) is the backstop.

`max_federation_handoffs_per_cycle` (controller config) caps promotions per tick, a second
bound on a burst against stale metrics.

## What the router matches on

Routing constraints are the subset of constraints marked `routing=True` in
`CONSTRAINT_REGISTRY` (`cluster/constraints.py`): `device-type`, `device-variant`,
`preemptible`, `region`, `zone`.

Two consequences catch people out:

- **`gpu-count` is not a routing constraint.** It is a consumable, checked against a worker's
  free GPUs when the peer schedules the job. `H100x1` and `H100x8` route identically. GPUs
  pack: several tasks share one 8-GPU node, unlike a TPU VM, which is atomic.
- **A peer that advertises no `region` satisfies no `region` constraint.** An advertised
  attribute the peer omits makes every constraint on that key fail. The CoreWeave backends
  advertise only `device-type` and `device-variant`, so any job carrying a region or zone
  constraint stays local.

That second point bites sub-jobs specifically. `IrisClient.submit` (`iris/client/client.py`)
gives a child job its parent worker's region unless the child names a region itself, which
keeps a child near its data. A GPU sub-job must opt out with fray's `ANY_REGION` sentinel —
`ResourceConfig(..., regions=[ANY_REGION])` — a region-EXISTS marker that suppresses the
inheritance and is then dropped before the wire.

## Only whole root jobs are federated

A peer runs a handed-off job under the same, cluster-invariant job id. A child job's id names
a parent the peer does not have, so a peer can only accept a root. `launch_job` refuses a
non-root job that routes to a peer, naming `--target-cluster` as the remedy.

So a job tree lives entirely on one cluster. A coordinator on `marin` cannot dispatch its
training sub-job to CoreWeave; pin the coordinator instead, and its whole tree runs there.

The identity rule reinforces this. A federated job must carry an accountable user: the peer's
`auth.allowed_submitters` gates on the submitter, and a `local_admin` (CIDR/loopback)
identity is refused before the handoff. In-cluster workers authenticate by network location,
so a job submitted by a worker is `local_admin` — a root submitted by a logged-in user is the
only thing that federates.

## What the peer checks when a handoff arrives

A handoff is authorized as a peer-to-peer delivery, not as a user submission. The peer
verifies the requesting cluster's signed identity and its own `allowed_submitters` against the
principal the parent asserts, then admits the job under the parent's job id.

What it does *not* re-run is the client-freshness gate. That gate makes humans upgrade a stale
`marin-iris` CLI, and the wire client here is the parent controller, which re-encodes the
request from its own stored job state — the submitter's `client_revision_date` is not part of
that state and does not survive the round-trip. The parent already gated the submitter's client
when it accepted the job, and a queued job may wait longer than the freshness window before it
is delivered, so gating on arrival would reject handoffs for a staleness the peer cannot
actually observe.

## What travels with the job

`FederationManager._inline_blobs` (`cluster/federation/manager.py`) carries the workspace
bundle and any offloaded workdir file into the handoff as bytes, because a peer reads its own
bundle store and cannot resolve a content id minted by the parent.

Environment travels too, and it wins over the peer's own defaults: a child inherits its
parent's `env`, and a job's explicit env overrides a cluster's `defaults.task_env`. Passing
`-e MARIN_PREFIX gs://…` to a job bound for CoreWeave therefore delivers a `gs://` path to a
pod holding only S3 credentials.

## Credentials do not travel

Each cluster's task pods carry that cluster's credentials, and only those:

| Cluster | Task pods can read |
| --- | --- |
| `marin` (GCP) | `gs://` via the `iris-worker` service account |
| `cw-*` (CoreWeave) | `s3://` via the `iris-task-env` secret (CoreWeave AI Object Storage) |

There is no cross-cloud identity. A job on CoreWeave cannot read `gs://`, and a job on
`marin` cannot read `s3://marin-us-east-02a` unless it is handed AWS credentials explicitly.
Every artifact a federated job touches must live in the peer's object store.

## Reaching a child's serving endpoint through the parent

A CoreWeave cluster's origin is not world-visible, so a capability URL minted against
`iris-cw-rno2a.oa.dev` cannot be used from outside. A serving endpoint on a child is
reached through the public parent (`iris.oa.dev`) instead, by a blind relay:

- The child mints the capability token itself. When the child has a
  `federation_public_parent` configured, the minted URL is
  `https://<parent>/proxy/t/cluster=<cluster>/<token>/<name>/…`, tagging the child
  cluster beneath the existing capability route and using the parent origin.
- The parent's proxy recognizes the `cluster=<cluster>` discriminator, looks `<cluster>` up in its
  `peers`, and forwards `/proxy/t/<token>/<name>/…` to that child's proxy. It does not
  read or verify the token, and mints no bearer of its own.
- The child validates its own token exactly as for a direct call and serves.

The child is the only auth boundary; the parent is a router. The relay form remains
beneath `/proxy/t/*`, so an edge that already exposes capability URLs needs no
federation-specific rule. The explicit `cluster=` segment cannot be confused with a
minted JWT and never exposes the child's other auth modes. The parent relays only to a
configured peer (an unknown tag is a 404). A root-relative redirect from the served app
will not round-trip today, because the child rewrites `Location` against its own
`/proxy/t/<token>/<name>` prefix without the cluster tag; direct-API endpoints (an
OpenAI-style `/v1/*` server) are unaffected.

## Observing federation

There is no `iris peers` command. Reachability, advertised shapes, and free capacity come
from the `ListPeers` RPC; handoff state lives in the parent's `federated_jobs` table.

```bash
# Which peers are reachable, what each advertises, and how much is free
uv run iris --cluster=marin rpc controller list-peers

# Where a job was handed off, to whom, and under which principal
uv run iris --cluster=marin query \
  "SELECT job_id, peer_id, owner_principal, handoff_state FROM federated_jobs"
```

Each backend in the `list-peers` output carries an `availability` block — the capacity metric
the queue gates on (`{"version": 2, "observation_epoch_ms": …, "amounts": {"h100": 4},
"total_amounts": {"h100": 512}, "held_by_band": [{"band": "PRIORITY_BAND_BATCH", "amounts":
{"h100": 360}}]}`). `amounts` is the free count; `total_amounts` is the matching denominator,
shown on the dashboard's Backends page as free/total meters per device variant; `held_by_band`
is what admitted work holds, per band, which a higher-priority job can reclaim. A backend with
**no** `availability` block supplies no metric and is matched on shape alone, so jobs route to
it without a capacity check.

A queued job and a delivered one are both `pending` on the parent, but they are waiting on
different things, and `job list` says which:

| Pending reason | Meaning |
| --- | --- |
| `Queued for a federation peer to report free capacity` | In the queue, no peer chosen yet |
| `Queued for peer X to report free capacity` | In the queue, pinned to `X` (`--target-cluster`) |
| `Awaiting acceptance by peer X` | Promoted and delivered; awaiting the peer's admission |
| `Handed off to peer X; awaiting first status report` | Admitted; the first sync has not landed |

A job that sits on the first two lines is waiting for capacity, not stuck: compare its device
request against the peer's advertised `amounts` plus the `held_by_band` entries below the job's
own band — those together are what it can reach. A job cancelled while queued never reaches the
peer and terminates as `Cancelled before handoff`.

A federated job's tasks live on the peer and are mirrored back, so `iris job summary` reports
its state from `marin`. Its logs are relayed asynchronously into `marin`'s finelog and lag
behind a log-heavy job; `job summary` is the reliable liveness answer.
