# Task States Reference

This document describes the `TaskState` enum, the state machine governing task
lifecycle, retry semantics, and how states appear in the dashboard.

A **task** is the unit of execution in Iris. Each job expands into one or more
tasks (controlled by `replicas`). Tasks are independently scheduled, retried,
and tracked. Job state is derived from task state counts -- there is no
independent job state machine.

## State Diagram

```
                          +-----------+
                          |  PENDING  |<-----------------+
                          +-----+-----+                  |
                                |                        |
                          dispatch to worker              |
                                |                        |
                                v                        |
                          +-----------+                  |
                          | ASSIGNED  |                  |
                          +-----+-----+                  |
                                |                        |
                        worker starts task               |
                                |                        |
                                v                        |
                          +-----------+                  |
                          | BUILDING  |                  |
                          +-----+-----+                  |
                                |                        |
                        build completes                  |
                                |                        |
                                v                        |
                          +-----------+                  |
                          |  RUNNING  |                  |
                          +-----+-----+                  |
                                |                        |
            +-------------------+-------------------+    |
            |                   |                   |    |
            v                   v                   v    |
      +-----------+       +-----------+     +------------+
      | SUCCEEDED |       |  FAILED   |---->| retry      |
      +-----------+       +-----------+     +------------+
                                |                  ^
                                | exhausted        |
                                v                  |
                          (terminal)               |
                                                   |
                          +-----------+            |
                          |WORKER_FAIL|------------+
                          +-----------+
                                |
                                | exhausted
                                v
                          (terminal)

                          +-----------+            |
                          | PREEMPTED |            |
                          +-----------+            |
                                ^                  |
                                | exhausted        |
                                |                  |
                          +-----------+            |
                          | preempt   |------------+
                          | (ctrl)    |
                          +-----------+

      Other terminal states: KILLED, UNSCHEDULABLE (never retried)
```

## State Table

| State | Proto Value | Terminal | Retriable | Set By | Dashboard Display |
|---|---|---|---|---|---|
| `UNSPECIFIED` | 0 | -- | -- | Default zero value; never used in practice | `unspecified` (grey) |
| `PENDING` | 1 | No | -- | Job submission (`_on_job_submitted`), retry requeue (`_requeue_task`) | `pending` (amber) |
| `ASSIGNED` | 9 | No | -- | Scheduler dispatch (`_on_task_assigned` / `create_attempt`) | `assigned` (orange) |
| `BUILDING` | 2 | No | -- | Worker Reconcile observation; worker sets this during bundle download and dependency sync | `building` (purple) |
| `RUNNING` | 3 | No | -- | Worker Reconcile observation; worker sets this when user command starts | `running` (blue) |
| `SUCCEEDED` | 4 | Yes | No | Worker Reconcile observation; task exited with code 0 | `succeeded` (green) |
| `FAILED` | 5 | Yes | Yes | Worker Reconcile observation; task exited with non-zero code | `failed` (red) |
| `KILLED` | 6 | Yes | No | Controller: job cancellation (`_on_job_cancelled`), job failure cascade (`_mark_remaining_tasks_killed`), per-task timeout | `killed` (grey) |
| `WORKER_FAILED` | 7 | Yes | Yes | Controller: worker death cascade (`ops.worker.fail`) | `worker_failed` (purple) |
| `UNSCHEDULABLE` | 8 | Yes | No | Controller: scheduling timeout expired (`apply_terminal_decisions_batch`) | `unschedulable` (red) |
| `PREEMPTED` | 10 | Yes | Yes | Controller: priority preemption with budget exhausted (`apply_terminal_decisions_batch`); K8s backend: control-plane disruption of the pod | `preempted` (orange) |
| `COSCHED_FAILED` | 11 | Yes | No | Controller: coscheduled sibling cascade (`_terminate_coscheduled_siblings`) | `cosched_failed` (red) |


## State Transitions in Detail

### PENDING

The initial state for every task. Set in two contexts:

1. **Job submission**: `_on_job_submitted` calls `expand_job_to_tasks`, which
   creates `ControllerTask` objects with `state=TASK_STATE_PENDING`. Tasks are
   enqueued into the priority-sorted scheduling queue.

2. **Retry requeue**: `_requeue_task` resets `task.state` to `TASK_STATE_PENDING`
   and re-inserts the task into the scheduling queue. This happens after a
   retriable `FAILED` or `WORKER_FAILED` when retry budget remains.

### ASSIGNED

Set by `_on_task_assigned` after the scheduler selects a worker and commits
resources. `create_attempt` creates a new `ControllerTaskAttempt` in
`TASK_STATE_ASSIGNED` state. The task is now bound to a specific worker and
consuming its resources.

The worker has not yet acknowledged the task -- it will receive the dispatch
in the next controller Reconcile cycle (the `DesiredAttempt.run` intent
carries the `AttemptSpec.request` payload on this one tick).

### BUILDING

Reported by the worker as a Reconcile observation. The worker transitions
internally:

- `PENDING -> BUILDING` when bundle download starts (`task_attempt.py:433`)
- Later, `BUILDING` again when dependency sync starts (`task_attempt.py:549`)

The controller processes this transition in `apply_task_updates`. Note: if the
worker reports `PENDING`, the controller ignores it to prevent regressing an
`ASSIGNED` task and confusing the building-count backpressure window.

### RUNNING

Reported by the worker via Reconcile after the user command starts executing
(`task_attempt.py:570`). The controller records `started_at` on the attempt.

### SUCCEEDED

Reported by the worker via Reconcile when the task process exits with code 0.
The controller sets `exit_code=0`, `finished_at`, and marks the task terminal.
No retry logic applies.

### FAILED

Reported by the worker via Reconcile when the task process exits with a
non-zero code. Triggers retry evaluation:

1. `handle_attempt_result` calls `_handle_failure`, which increments
   `failure_count` and compares against `max_retries_failure`.
2. If `failure_count <= max_retries_failure`: returns `SHOULD_RETRY`. The caller
   (`_on_task_state_changed`) calls `_requeue_task`, which resets state to
   `PENDING` and re-enqueues the task. Resources are released from the current
   worker.
3. If `failure_count > max_retries_failure`: returns `EXCEEDED_RETRY_LIMIT`. The
   task remains in `FAILED` state and is terminal. `error` and `exit_code` are
   recorded.

### KILLED

Set by the controller in three scenarios:

1. **User cancellation**: `_on_job_cancelled` iterates non-terminal tasks and
   transitions each to `KILLED`. Tasks with workers assigned are queued for
   kill RPCs.

2. **Job failure cascade**: When a job exceeds `max_task_failures`,
   `_finalize_job_state` calls `_mark_remaining_tasks_killed` to terminate all
   surviving tasks.

3. **Parent job termination**: `_cancel_child_jobs` recursively cancels child
   jobs when a parent reaches a terminal state (except `SUCCEEDED`).

`KILLED` is always terminal and never retried.

### WORKER_FAILED

Set by the controller when a worker dies. Worker failure (`ops.worker.fail`)
closes all tasks on the dead worker, emitting a `TASK_STATE_WORKER_FAILED`
transition for each non-terminal task.

Retry evaluation uses the preemption budget:

1. `_handle_failure` increments `preemption_count` and compares against
   `max_retries_preemption` (default: 100).
2. If budget remains: `SHOULD_RETRY` -- task is requeued to `PENDING`.
3. If exhausted: `EXCEEDED_RETRY_LIMIT` -- task stays in `WORKER_FAILED`
   and is terminal.

**Coscheduled jobs**: When a task in a coscheduled (gang-scheduled) job fails
terminally, `_terminate_coscheduled_siblings` transitions all running siblings
to `COSCHED_FAILED` (always terminal — see below). This prevents other hosts
from hanging on collective operations.

### PREEMPTED

Set by the controller when a higher-priority task evicts a lower-priority
running task. The preemption pass (`apply_preemptions`) selects victims
from lower priority bands and submits them to `apply_terminal_decisions_batch`.

Also reported by the K8s backend when the cluster control plane disrupts an
attempt's pod, marked by a `DisruptionTarget` or Kueue `TerminationTarget` pod
condition (scheduler preemption, Kueue workload eviction, node drain, API
eviction). Kueue deletes the pod it evicts, so the condition is readable only
while the pod terminates; the backend caches it and reports it as the attempt's
`terminal_reason` once the pod is gone. A pod that vanishes with no disruption
ever observed is reported `WORKER_FAILED` instead — same preemption budget,
without claiming a cause. These updates run through `apply_one_transition` and
do cascade coscheduled siblings.

Retry evaluation uses `_resolve_task_failure_state` with the preemption budget:

1. **ASSIGNED tasks**: always retry to PENDING regardless of budget (the task
   never started executing, so preemption is free).
2. **BUILDING or RUNNING tasks**: `preemption_count` is incremented and compared
   against `max_retries_preemption`.
   - If `preemption_count <= max_retries_preemption`: task is requeued to
     `PENDING` for retry. The current attempt is marked `PREEMPTED`.
   - If `preemption_count > max_retries_preemption`: task state is set to
     `PREEMPTED` (terminal). Both the attempt and the task are `PREEMPTED`.

`PREEMPTED` is in both `TERMINAL_TASK_STATES` and `FAILURE_TASK_STATES`.
When a coscheduled task becomes terminally `PREEMPTED`, the job state is
recomputed. If all tasks in the job are terminal, the batches.py
`_finalize_terminal_job` orchestrator
kills any remaining non-terminal tasks and cascades to child jobs. Note that
unlike `WORKER_FAILED` reported via Reconcile, PREEMPT decisions do not
directly cascade coscheduled siblings — the cascade only occurs through job
finalization.

### UNSCHEDULABLE

Set by the controller's scheduling loop when a task's scheduling deadline
expires (via `apply_terminal_decisions_batch`). The deadline is
derived from the job's `scheduling_timeout` field.

`UNSCHEDULABLE` is always terminal. If any task becomes unschedulable, the
entire job transitions to `JOB_STATE_UNSCHEDULABLE` and all remaining tasks
are killed.

### COSCHED_FAILED

Set by the controller in `_terminate_coscheduled_siblings` when a coscheduled
(gang-scheduled) sibling task hits its terminal failure budget. The cascaded
task itself was healthy, so the state is distinct from `WORKER_FAILED` (which
implies a real worker death and bumps the preemption counter) and from
`KILLED` (which implies operator-initiated cancellation).

`COSCHED_FAILED` is always terminal — `task_is_finished` returns `True`
unconditionally, with no counter math involved. At the job level, it rolls up
into `JOB_STATE_WORKER_FAILED` alongside `WORKER_FAILED` and `PREEMPTED`,
because the triggering event was a worker-failure pattern on the originating
task.

## Retry Semantics

Iris maintains two independent retry budgets per task:

| Budget | Counter | Limit Field | Default | Trigger States |
|---|---|---|---|---|
| Failure | `failure_count` | `max_retries_failure` | 0 (no retries) | `FAILED` |
| Preemption | `preemption_count` | `max_retries_preemption` | 100 | `WORKER_FAILED`, `PREEMPTED` |

### Retry flow

1. Worker reports terminal state via Reconcile.
2. `handle_attempt_result` delegates to `_handle_failure`.
3. The appropriate counter is incremented.
4. If `counter <= limit`: `TaskTransitionResult.SHOULD_RETRY`.
   - `_on_task_state_changed` calls `_requeue_task`.
   - Task state is reset to `PENDING`. A new attempt will be created when the
     scheduler re-dispatches.
   - Worker resources are released via `_cleanup_task_resources`.
5. If `counter > limit`: `TaskTransitionResult.EXCEEDED_RETRY_LIMIT`.
   - Task remains in its failure state and is terminal.
   - `is_finished()` returns `True`.
   - The job's `_compute_job_state` may trigger a job-level state change
     (e.g., `JOB_STATE_FAILED` if `max_task_failures` is exceeded).

### What counts toward job failure

Only `TASK_STATE_FAILED` counts toward the job's `max_task_failures` threshold.
Worker failures and preemptions do not count. This means a job can survive
unlimited preemptions as long as the per-task preemption budget is not
exhausted. `TASK_STATE_PREEMPTED` and `TASK_STATE_WORKER_FAILED` are grouped
together for job state derivation: if all tasks are terminal and any are in
one of these states, the job becomes `JOB_STATE_WORKER_FAILED`.

### States that are never retried

- `SUCCEEDED`: task completed successfully
- `KILLED`: explicit termination by user or cascade
- `UNSCHEDULABLE`: scheduling timeout expired
- `PREEMPTED`: only when preemption budget is exhausted (otherwise retried as `PENDING`)

## Terminal State Summary

A task is considered finished (`is_finished() == True`) when:

| State | Condition |
|---|---|
| `SUCCEEDED` | Always finished |
| `KILLED` | Always finished |
| `UNSCHEDULABLE` | Always finished |
| `FAILED` | Finished when `failure_count > max_retries_failure` |
| `WORKER_FAILED` | Finished when `preemption_count > max_retries_preemption` |
| `PREEMPTED` | Finished when `preemption_count > max_retries_preemption` |

The distinction matters: a task in `FAILED` state with retry budget remaining
is in a terminal state at the attempt level but is not finished at the task
level. `can_be_scheduled()` returns `True` for such tasks.

## Dashboard Display

The dashboard uses `stateToName()` from `shared/utils.js` to convert proto enum
strings (e.g., `TASK_STATE_RUNNING`) to lowercase display names by stripping the
`TASK_STATE_` prefix. Each name maps to a CSS class `status-{name}`:

| Display Name | CSS Class | Color |
|---|---|---|
| `pending` | `.status-pending` | Amber (#9a6700) |
| `assigned` | `.status-assigned` | Orange (#bc4c00) |
| `building` | `.status-building` | Purple (#8250df) |
| `running` | `.status-running` | Blue (#0969da) |
| `succeeded` | `.status-succeeded` | Green (#1a7f37) |
| `failed` | `.status-failed` | Red (#cf222e) |
| `killed` | `.status-killed` | Grey (#57606a) |
| `worker_failed` | `.status-worker_failed` | Purple (#8250df) |
| `unschedulable` | `.status-unschedulable` | Red (#cf222e) |
| `preempted` | `.status-preempted` | Orange (#bc4c00) |

The job detail page shows per-task attempt history. Each attempt has its own
state badge, and worker failures are annotated with "(worker failure)" in the
attempt rows.

Pending tasks display a `pending_reason` diagnostic below the state badge when
the controller can identify why the task cannot be scheduled (e.g., no workers
match constraints).

## Job State Derivation

Job state is computed from task state counts in `_compute_job_state()`:

1. **SUCCEEDED**: All tasks are in `TASK_STATE_SUCCEEDED`.
2. **FAILED**: Count of `TASK_STATE_FAILED` tasks exceeds `max_task_failures`.
3. **UNSCHEDULABLE**: Any task is `TASK_STATE_UNSCHEDULABLE`.
4. **KILLED**: Any task is `TASK_STATE_KILLED` (and job is not already terminal).
5. **RUNNING**: Any task is `ASSIGNED`, `BUILDING`, or `RUNNING`.
6. **PENDING**: Default (no tasks have started).

The ordering matters -- earlier rules take priority. A job with one succeeded
task and one failed task (beyond tolerance) is `FAILED`, not `RUNNING`.
