# Configuring infr

infr has one configuration value, resolved once at startup from four layers and
then passed explicitly to the backends and sessions. Every knob is a typed field
on `infr_core::config::Config`; nothing reads the environment behind your back.

A commented starting point lives at [`infr.example.toml`](../infr.example.toml).

## Precedence

Four layers. **Later wins**, and a layer only overrides a field it actually
specifies — leaving `temp` out of your config file does not reset it to the
default, it just leaves the question to the layers below.

| Layer                  | Source                                 | Notes                                                |
| ---------------------- | -------------------------------------- | ---------------------------------------------------- |
| 1. Defaults (lowest)   | `impl Default for Config`              | The shipped behaviour.                               |
| 2. Config file         | TOML (see below)                       | Absent file = no-op. Malformed file = error.         |
| 3. Environment         | `INFR_*`                               | Same names as always — nothing was renamed.          |
| 4. CLI flags (highest) | `--dev`, `--ctx`, `--temp`, …, `--set` | A dedicated flag beats a `--set` for the same field. |

Two consequences worth knowing:

- **Every documented `INFR_*` variable still works.** The campaign that
  introduced `Config` changed _where the value goes_, not what you type. Your
  existing `INFR_PROF_OPS=1 infr bench …` scripts are unaffected.
- **A flag beats an inherited environment variable**, because the CLI layer sits
  above the env layer. `INFR_CTX=32k infr run … --ctx 8192` runs at 8192.

## The config file

### Lookup

The **first existing** file wins. There is no merging across files.

1. `--config <PATH>` — and it is an **error** if that path does not exist.
2. `./infr.toml`
3. `$XDG_CONFIG_HOME/infr/config.toml`, else `~/.config/infr/config.toml`

Finding no file at all is a no-op, never an error.

### Format

TOML. The section path is the struct path, so a key's full name is exactly what
`--set` takes: `[kernels.vulkan] flash_splits = 2` is
`--set kernels.vulkan.flash_splits=2`.

The file speaks the **positive** field names. Where the environment has an
`INFR_NO_*` disable-switch, the config has the thing being enabled, defaulting
to `true`:

```toml
[device]
dev = "Vulkan1"      # same grammar as --dev / INFR_DEV
ctx = "32k"          # the shared size grammar: 8192 / 256k / 50%

[kv]
type_k = "q8_0"
type_v = "q8_0"

[paging]
cache = "8g"         # force the paged expert cache with an 8 GiB budget

[kernels.vulkan]
flash_splits = 2
gemm_warp = false    # NOT `no_gemm_warp` — INFR_NO_GEMM_WARP's field, inverted

[multi]
pipeline = [0, 1]    # or ["Vulkan0", "Vulkan1"]

[serve]
max_tokens_cap = 8192
stats_interval_secs = 10  # throughput line every 10 s; 0 turns it off
```

Value grammars: booleans accept `true`/`false` (and `1`/`0`, `yes`/`no`,
`on`/`off` from `--set`); sizes take the shared `8192` / `256k` / `50%` grammar;
device lists take an array of indices or `VulkanN` strings; an `Option` field is
cleared with `""` or `"none"`.

### Unknown keys warn; wrong types fail

An unrecognized key — or a whole unrecognized section — is **warned about on
stderr and ignored**, with a did-you-mean:

```
[infr] config: unknown key `bogus` (ignored)
[infr] config: unknown key `kernels.vulkan.flash_split` (ignored) — did you mean `kernels.vulkan.flash_splits`?
```

That is deliberate: a config file written for a newer infr must not hard-fail on
an older binary, and removing a knob must not break everyone who has it in their
file. Typo protection comes from the warning line.

A **known** key given a value that does not parse into its type is a hard error:

```
Error: config `device.ctx`: expected a size/count like 8192, 256k or 50% (got "banana")
```

Note the asymmetry with the environment, which is frozen at today's behaviour:
`ctx = "banana"` in a file fails to load, while `INFR_CTX=banana` is silently
ignored and falls back. Five environment keys do reject a bad value loudly, at
startup, on every subcommand: `INFR_SG`, `INFR_SUBMIT_DISPATCHES`,
`INFR_PIPELINE`, `INFR_TENSOR_PARALLEL`, `INFR_EXPERT_PARALLEL`.

### Diagnostics announce themselves

If the **file** turns on a `prof.*` or `debug.*` knob, infr prints one line at
startup naming the file and the fields:

```
[infr] config: /home/you/.config/infr/config.toml enabled diagnostics: prof.ops
```

So "why is my server printing timings" is answerable from the command line, even
when the cause is a global config file you forgot about.

## `--set`

Most of the 155 knobs have no dedicated flag. `--set <config.path>=<value>`
reaches all of them, using the same path grammar as the TOML file:

```bash
infr bench "$M" -p 512 -n 0 --set kernels.vulkan.flash_splits=2
infr run "$M" "hi" --set kv.type_k=q8_0 --set kv.type_v=q8_0
```

- The path is the **config path**, never the `INFR_*` name. They are not 1:1:
  `INFR_NO_GEMM_WARP` is `kernels.vulkan.gemm_warp=false`, `INFR_NO_GEMV_REG`
  and `INFR_GEMV_VARIANT` are both `kernels.vulkan.gemv.variant`, and
  `INFR_MMV_MW` is tri-state.
- An **unknown** path is a hard error with a did-you-mean (unlike the file layer
  — you typed it for this run, so silently ignoring it would give you a wrong
  answer with no second chance to notice):
  ```
  Error: unknown config path `kernels.vulkan.flash_split` — did you mean `kernels.vulkan.flash_splits`?
  ```
- The **same path twice** is an error, not a silent last-wins:
  ```
  Error: `--set kv.slots=` given more than once
  ```
- `--set` is **additive** to the dedicated flags, and loses to them. Passing
  both prints a warning naming the field:
  ```
  $ infr run "$M" "hi" --ctx 4096 --set device.ctx=8192
  [infr] config: `--set device.ctx=8192` ignored — the dedicated flag for `device.ctx` wins
  ```

### The dedicated flags

`--config` and `--set` are global. The rest are on `run` / `serve` (device flags
also on `bench`); `infr <cmd> --help` is the authority.

| Flag                     | Config path         | Env             |
| ------------------------ | ------------------- | --------------- |
| `--dev`                  | `device.dev`        | `INFR_DEV`      |
| `--ctx`                  | `device.ctx`        | `INFR_CTX`      |
| `-u` / `--ubatch`        | `device.ubatch`     | `INFR_UBATCH`   |
| `-t` / `--threads`       | `device.threads`    | —               |
| `--temp`                 | `sampling.temp`     | `INFR_TEMP`     |
| `--top-k`                | `sampling.top_k`    | `INFR_TOP_K`    |
| `--top-p`                | `sampling.top_p`    | `INFR_TOP_P`    |
| `--seed`                 | `sampling.seed`     | `INFR_SEED`     |
| `--max-new`              | `sampling.max_new`  | `INFR_MAX_NEW`  |
| `--no-think` / `--think` | `sampling.no_think` | `INFR_NO_THINK` |

`device.threads` has no `INFR_*` twin — it is published as `RAYON_NUM_THREADS`,
because rayon's global pool has no other input.

## What is tunable

The authority is
[`crates/infr-core/src/config/manifest.rs`](../crates/infr-core/src/config/manifest.rs):
every `INFR_*` key, the config path it lands on, and its value grammar, in one
table that the tests check against the tree. `Config::all_paths()` is the full
path list. What follows is the shape plus the knobs people actually reach for.

**`[device]`** — which GPU (`dev`), how much context (`ctx`), the prefill
micro-batch (`ubatch`, `ubatch_parallel`), CPU `threads`, and two low-level
device knobs (`submit_dispatches` for the iGPU submit splitter, `subgroup_pref`
to force subgroup 16 or 32).

**`[sampling]`** — `temp` (0 = greedy), `top_k`, `top_p`, `seed`, `max_new`,
`ignore_eos`, `no_think`. **Provenance matters here**: `infr run` / `infr serve`
fill `temp` / `top_k` / `top_p` / `max_new` from the model's own recommended
sampling (an arch-family table plus any `generation_config.json` beside the
model) only for knobs that **no layer specified**. Putting `temp` in your config
file pins it and suppresses that fallback. The `Config` defaults, which library
callers get, are greedy: `temp = 0.0`, `top_k = 20`, `top_p = 0.95`,
`max_new = 2048`. On `serve` these are the server defaults — a per-request
OpenAI `temperature`/`top_p` still overrides them.

**`[kv]`** — cache element format (`type_k` / `type_v`, plus the legacy
`force_q8` alias), prefix-cache `slots`, the sliding-window `ring`, and the
host-overflow trio (`overflow`, `overflow_vram_mb`, `overflow_reserve_mb`) that
spills KV to system RAM when VRAM runs out.

**`[paging]`** — the MoE expert cache and dense layer streaming: `cache` sizes
the paged VRAM budget (and forces paging even when the weights would have fit),
`ring` overrides the upload staging ring, `stats` prints per-pool
hit/miss/eviction counts.

**`[kernels]`** — two backend-independent graph-shape gates (`qkv_fuse`,
`gated_rmsnorm`) plus one sub-section per backend. Everything under them is a
kernel **tier** override: the engine picks the best tier the device supports,
and these exist to force one off when bisecting a correctness or perf problem.

- **`[kernels.vulkan]`** (the biggest section — 62 of the 155 keys): capability
  masks (`coopmat`, `f16`, `i8_dot`, `coopmat_8x8`, `i8_coopmat`), GEMM/GEMV
  tiers (`gemm_warp`, `mmq`, `mmv`, `mrow`, `moe_small_m`, the
  `[kernels.vulkan.gemv]` sub-table), attention (`flash_warp`, `flash_splits`,
  `flash_min_rows`, `pv_splits`), DeltaNet (`dn_chunk_scan`, `dn_chunk`,
  `dn_split`), and the plumbing switches (`push_desc`, `pipeline_cache_disk`,
  `no_replay`, `no_vram_guard`, the BDA chunk caps). Note `f16 = false` also
  disables coopmat regardless of `coopmat`, matching `INFR_NO_F16`'s effect
  today.
- **`[kernels.metal]`** — the Apple backend's native/CMM/RT kernel families per
  dtype, plus `deltanet`, `moe` and `pipeline_cache` (persist the compiled
  `MTLComputePipelineState`s as an `MTLBinaryArchive` under `~/.cache/infr`, so
  a launch does not re-run the driver's AIR → GPU-ISA back end for every kernel;
  the MSL → AIR front end is not cacheable and still runs every launch). Like
  `kernels.cpu.reference` it has no `INFR_*` twin.
- **`[kernels.cpu]`** — `spin` (spin-pool idle ceiling), `spinpool`,
  `repack_mb`, and `reference` (the bit-reference kernel path, which has no
  `INFR_*` twin and never had one).

**`[spec]`** — MTP / speculative decode: `mtp`, `k`, `decode_chain`, `draft` (a
draft-model path), the GPU-side sampling steps (`gpu_argmax`, `gpu_sample`,
`gpu_embed`, …). MTP is currently parked; see the README.

**`[multi]`** — multi-GPU splits: `pipeline`, `tensor_parallel` (both need ≥ 2
devices) and `expert_parallel` (≥ 1); too few devices is a hard error. The three
`*_p2p` flags choose GPU-to-GPU transport (`true`, the default) over staging
through host RAM.

**`[prof]`** — every key is `INFR_PROF_*`, and every one is settable three ways
(env, `--set prof.<name>=…`, or a `[prof]` section in the file):

| knob                | what it does                                                                                                            |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `ops`               | per-op device profiling on EVERY backend — vulkan, metal, cpu                                                           |
| `op_shapes`         | itemize per-op labels by shape rather than folding a kind into one row (vulkan)                                         |
| `stages`            | host-side stage timing: throughput, decode setup-vs-execute, prefill build/compile/execute, MTP verify, diffusion steps |
| `vram`              | log live VRAM in use after weight load                                                                                  |
| `out`               | ALSO write the exit report as JSON to this path                                                                         |
| `diffusion_trace`   | per-step schedule/entropy trace for diffusion models                                                                    |
| `metal_device_time` | `off` / `flush` / `counters` — how Metal obtains per-op device time                                                     |
| `metal_debug`       | extra Metal profiler output                                                                                             |

Two consolidations are worth knowing if you have old scripts. Per-op profiling
answered to a different knob on each backend (`INFR_PROF2` on vulkan,
`INFR_PROF_OPS` on cpu, `INFR_METAL_PROFILE` on metal) — it is all `ops` now.
And host-side stage timing was five knobs, one per pipeline (`INFR_PROF`,
`INFR_PROF_DEC`, `INFR_PROF_PF`, `INFR_MTP_TIME`, `INFR_DIFFUSION_TIME`) — it is
all `stages`. Old spellings were dropped cleanly and are simply no longer read.

**`[serve]`** — `api_key` (bearer token; an **empty** value means no auth — it
gates `/v1/chat/completions` and `/v1/models`, never `/health`),
`max_tokens_cap`, `request_timeout_secs` (per-request wall-clock deadline in
seconds; `0`, the default, means no deadline — a deadline truncates a legitimate
slow reply, so it is opt-in), and `stats_interval_secs` (`INFR_SERVE_STATS_SECS`
— how often the server logs its throughput line, default `5`; `0` switches the
line off). Per-request sampling is not here — it stays on the request.

The throughput line is **activity-only**: an interval in which nothing happened
emits nothing, so an idle server leaves a clean log and there is no heartbeat to
mistake for load. Its `prefill_tps`/`decode_tps` are the whole server's tokens
divided by the WALL time of that one interval — not cumulative, and not the same
number as the per-request `prefill_tps`/`decode_tps` on the `request done` line,
which are that one request's own speeds (`prompt_tokens / TTFT` and
`gen_tokens / (total - TTFT)`). Request logging carries **counts only, never
prompt text**.

**`[hub]`** — model acquisition (`infr pull`, and the auto-pull `infr run` /
`infr serve` do when a model is missing). One knob: `pull_jobs`
(`INFR_PULL_JOBS`, default `8`) — how many CONNECTIONS one model's download may
use at once. It is the CONNECTION that is slow, not the link: measured against
the same CDN objects, one connection sustained 8.8 MB/s and five sustained 78.7
MB/s — 15.7 MB/s each, so the per-connection rate rose rather than fell.

What the connections are spent on depends on the model, and the setting does not
have to say: a split model is `-NNNNN-of-MMMMM` shards fetched one file per
connection, while a model shipped as one file is split into byte ranges instead
and reassembled (on `unsloth/DeepSeek-V3.2-GGUF`'s single 161 GB `UD-TQ1_0`,
same 60-second window: 11.4 MB/s at `1`, 80.5 MB/s at the default `8`; the whole
161 GB pull ran at 80.0 MB/s end to end). One number covers both because a bound
per axis would multiply — 8 files x 8 ranges is 64 sockets on a repo whose file
count is the publisher's choice (`DeepSeek-V3.2-REAP` ships 236).

Ranges need the server to support them; one that does not (no `Accept-Ranges`,
or a `200` where a `206` was asked for) falls back to a single stream, and a
file of 64 MiB or less is never split at all. `0` and `1` both mean strictly one
connection — the setting for a metered link, or for a proxy that objects to
several connections from one client.

**`[debug]`** — poison/barrier/dump switches: `coopmat` (print the enumerated
and chosen coopmat shapes — useful on Intel Arc), `bda_chunk`, `wide_dispatch`,
`chat`, `moe_counts`, `moe_counts_dump`, `poison_uninit`, `no_barrier`,
`full_barrier`.

## The `INFR_*` keys that are deliberately NOT config

Four keys keep reading the environment directly, for reasons a runtime `Config`
cannot fix:

| Key                        | Why                                                                                                                                         |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `INFR_PROFILE`             | **Build-time** input: read by `build.rs` in core/cpu/gguf/llama/vulkan to set `cfg(infr_profile)`. No runtime value exists when it is read. |
| `INFR_TEST_GGUF`           | Test fixture: points `infr-gguf`'s tests at a `.gguf` on disk.                                                                              |
| `INFR_TEST_MODEL`          | Test fixture: overrides the HF-cache lookup for the model-backed tests.                                                                     |
| `INFR_LLAMA_DIFFUSION_CLI` | Dev fixture: points `infr compare` at a `llama-diffusion-cli` binary on disk.                                                               |

Two related notes. `INFR_DIFFUSION_VISUAL` is also not a `Config` field — since
it only steers CLI presentation it became the plain flag
`infr run --diffusion-visual`, whose clap `env =` fallback keeps the old
spelling working. And `INFR_CPU` / `INFR_METAL` are **dead**: they were removed
as backend selectors with no aliases. Use `--dev cpu` / `--dev metal`, or
`INFR_DEV=cpu` / `INFR_DEV=metal`.

## Library callers

`Config` is a value, not a global. Build one and hand it out:

```rust
use infr_core::config::{Config, ConfigOverrides};
use std::sync::Arc;

// The full four-layer resolve (file + env + the CLI overrides you pass).
let cfg = Arc::new(Config::load(&ConfigOverrides::default())?);

// Or construct exactly what you want — no environment, no file, no ordering hazard.
let cfg = Arc::new(Config { kv: infr_core::config::KvCfg { slots: 8, ..Default::default() },
                            ..Default::default() });
```

Backends take it at construction (`VulkanBackend::new_with(cfg)`,
`Backend::new_with(device_id, cfg)`, `CpuBackend::new_with(cfg)`), and
`Config::load_from_env()` is the defaults-plus-environment fold for a caller
that wants the environment honoured but no config file.

The migration history — the layer machinery, the per-knob polarity tables, and
the slice-by-slice record — is in [`config-plan.md`](config-plan.md).
