# Agentic single-node benchmarks

**MVP / experimental.** Nothing in this directory is an official InferenceX
benchmark. Results are not published on https://inferencex.com and are not
intended to be cited.

These launchers exist to develop and validate the agentic-coding scenario
type before it is promoted to first-class status. The scripts themselves
are best-effort and mainly serve as a reference implementation of how the
plumbing (env vars, scenario routing, result paths) should work. Specific
models and configs may be broken at any given time — multi-node in
particular is not yet first-class.

## DRAM KV offload memory policy

Agentic scenarios use `kv-offloading` for the resource tier and
`kv-offload-backend` for the backend implementation. `kv-offloading` is
currently either `none` or `dram`; when it is `dram`, the backend must be set:

```yaml
- dram-utilization: 0.80
  search-space:
  - { tp: 4, kv-offloading: dram, kv-offload-backend: { name: vllm-native }, conc-list: [16, 32] }
  - { tp: 8, kv-offloading: none, conc-list: [16, 32] }
```

`kv-offload-backend.name` is required when offloading is enabled. Its
`version` is optional: omit it for framework-native implementations without an
independent release, and include it for separately versioned packages such as
LMCache or Mooncake.

Agentic matrix generation uses a 3600-second default duration. Reusable
workflow callers can still override the `duration` input explicitly.

Agentic master configs must use an exact `cluster:<name>` runner label so every
search-space point lands on the same hardware fleet. Machine-level host memory
is declared once in the `configs/runners.yaml` `hardware` entry matching
that runner label:

```yaml
hardware:
  cluster:b300-nv:
    available-cpu-dram-mib: 2964436
    gpus-per-node: 8
```

The matrix generator combines the master config utilization with runner
hardware metadata and emits the aggregate budget as
`floor(min(available MiB, 2,861,022) * 1,048,576 * utilization * tp / gpus-per-node / 1,000,000,000)`.
The `2,861,022 MiB` cap is the 3 TB decimal DRAM limit. For example, TP4 in
an eight-GPU B300 search at 80% utilization receives 1,199 GB while TP8
receives 2,399 GB.

Benchmark scripts must consume `TOTAL_CPU_DRAM_GB`; they must not replace it
with model-specific constants. Backends with per-rank or per-pool settings must
divide this aggregate budget accordingly. DSv4 SGLang is the exception because
it exposes only `--hicache-ratio`; its empirically measured ratios are capped
in the model launchers to remain below the generated byte budget.
