<!--Copyright 2026 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.

⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->

# Continuous batching architecture

Traditional batching processes a fixed group of requests together and waits for the slowest one to finish before starting the next group. This causes the GPU to idle between batches.

With continuous batching, at every generation step, the scheduler checks for finished requests and replaces them immediately with waiting ones. Short requests are kicked out as soon as they're done, and the GPU stays occupied the entire time. The result is significantly higher throughput and lower average latency.

## Request lifecycle

A request moves through four states from submission to completion.

```text
       WAIT IN QUEUE          LOAD PROMPT INTO KV       STREAM OUTPUT           DONE
    ┌────────────────┐    ┌─────────────────────┐    ┌─────────────────┐    ┌────────┐
    │    PENDING     │───▶│     PREFILLING      │───▶│    DECODING     │───▶│FINISHED│
    │  ○ ○ ○ · · ·   │    │ ████████░░░░░░░░░░  │    │  → → → → · ·    │    │   ✓    │
    └────────────────┘    └─────────────────────┘    └─────────────────┘    └────────┘
         others ahead            prompt chunks              +1 token/step
```

1. Pending — the request is queued and waiting to be scheduled.
2. Prefilling — prompt tokens are processed in a forward pass.
3. Decoding — output tokens are generated one at a time.
4. Finished — generation is complete and the result is available.

A request is moved from pending to prefilling when the scheduler finds enough token budget and cache space to admit the request. If the prompt is too long to fit in a single step, the scheduler splits it across multiple forward passes (chunk prefill).

## Chunked prefill

Processing a long prompt in a single step is expensive because it blocks other requests from generating. Chunked prefill solves this by splitting large prefills across multiple steps.

```text
Blocking (long prefill blocks the batch)
  Step:   1        2        3        4
  Req A  [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓]   ← one huge prefill
  Req B  ················ wait ················
  Req C  ················ wait ················

Chunked (same prompt split and interleave with decode)
  Step:   1          2          3          4
  Req A  [▓▓ pre]    [▓▓ pre]   [▓▓ pre]   [→ dec]
  Req B  [→ dec]     [→ dec]    [▓▓ pre]   [→ dec]
  Req C  [→ dec]     · idle ·   [→ dec]    · idle ·
```

When a new request's prompt exceeds the available token budget (set by `max_batch_tokens` in [`ContinuousBatchingConfig`]), the scheduler processes as many tokens as possible and holds the rest. On subsequent steps, it continues from where it left off, interleaving prefill work with ongoing decode steps. This keeps the batch productive and reduces time-to-first-token for other requests.

## Scheduler

The scheduler decides which requests join each forward pass based on two budgets and a request cap.

- Token budget — the maximum number of query tokens processed in a single forward pass, set by `max_batch_tokens` in [`ContinuousBatchingConfig`].
- Cache budget — the total number of KV pages that can be read in a single pass, which is bounded by the total cache size.
- Request cap — the maximum number of requests in a single forward pass, set by `max_requests_per_batch`. The logits tensor scales with the vocabulary size, so this cap bounds peak memory for large-vocabulary models.

### FIFO

[`FIFOScheduler`] is the default scheduler. It fills the batch in priority order. Active decode requests are processed first, then active prefill requests, and then new waiting requests in arrival order. This maximizes throughput by pushing existing requests through before pulling in new ones.

### PrefillFirst

[`PrefillFirstScheduler`] completes chunked prefill operations before resuming decode. This reduces fragmentation for workloads dominated by long prompts.

Set the scheduler in [`ContinuousBatchingConfig`].

```py
cb_config = ContinuousBatchingConfig(scheduler_type="prefill_first")
```

## Memory management

The KV cache is paged so requests of different lengths can share a fixed memory pool without fragmentation. The cache divides memory into fixed-size blocks, and each request holds a list of block IDs that the attention kernel reads from and writes to.

```text
Paged KV cache (num_blocks × block_size tokens per block)

    0   1   2   3   4   5   6   7   8   9   …
  ┌───┬───┬───┬───┬───┬───┬───┬───┬───┬───┐
  │ B │ · │ A │ B │ · │ A │ · │ A │ · │ · │
  └───┴───┴───┴───┴───┴───┴───┴───┴───┴───┘
  A  request A: 2, 5, 7   B  request B: 0, 3   ·  free: 1, 4, 6, 8, 9, …
```

A page holds the key/value state for one token in one layer. A block is a span of `block_size` pages (default 256) and is the unit of allocation. Blocks are allocated per layer group. Layers within a group share a block ID, which keeps bookkeeping uniform for mixed-attention models.

The cache reserves two extra blocks on top of `num_blocks` for padding tokens to read from and write to, plus a sentinel index for sliding window groups. This reservation requires `block_size` to be at least 4, so the manager rejects smaller values.

### Cache sizing

The manager infers the number of blocks at startup from free GPU memory. The manager solves an equation that accounts for KV tensors, attention masks, activations, and bookkeeping indices, then sizes the pool to fit inside `max_memory_percent` (default 0.9) of the available memory.

You can pin the values explicitly in [`ContinuousBatchingConfig`].

```py
cb_config = ContinuousBatchingConfig(
    block_size=256,
    num_blocks=4096,
    max_batch_tokens=512,
    max_memory_percent=0.8,
)
```

### Admission

Before a request joins the batch, the scheduler checks that enough free blocks exist for every layer group. If any group would fall short, the request is rejected and nothing is allocated.

To avoid filling the cache until [offloading](#offloading) is the only option, the scheduler enforces a `safety_margin`. Once free blocks fall below `safety_margin * num_blocks`, new prefills are held back and only active decodes continue. The FIFO scheduler defaults to `0.15` (15% of blocks held in reserve), while `prefill_first` defaults to `0.0`. Set `safety_margin` in [`ContinuousBatchingConfig`] to tune it, where `0` disables the margin.

### Prefix caching

When two requests share a prompt prefix, they can share the blocks that hold the KV for that prefix. Each completed block is content-hashed. A later request with a matching prefix reuses the block and skips the prefill for those tokens. Shared blocks are reference-counted and only return to the free pool once every request using them has finished.

Prefix caching is enabled by default and is active only when a model has exclusively full-attention layers. Set `allow_block_sharing=False` in [`ContinuousBatchingConfig`] for workloads with short prompts and long generations, where the bookkeeping outweighs the savings.

### Eviction

Freed blocks aren't wiped and they stay "initialized" with their content and hash intact so a later prefix match can reuse them. When the uninitialized pool runs low, the manager lazily demotes the most recent initialized block back to uninitialized, dropping their hash. The prefix cache is a best-effort layer that the allocator discards to keep serving new requests.

### Soft reset

A soft reset is the fallback path for [offloading](#offloading). When the manager can't copy a request's KV cache to the CPU swap pool, because CPU offloading is disabled or the pool is full, it soft resets the request instead. It appends the request's generated tokens to its original prompt, frees its cache blocks, and adds it back to the queue. The [Offloading](#offloading) section covers which requests are chosen.

When the cache has space again, the request resumes from where it left off with its generation history encoded in the prompt.

## CUDA graphs

Every generation step involves CPU overhead, from assembling the batch to dispatching GPU kernels and reading the results. CUDA graphs eliminate the overhead by recording the full GPU execution sequence once and replaying it for batches with matching shapes.

Because the batch shapes change every step, the continuous batching system handles this with padding and caching.

1. Query lengths are padded to the nearest multiple of `q_padding_interval_size`.
2. KV lengths are padded to the nearest multiple of `kv_padding_interval_size`.
3. Recorded graphs are stored in a cache.

When a batch's padded shape matches a cached graph, the graph replays without any CPU dispatch. New shapes trigger a new graph capture.

CUDA graphs require static control flow and are incompatible with attention masks. They're auto-detected by default and disabled when conditions aren't met.

## Async batching

Async batching uses two I/O buffer pairs and two CUDA streams to overlap CPU and GPU work.

```text
Sequential
  CPU  ── [prep N] ·········idle········· [prep N+1] ·········idle·········
  GPU  ·········idle········· [compute N] ·········idle········· [compute N+1]

Async
  CPU  ── [prep N] [prep N+1] [prep N+2] ──
  GPU  ── ········ [compute N] [compute N+1] ──
```

While the GPU computes batch N, the CPU prepares batch N+1. However, overlapping the two requires roughly double the VRAM.

Async batching requires CUDA graphs to be active, since graph replay provides the stable tensor addresses needed for stream overlap to be correct.

## Offloading

Requests that generate long outputs can grow until the KV cache has no room to allocate the next block. Rather than crash, the manager offloads active requests back to the waiting queue to free blocks for the requests that remain.

Offloading is demand-driven. The scheduler reports the *stalled* requests, the active requests that failed to allocate the blocks they need, along with how many blocks each one demands. The manager offloads the fewest requests needed to free enough blocks to cover that demand. It offloads newest first, so requests already scheduled into the current batch keep their cache, and the last remaining active request is never offloaded.

Two situations trigger offloading.

- A batch is scheduled, but some active requests stalled. The manager offloads enough of the stalled requests that the next batch can allocate blocks for the rest.
- The cache is full and no batch can be scheduled at all. The manager offloads requests and retries scheduling in a loop within the same step, until a batch fits or no request can be offloaded. It no longer waits for the next forward pass to recover.

An offloaded request takes one of two paths. When `cpu_offload_space` is greater than `0.0`, the manager copies the request's KV cache blocks to a pre-allocated pinned CPU buffer, batching every request that fits the swap pool into a single transfer. After GPU cache space becomes available, the manager copies the blocks back to the GPU and resumes the request without recomputing its prompt and generated tokens. Requests that don't fit the pool, or every offloaded request when CPU offloading is disabled, fall back to a [soft reset](#soft-reset).

## Next steps

- The [Continuous batching blog post](https://huggingface.co/blog/continuous_batching) covers KV caching, chunked prefill, and dynamic scheduling with performance benchmark numbers.
- For usage examples, see the [Continuous batching](./continuous_batching) doc.
