[RFC]: Programmatic Session-Aware KV Cache Management

### Motivation. Authors: @karen-sy **TL;DR: We propose a narrow, router-initiated hint surface as the primary API for programmatic, session-aware KV management directives.** # Introduction > [!NOTE] For this RFC, we will define “router” as an orchestrator on top of multiple engine units. > Performant inference on agentic workloads requires session-aware KV block management. To predict the priority value of a given KV cache block in such workloads, we need complementary information from the router and the engine. The router is aware of session lifecycle and reuse patterns across workers; the engine is individually aware of the KV cache status it holds. Hence, for vLLM to execute a maximally informed cache policy, (1) the router should express policy intent, and (2) vLLM should resolve that intent against exact cache state and execute it. Currently, KV cache lifecycle is managed naively, mostly via LRU: ```bash request arrives -> router chooses target worker via KV overlap/load -> target worker checks local cache hit: reuse miss: recompute or apply local offload policy ``` This is insufficient for multiple scenarios, such as: - another worker already holds the prefix; - the workload knows a request will resume soon; - a session has ended and its KV should be demoted/freed; - the orchestrator wants to protect high-value KV; - or local policy cannot tell a short tool gap from a long one. The missing abstraction is an observable surface where the **orchestrator biases the cache manager** without owning scheduler or memory internals. To this end, **we propose a narrow, router-initiated hint surface (”KvHint”) which enables an external router to pass KV cache intent to vLLM.** This avoids invasive changes to the existing engine scheduler and cache manager: vLLM keeps ownership of scheduling and memory and is free to clip, defer, reject, or accept any `KvHint`. **`KvHint`s communicate intent for a broad range of KV management policies.** Some are session-scoped (e.g. “Evict all blocks for Session-123”), while others (e.g. “Retain a single specific block with hash 0x1234”) do not require session identity. Especially for policies which require long-term (spanning session lifecycle) KV management, we intend to leverage G2+ cache as a shared retention and movement substrate. ```mermaid flowchart LR R["Router policy"] --> K["Provider-neutral KvHint"] K --> E["vLLM engine<br/>(scheduler, cache manager, KV connectors)"] E <--> G1["G1: HBM KV"] E <--> G2["G2: host KV"] E -. "Phase 3" .-> X["Shared G2 or G3+ storage<br/>(e.g. Mooncake)"] ``` ## **Preliminaries: vLLM KV Cache** vLLM’s prefix cache separates logical prefix identity from physical cache allocation. Full reusable token blocks are represented by chained `BlockHash` values: ``` hA = hash(ROOT, A, extra_keys) hAB = hash(hA, B, extra_keys) hABC = hash(hAB, C, extra_keys) ``` The existing logical-to-physical path is: ``` +-------------------+ add group_id +-------------------------+ key in +-----------------------+ resolves +-------------------+ | BlockHash | ---------------> | BlockHashWithGroupId | ---------> | BlockHashToBlockMap | ----------> | KVCacheBlock | | - logical prefix | | - group-specific key | | - logical lookup | | - physical ID | | - chained parent | | - hash + cache group ID | | - duplicate copies | | - reference count | +-------------------+ +-------------------------+ +-----------------------+ +-------------------+ ``` We propose associating session membership and in-session position with the logical `BlockHash` layer through a separate `SessionPrefixIndex`: ``` KvHint(selector = session S) | v SessionPrefixIndex session S -> {hA, hAB, hABC} # logical BlockHash values | +-- G1: add cache group ID | hAB -> (hAB, group 0) | = BlockHashWithGroupId | | | v | BlockHashToBlockMap | | | v | KVCacheBlock copy/copies | +-- G2: pass the logical keys to the connector, which resolves its connector-owned copies ``` # **High Level Roadmap** This is a three-phase plan. Phase 1 introduces engine-local session-to-KV resolution. Phase 2 defines the `KvHint` selector and action surface and implements bounded in-engine execution. Phase 3 integrates router policy, shared storage, and external orchestration frameworks such as Dynamo. At the end, the programmatic KV management should be a bidirectional process between the engine and control plane: ``` Engine -> control plane: association, residency, eviction and transfer events accepted / clipped / deferred / rejected / missing outcome Control plane -> engine: bounded KvHints/directives ``` ## Phase 1: Session-addressable KV cache **Requirements**: Tie session associate/membership information with vLLM KV blocks. The abstraction should support a many-to-many association: ``` one session -> many branches/block extents one logical block -> many sessions one logical key -> potentially multiple physical copies ``` ### Proposed design: first-class index of KV blocks’ session membership and in-session position. ```mermaid flowchart TB subgraph O["Orchestrator"] direction LR R["Request session_id"] --> S["Session-relative policy"] --> K["session_id<br/>optional BlockHash boundary"] end subgraph V["vLLM engine"] direction LR H["Request BlockHash chain"] --> I["SessionPrefixIndex"] --> G["BlockHashWithGroupId"] G --> M["G1 map / G2 connector lookup"] --> P["Physical KV copies"] end O -->|"session metadata / selector"| V ``` The `SessionPrefixIndex` stores logical topology using `BlockHash`. Cache-group IDs are added only when resolving selected logical nodes to physical copies. ``` SessionPrefixIndex ROOT / \ hA hX / | \ \ hAB hAD hAE hXY | | hABC hXYZ Parent links: parent[hABC] = hAB; parent[hAB] = hA; parent[hAD] = hA, ... Session frontiers: session_1 -> {hABC}, session_2 -> {hAD}, ... Concurrent branches: session_5 -> {hABC, hAD} ``` ### Tasks: 1. Add first-class, request-level `session_ids` to vLLM requests metadata. (#https://github.com/vllm-project/vllm/pull/48048) 2. **Implement `SessionPrefixIndex`.** Store `BlockHash -> parent BlockHash` links and `session_id -> frontier BlockHash` set. Shared ancestors are stored once, while multiple frontiers represent concurrent branches. Optional per-node reference counts support pruning and shared-session accounting. 3. **Maintain the index synchronously with cache activity.** Maintain the index synchronously during prefix lookup, full-block admission, and decode-block completion. Cache hits must establish session membership even though they produce no `BlockStored` event. Removing a G1 copy must preserve the extent while a G2 or duplicate copy remains. When the last local copy disappears, move affected frontiers to their nearest locally resident ancestor or remove the branch; retain parent-only nodes while they connect resident descendants. 4. **Validate correctness and performance. Validate the indexer behavior and overhead.** ### Alternative design: Orchestrator externally manages KV blocks’ session membership and in-session position ```mermaid flowchart TB subgraph O["Orchestrator"] direction LR R["Global session/radix index"] --> S["Resolve session_id"] --> B["Exact engine block selector"] end subgraph V["vLLM engine"] direction LR G["Add cache-group IDs"] --> M["G1 map / G2 connector lookup"] M --> P["Validate current copies"] --> E["Execute directive"] end O -->|"block selector"| V ``` Alternatively, the orchestrator can maintain the session-prefix index and lower every session-relative selector into exact engine block keys. **Advantages of alternative design:** - vLLM avoids maintaining a duplicate session-to-logical-block index and its memory/update cost. - The orchestrator already maintains these associations to evaluate session-aware policy. **Disadvantages of alternative design:** - vLLM lacks a first-class abstraction for addressing “block(s) associated with `session_id`.” - An externally resolved `list[blocks]` is an orchestrator-side snapshot; this may become stale as blocks are admitted, removed, or moved before or during execution, causing correctness issues. ## Phase 2: KvHint-driven KV cache management **Requirements**: Build vLLM-side plumbing to consume external KVHints, and translate the hints’ semantics into in-engine KV block management procedures. ### Tasks: 1. **Define and carry a provider-neutral `KvHint`** . Propagate the envelope from an external orchestrator through request preprocessing and the engine request path to the scheduler, cache manager, and relevant KV connectors. 2. **Implement the semantics of each hint carried by `KvHint`**. Resolve the hint’s target blocks via `SessionPrefixIndex` or exact block hashes, validate the engine’s actual (G1/G2) KV states, execute (or defer/reject) the hint’s cache action accordingly, and report the outcome. We define the semantics of various `KvHint`s in the “Hint Taxonomy (conceptual)” section. ## Phase 3: Integration with Orchestrator Frameworks (e.g. Dynamo) **Requirements**: Show end-to-end programmatic KV management: an orchestrator framework emits a policy decision as a `KvHint → vLLM` consumes the `KvHint` and performs cache management correctly based on engine state. ### Tasks: 1. Dynamo is actively developing router-based `KvHints` as part of the KVCC library (https://github.com/ai-dynamo/dynamo/issues/11673). We plan to achieve API surface compatibility, such that Dynamo router’s `KvHints` may be elegantly consumed by / lowered to vLLM’s `KvHint` APIs developed in the previous two Phases. # **Hint Taxonomy (conceptual)** Our design of hints is aligned with: [SGLang #27574](https://github.com/sgl-project/sglang/issues/27574) A KvHint combines: ``` target: which logical KV blocks are affected action: Share | Prefetch | Demote | Pin | Retain bounds: optional TTL, priority, destination, or resource limit ``` vLLM resolves the target against SessionPrefixIndex or exact block keys, applies the action to current G1/G2 copies, and reports an accepted, clipped, deferred, rejected, or missing outcome. ```mermaid flowchart LR subgraph W1["Worker A"] direction LR G1A["G1: HBM KV"] -->|"Demote"| G2A["G2: host KV"] G2A -->|"Prefetch"| G1A end subgraph W2["Worker B"] direction LR G1B["G1: HBM KV"] -->|"Demote"| G2B["G2: host KV"] G2B -->|"Prefetch"| G1B end W1 -->|"Share selected KV"| W2 W1 -.->|"Pin / Retain"| W1 W2 -.->|"Pin / Retain"| W2 ``` ### Retain Bias eviction order without guaranteeing protection. A retain hint assigns a relative priority, optionally for a bounded duration; under pressure, vLLM prefers to evict lower-priority KV first. Retained KV may still be evicted when necessary. There is prior draft work on this: https://github.com/vllm-project/vllm/issues/37003 ### Share Make KV available on another worker without recomputation. The orchestrator selects the source, destination, and target KV; vLLM and its connectors validate current source copies, perform the transfer, and publish destination visibility only after completion. Sharing does not necessarily remove the source copy. ### Prefetch Move or copy KV into G1 before it is expected to be used. The common local operation promotes KV from G2 host memory to G1 HBM. If the target is already resident in G1, the hint may complete as a no-op. ### Demote Move KV to G2 instead of discarding it. vLLM first ensures that a valid G2 copy exists, then allows the G1 copy to be reclaimed. Transfer completion and G1 reclamation are separate operations. ### Pin Protect high-value KV from ordinary eviction for a bounded TTL. Pin is not permanent and does not necessarily require G1 residency: vLLM may preserve the target in G2 while allowing its G1 copy to be reclaimed. The engine may clip or reject pins that exceed configured limits. # **Design Principles** 1. **Orchestrator owns policy; engine executes.** The agent-graph / workflow intelligence lives outside. The engine understands priority, TTL, session membership, tier - nothing about why. 2. **Zero overhead when unused.** Un-hinted workloads behave exactly like today. In other words, we are not replacing local prefix matching or LRU principles, but rather augmenting the scope of its capacities. 3. **Hints are soft, bounded, and safe to reject.** The engine may accept, clip, defer, or ignore. Every hint is observable. Nothing a client says can pin memory unboundedly or deadlock the scheduler. 4. **Router-initiated by default.** Workloads can still emit intent, but the router is where workload context merges with global KV placement, worker load, health, and admission. In production environments, the router has: a global KV index from events, built-in HA/fault-tolerance, existing overlap/load routing + admission control, and (with the harness<->orchestrator work) trajectory awareness, not just request awareness. 5. **API surface is not finalized here**: We would like to open discussion for what goes into a `KvHint { retention: [{prefix_tokens, ttl_seconds}] }` .