---
title: Gateway Caching
description: Serve byte-identical requests entirely from LLM Gateway at $0 cost.
icon: Zap
---

import { Callout } from "fumadocs-ui/components/callout";

# Gateway Caching

Gateway caching serves a previously-seen, byte-identical request entirely from LLM Gateway without forwarding it to the upstream provider. Repeated identical calls cost **$0** — there is no inference and no provider charge. It is most useful for API workloads with deterministic inputs (classification, batch jobs, FAQ lookups, retries) rather than free-form chat.

<Callout type="info">
	If you want to reduce the cost of long, partially-shared prompts in chat apps
	or coding tools, you want [Provider Cache
	Control](/features/caching/provider-cache-control) instead. That discounts the
	cached portion of your prompt on every call — it does not require
	byte-identical requests. See the [Caching Overview](/features/caching) for a
	side-by-side comparison.
</Callout>

## How It Works

When you make an API request:

1. LLM Gateway generates a cache key based on the request parameters
2. If a matching cached response exists, it's returned immediately
3. If no cache exists, the request is forwarded to the provider
4. The response is cached for future identical requests

This means repeated identical requests are served instantly from cache without incurring additional provider costs.

## Cost Savings

Caching can dramatically reduce costs for applications with repetitive requests:

| Scenario                    | Without Caching | With Caching | Savings |
| --------------------------- | --------------- | ------------ | ------- |
| 1,000 identical requests    | $10.00          | $0.01        | 99.9%   |
| 50% duplicate rate          | $10.00          | $5.00        | 50%     |
| Retry after transient error | $0.02           | $0.01        | 50%     |

<Callout type="info">
	Cached responses are free from provider costs. You only pay for the initial
	request that populates the cache.
</Callout>

## Requirements

<Callout type="info">
	Caching is **free** and **independent** of [Data
	Retention](/features/data-retention). Cached responses live in a short-lived
	cache (TTL-bound, typically seconds to minutes) and are not stored as
	long-term request data — you do not need to enable data retention to use
	caching.
</Callout>

To use caching:

1. Enable **Caching** in your project settings under Preferences
2. Configure the cache duration (TTL) as needed
3. Make requests as normal—caching is automatic

## Cache Key Generation

The cache key is generated from these request parameters:

- Model identifier
- Messages array (roles and content)
- Temperature
- Max tokens
- Top P
- Tools/functions
- Tool choice
- Response format
- System prompt
- Other model-specific parameters

<Callout type="info">
	Requests with different parameter values, even slight variations, will not
	share cache entries.
</Callout>

## Cache Behavior

### Cache Hits

When a cache hit occurs:

- Response is returned immediately (sub-millisecond latency)
- No provider API call is made
- No inference costs are incurred

### Cache Misses

When a cache miss occurs:

- Request is forwarded to the LLM provider
- Response is stored in cache
- Normal inference costs apply
- Future identical requests will hit the cache

## Streaming and Caching

Caching works with both streaming and non-streaming requests:

- **Non-streaming**: Full response is cached and returned
- **Streaming**: The complete response is reconstructed from cache and streamed back

## Cache TTL (Time-to-Live)

Cache duration is configurable per project in your project settings. You can set the cache TTL from 10 seconds up to 1 year (31,536,000 seconds).

The default cache duration is 60 seconds. Adjust this based on your use case—longer durations work well for static content, while shorter durations are better for frequently changing data.

## Identifying Cached Responses

A cache hit replays the stored response body, so it is byte-identical to the original — same `id`, same content. Two markers tell you it was a replay:

- the `x-llmgateway-cache: HIT` response header (set on every lane, including `/v1/messages`, which has no metadata envelope)
- `metadata.cached: true` on the response body (and on the final metadata chunk of a streamed replay)

All cost fields are zeroed on a replay, because no upstream call was made:

```json
{
	"usage": {
		"prompt_tokens": 12,
		"completion_tokens": 48,
		"total_tokens": 60,
		"cost": 0,
		"cost_details": {
			"total_cost": 0,
			"input_cost": 0,
			"output_cost": 0
		}
	},
	"metadata": {
		"cached": true
	}
}
```

Token counts are **not** zeroed — they still describe the completion you are receiving, and are what the dashboard records for analytics. Note that any prompt-caching fields (`prompt_tokens_details.cached_tokens`, `cache_write_tokens`) describe the _original_ upstream call, not the replay.

## Bypassing the Cache for a Single Request

Send `x-no-cache: true` to skip the cache for one request — the call goes upstream and its response is not stored. Useful when a client retries an identical request and expects a fresh sample (for example an agent loop) without disabling caching for the whole project.

```bash
curl https://api.llmgateway.io/v1/chat/completions \
  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-no-cache: true" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Hello"}]}'
```

## Use Cases

### Development and Testing

During development, you often send the same prompts repeatedly:

```typescript
// This prompt will only incur costs once
const response = await client.chat.completions.create({
	model: "gpt-4o",
	messages: [{ role: "user", content: "Explain quantum computing" }],
});
```

### Chatbots with Common Questions

FAQ-style interactions often have repeated questions:

```typescript
// Common questions are served from cache
const faqs = [
	"What are your business hours?",
	"How do I reset my password?",
	"What is your return policy?",
];
```

### Batch Processing

Processing large datasets with potentially duplicate items:

```typescript
// Duplicate items in batch are served from cache
for (const item of items) {
	const response = await client.chat.completions.create({
		model: "gpt-4o",
		messages: [{ role: "user", content: `Classify: ${item}` }],
	});
}
```

## Best Practices

### Maximize Cache Hits

- Use consistent prompt formatting
- Normalize input data before sending
- Use deterministic parameters (temperature: 0)
- Avoid including timestamps or random values in prompts

### Appropriate Use Cases

Caching is most effective for:

- Static knowledge queries
- Classification tasks
- FAQ responses
- Development/testing
- Retry scenarios

### When to Avoid Caching

Caching may not be suitable for:

- Real-time data requirements
- Highly personalized responses
- Time-sensitive information
- Creative tasks requiring variety
- Chat or coding tools where prompts overlap but are not byte-identical — use [Provider Cache Control](/features/caching/provider-cache-control) instead

## Pricing

Caching is **completely free**. Cached responses are held in a short-lived
in-memory cache (bounded by your configured TTL) and do not incur storage
charges. Storage costs only apply if you separately enable [Data
Retention](/features/data-retention) for full request/response payloads.

<Callout type="success">
	Caching reduces both inference cost and latency at no additional charge.
</Callout>
