[Bug]: LiteLLM proxy silently returns understated token counts when Bedrock CountTokens is unsupported for a model

### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? `bedrock-runtime`'s `CountTokens` API does not support most current Anthropic models — including **Claude Opus 5 and Sonnet 5** (full matrix below; of the current line-up only Sonnet 4.6 works). When it rejects a model, the proxy logs a warning and falls through to its local tokenizer. For these models the local tokenizer is badly miscalibrated, so `POST /v1/messages/count_tokens` returns a **confidently wrong, much lower number** with a `200 OK` and no indication to the caller that the value is an estimate. Clients that use `count_tokens` for context management (Claude Code / Claude Desktop do) therefore believe they have far more headroom than they do, keep growing the conversation, and eventually get a hard `400 prompt is too long` from the provider — long after any context-compaction threshold should have fired. ### User Flow **Before a (hypothetical) fix: a developer running Claude Code against the proxy is told their conversation is ~675K tokens when the model actually measures it at ~1.19M, so their client never trims it and the next turn is rejected outright.** 1. They export `ANTHROPIC_BASE_URL=https://litellm-domain` and `ANTHROPIC_MODEL=claude-opus-5`, then start a long refactoring session in Claude Code. 2. As the conversation grows, their client sends `POST https://litellm-domain/v1/messages/count_tokens` with `{"model": "claude-opus-5", "messages": [...the whole conversation...]}`. 3. It comes back `200 OK` with `{"input_tokens": 675431}` — a plain number, with nothing marking it as an estimate rather than a figure the model itself confirmed. 4. Their client compares 675,431 against the 1,000,000-token window Opus 5 advertises, concludes it is at ~68% capacity, and does not summarise or trim anything. 5. They send the next turn: `POST https://litellm-domain/v1/messages` with `{"model": "claude-opus-5", "max_tokens": 4096, "messages": [...the same conversation...]}`. 6. It comes back **`400`** with `{"type":"error","error":{"type":"invalid_request_error", "message":"...prompt is too long: 1227544 tokens > 1000000 maximum"}}` — the model counted the identical conversation as **1,227,544** tokens, roughly 1.8× the number they were given in step 3. 7. Every retry fails the same way. Nothing they can request from the proxy reports a number anywhere near 1.2M, so from their seat the rejection has no visible cause. 8. If the proxy has `fallbacks` configured, they never see the 400 at all: the turn returns `200` and at `https://litellm-domain/ui/?page=logs` that request is listed against **`claude-sonnet-4-6`**, not the `claude-opus-5` they asked for — with the whole conversation re-charged as fresh input, so its cost on that page is roughly 9× the neighbouring turns. **After a (hypothetical) fix: the same call reports a count the model agrees with, so their client trims in time and the turn succeeds on the model they asked for.** 1. They export `ANTHROPIC_BASE_URL=https://litellm-domain` and `ANTHROPIC_MODEL=claude-opus-5`, then start a long refactoring session in Claude Code. 2. As the conversation grows, their client sends `POST https://litellm-domain/v1/messages/count_tokens` with `{"model": "claude-opus-5", "messages": [...the whole conversation...]}`. 3. It comes back `200 OK` with `{"input_tokens": 1227544}` — matching what the model itself counts. (Equally acceptable from the user's seat: a `4xx` saying the count is unavailable for this model, which at least tells their client not to trust a guess.) 4. Their client compares 1,227,544 against the 1,000,000-token window, sees it is over capacity, and summarises the earlier part of the conversation before sending anything. 5. They send the next turn: `POST https://litellm-domain/v1/messages` with `{"model": "claude-opus-5", "max_tokens": 4096, "messages": [...the trimmed conversation...]}`. 6. It comes back **`200 OK`** with `"model": "claude-opus-5"` and a normal `usage` block. 7. The session continues. Trimming happens whenever the reported count approaches the window, so the wall is never hit. 8. At `https://litellm-domain/ui/?page=logs` every turn is listed against `claude-opus-5`, at costs consistent with its neighbours. ### Proof the bug occurs ## Steps to reproduce 1. Configure any model whose Bedrock `CountTokens` support is absent — e.g. ```yaml model_list: - model_name: claude-opus-5 litellm_params: model: bedrock/global.anthropic.claude-opus-5 aws_region_name: us-east-1 ``` 2. `POST /v1/messages/count_tokens` with `{"model": "claude-opus-5", "messages": [...]}` 3. Observe `200 OK` with an `input_tokens` value, and in the proxy log: ``` LiteLLM:ERROR: handler.py:123 - HTTP error in CountTokens handler: Client error '400 Bad Request' for url 'https://bedrock-runtime.us-east-1.amazonaws.com/model/anthropic.claude-opus-5/count-tokens' LiteLLM:WARNING: bedrock_token_counter.py:91 - Bedrock CountTokens API error: status=400, message={"message":"The provided model doesn't support counting tokens."} LiteLLM Proxy:WARNING: proxy_server.py:8772 - Provider token counting failed (400): {"message":"The provided model doesn't support counting tokens."}. Falling back to local tokenizer. ``` ## Scope: this affects every current-generation Claude model on Bedrock This is not one edge-case model. Probed directly against `bedrock-runtime` in `us-east-1` (`aws bedrock-runtime count-tokens --model-id <id> --input '{"invokeModel":{"body":"…"}}'`, body containing a valid Anthropic Messages payload with `max_tokens`): | Model ID | Bedrock `CountTokens` | |---|---| | `anthropic.claude-opus-5` | ❌ `ValidationException: The provided model doesn't support counting tokens.` | | `anthropic.claude-opus-4-8` | ❌ same | | `anthropic.claude-sonnet-5` | ❌ same | | `us.anthropic.claude-haiku-4-5-20251001-v1:0` | ❌ same | | `anthropic.claude-haiku-4-5` | ❌ `The provided model identifier is invalid.` | | **`anthropic.claude-sonnet-4-6`** | ✅ **works** — returns `{"inputTokens": 26}` | | `global.anthropic.claude-sonnet-4-6` | ❌ unsupported — **the geo prefix alone breaks a model that works bare** | So of the current Claude line-up, **only Sonnet 4.6 can be counted on `bedrock-runtime`.** Every other model — including Opus 5 and Sonnet 5, the current defaults — takes the local-tokenizer path on every `count_tokens` call. Two diagnostic notes for whoever picks this up: - A **different** error message (`max_tokens: Field required`) means the model *is* supported and only the request body was wrong. `"doesn't support counting tokens"` is the real rejection, and is what should gate any retry logic. - The proxy already strips the geo/global prefix when building the CountTokens URL (the logged URL is `/model/anthropic.claude-opus-5/count-tokens` for a model configured as `bedrock/global.anthropic.claude-opus-5`), which is correct — passing the prefixed ID through would fail even for Sonnet 4.6. Worth preserving if this code is touched. ## Measured error Identical text throughout. `claude-sonnet-4-6` *is* supported by Bedrock CountTokens, so it gives an authoritative reference count; the target model's true count comes from the provider's own rejection message, which states both the count and the limit. **Tokenizer ratio** — same text, two models: | source | tokens | |---|---| | `anthropic.claude-sonnet-4-6` via Bedrock `CountTokens` (authoritative) | 745,753 | | `global.anthropic.claude-opus-5` — from `ValidationException: prompt is too long: 1227544 tokens > 1000000 maximum` | **1,227,544** | | ratio | **1.646×** | **What the proxy reports** — same 400-paragraph sample: | source | tokens | |---|---| | Bedrock `CountTokens` for `claude-sonnet-4-6` (authoritative) | 12,025 | | proxy `count_tokens` for `claude-sonnet-4-6` | 12,025 ✅ exact (provider path used) | | proxy `count_tokens` for `claude-opus-5` | **11,208** ⬅ local tokenizer, 0.93× the sonnet count | | `claude-opus-5` real count (0.93 → 1.646 correction) | **~19,793** | **The proxy understates this model by ~1.77×.** Note the local tokenizer returns *slightly fewer* tokens than the older model's tokenizer, while the target model's real tokenizer produces considerably more — the two errors compound. ## Impact With a 1,000,000-token model limit, the effective ceiling expressed in numbers the proxy reports is about **565K**. A client that compacts at, say, 80% of the advertised window would trigger at 800K reported — i.e. **~1.41M real, which is past the hard limit.** The safety mechanism is unreachable: the request fails before the threshold is ever approached. Observed in production: a long agentic session ran to ~1.19–1.22M real tokens while the client believed it was around 675K, then failed with `prompt is too long`. Requests below the boundary succeeded normally, so the failure is purely a function of size. Two secondary effects worth noting: 1. **Cost.** If `litellm_settings.fallbacks` is configured, the resulting 400 is silently retried on the fallback model. Prompt caches are per-model, so the fallback runs cold and re-reads the entire context at full price. In our case the per-request cost rose roughly 9×, with no error surfaced to the client and — see below — nothing in the logs. 2. **CPU.** Local tokenization of very large contexts is expensive. Repeated `count_tokens` calls over ~150K-token conversations saturated one of two vCPUs on the proxy host, causing an outage: the app stayed internally healthy while throughput collapsed and health checks timed out. ## Reproduction helper Deliberately overshooting is free — a rejected request is not billed, and the error names both the real token count and the limit, which is the only way we found to obtain a ground-truth count for a model Bedrock will not count: ```python import json, boto3 br = boto3.client("bedrock-runtime", region_name="us-east-1") br.invoke_model(modelId="global.anthropic.claude-opus-5", body=json.dumps({ "anthropic_version": "bedrock-2023-05-31", "max_tokens": 16, "messages": [{"role": "user", "content": "<oversized text>"}]})) # botocore.errorfactory.ValidationException: # prompt is too long: 1227544 tokens > 1000000 maximum ``` ### What part of LiteLLM is this about? Proxy ### What LiteLLM version are you on ? 1.96.2 ### Twitter / LinkedIn details _No response_