ITADN

[Bug]: Responses API — reasoning items (incl. encrypted_content) are silently dropped for providers without a native ResponsesAPIConfig (anthropic, bedrock/converse, non-gpt databricks)

#36197Opengilday 创建于 13 天前
llm translationclaude code
G
gildaycommented
### What happened? When a **Responses API** request carrying `reasoning` items is routed to a provider that has **no native `ResponsesAPIConfig`** (`anthropic/`, `bedrock/converse/`, non-`gpt` `databricks/`), LiteLLM converts the request to that provider's chat dialect and **silently drops every reasoning item**, including `encrypted_content`. Tool plumbing survives the conversion correctly — this is specifically about reasoning state. The practical effect is that a reasoning model loses its reasoning across every tool-call turn and re-derives it from scratch. That is the exact penalty the Responses API exists to avoid ([OpenAI measures it at +5% on TAUBench and 40–80% better cache utilization](https://developers.openai.com/blog/responses-api)). Because nothing errors, a caller migrating to `/v1/responses` for reasoning persistence gets none of it on these providers and has no signal that anything was lost. Interestingly, LiteLLM *does* translate the reasoning **effort knob** on this path — it emits `thinking: {"type": "enabled", "budget_tokens": 2048}` (or `additionalModelRequestFields.thinking` for Bedrock). So the request tells Claude to think fresh while discarding the prior thinking. That combination is arguably worse than dropping both, because it spends tokens re-deriving reasoning that was already paid for. ### Relevant log output / evidence Measured by pointing `api_base` at a local HTTP server that records the request body. Config: ```yaml model_list: - model_name: p-anthropic litellm_params: model: anthropic/claude-sonnet-4-5-20250929 api_key: sk-ant-fake api_base: http://127.0.0.1:9099 - model_name: p-bedrock litellm_params: model: bedrock/converse/us.anthropic.claude-sonnet-4-5-20250929-v1:0 aws_access_key_id: AKIAFAKEFAKEFAKEFAKE aws_secret_access_key: fakefakefakefakefakefakefakefakefakefake aws_region_name: us-east-1 aws_bedrock_runtime_endpoint: http://127.0.0.1:9099 general_settings: master_key: sk-probe-local ``` Request to `POST /v1/responses`: ```json { "model": "p-anthropic", "reasoning": {"effort": "medium"}, "input": [ {"role": "user", "content": "Check CWE-79 then CWE-89 and summarize."}, {"type": "reasoning", "id": "rs_001", "summary": [{"type": "summary_text", "text": "First I should look up CWE-79."}], "encrypted_content": "ENCRYPTED_BLOB_ONE"}, {"type": "function_call", "call_id": "call_1", "name": "lookup", "arguments": "{\"cwe\":\"79\"}"}, {"type": "function_call_output", "call_id": "call_1", "output": "XSS details"}, {"type": "reasoning", "id": "rs_002", "summary": [{"type": "summary_text", "text": "Now CWE-89."}], "encrypted_content": "ENCRYPTED_BLOB_TWO"}, {"type": "function_call", "call_id": "call_2", "name": "lookup", "arguments": "{\"cwe\":\"89\"}"}, {"type": "function_call_output", "call_id": "call_2", "output": "SQLi details"} ], "tools": [{"type": "function", "name": "lookup", "description": "Look up a CWE", "parameters": {"type": "object", "properties": {"cwe": {"type": "string"}}, "required": ["cwe"]}}] } ``` What each provider actually receives upstream, counting how many of the two `encrypted_content` blobs survive: | `litellm_params.model` | Upstream route | Reasoning blobs surviving | Tool pairs | | --- | --- | --- | --- | | `openai/gpt-5.4-mini` | `POST /responses` | **2 / 2** | intact | | `azure/gpt-5.4-mini` | `POST /openai/responses?api-version=…` | **2 / 2** | intact | | `databricks/pixee-gpt54-mini` | `POST /responses` | **2 / 2** | intact | | `anthropic/claude-sonnet-4-5-…` | `POST /v1/messages` | **0 / 2** | intact | | `bedrock/converse/us.anthropic.claude-sonnet-4-5-…` | `POST /model/{id}/converse` | **0 / 2** | intact | | `databricks/claude-sonnet-4-5` (no `gpt` in name) | `POST /chat/completions` | **0 / 2** | intact | The reasoning `summary` text is dropped along with `encrypted_content`. Identical results on **v1.86.1** and **v1.95.0**. ### Root cause `ProviderConfigManager.get_provider_responses_api_config` (`litellm/utils.py`, ~L8214) has no entry for `ANTHROPIC` or for `bedrock/converse`, so those requests take the chat-conversion fallback, and that conversion has no representation for a reasoning item. The mapping exists, but only in the opposite direction: `LiteLLMAnthropicToResponsesAPIAdapter` (`litellm/llms/anthropic/experimental_pass_through/responses_adapters/transformation.py`) implements `translate_thinking_to_reasoning` for the Anthropic-in → Responses-out path. There is no Responses-in → Anthropic-out counterpart that replays a `reasoning` item as a `thinking` block with its `signature`. ### Suggested fix Replay reasoning items as Anthropic `thinking` / Bedrock `reasoningContent` blocks, preserving the signature, rather than dropping them. Anthropic's extended thinking already has the matching concept, and pydantic-ai's native `AnthropicModel` does exactly this round-trip today (`BetaThinkingBlockParam` with `signature`), so there is a working reference for the shape. If a full mapping is not desirable, **failing loudly would be better than dropping silently** — today there is no way for a caller to detect the loss. Two related notes: 1. **Please consider capability-driven dispatch rather than model-name substrings.** The Databricks branch still gates on `if model and "gpt" in model.lower()`, so renaming a model silently changes the wire format and the fidelity of the request. `BEDROCK_MANTLE` in the same function already does this the better way, and the comment there states the principle explicitly: *"Both decisions are data-driven from the model's price-map entry, with no model-name logic."* Extending `mantle_supports_responses`-style capability lookup to the other providers would remove a whole class of surprise. 2. This is adjacent to #35677, which is a different symptom of the same hardcoded-model-name pattern on the Anthropic Responses path. ### Are you a ML Ops Team? No ### What LiteLLM version are you on? v1.86.1 and v1.95.0 (both reproduce) --- *(Filed on Johnathan Gilday's behalf by Claude Code.)*
0 条评论