ITADN

[Bug]: tag filtering 401s the auto-router's routed tier unless every tier deployment repeats the marker's tag

#36621Openmateo-berri 创建于 9 天前
bugllm translation
M
mateo-berricommented
## What happened? With `enable_tag_filtering: true` and an `auto_router/` marker carrying `tags: ["route"]`, a request tagged `route` correctly selects the marker and the semantic router picks its tier, then the request dies with a 401: `Not allowed to access model due to tags configuration. Passed model=gemini-flash and tags=['route']`. After the rewrite, deployment selection for the tier model group re-applies the caller's tags, and the tier deployments do not carry `route`, so the pool comes up empty The tag already did its job selecting the router. Requiring every tier deployment to repeat it forces admins to tag their whole tier fleet, which also changes those models' behavior for direct traffic: tagged direct requests can suddenly reach them, and untagged direct requests can lose access under strict tag semantics Found while reproducing a customer report (ticket #7159) ## User Flow Before a (hypothetical) fix: the exact request the semantic router exists to serve is rejected 1. They send POST http://localhost:4000/v1/chat/completions with `{"model": "gpt4o", "tags": ["route"], "messages": [{"role": "user", "content": "What is the capital of France?"}]}` 2. The response is HTTP 401: `{"error":{"message":"Not allowed to access model due to tags configuration. Passed model=gemini-flash and tags=['route']","code":"401"}}`, naming a model they never put in the request 3. Only after the admin adds `tags: ["route"]` to the gemini-flash deployment, and to every other tier the router can pick, does the 401 stop After a (hypothetical) fix: the same tagged request succeeds with no tags added to any tier deployment 1. The same POST returns 200 with the answer produced by the Gemini tier 2. The gemini-flash deployment keeps serving its direct traffic exactly as before, with no tag changes ## Proof the bug occurs Config the proxy ran with (env vars: OPENAI_API_KEY, GEMINI_API_KEY): ```yaml model_list: - model_name: gpt4o litellm_params: model: openai/gpt-4o api_key: os.environ/OPENAI_API_KEY api_base: https://api.openai.com/v1 - model_name: gpt4o litellm_params: model: auto_router/gpt4o-router auto_router_config: '{"routes": [{"name": "gemini-flash", "utterances": ["What is the capital of France?", "capital city questions", "geography questions"], "score_threshold": 0.3}]}' auto_router_default_model: gemini-flash auto_router_embedding_model: text-embedding tags: ["route"] - model_name: gemini-flash litellm_params: model: gemini/gemini-3.6-flash api_key: os.environ/GEMINI_API_KEY - model_name: text-embedding litellm_params: model: openai/text-embedding-3-small api_key: os.environ/OPENAI_API_KEY router_settings: enable_tag_filtering: true general_settings: master_key: sk-repro-1234 ``` Version: litellm_internal_staging at commit 7a55ca811b, proxy booted with `python litellm/proxy/proxy_cli.py --config repro_config.yaml --port 47613 --detailed_debug` (needs `pip install "semantic-router>=0.1.15"`) Tagged request: ``` curl -sS -X POST http://localhost:47613/v1/chat/completions \ -H "Authorization: Bearer sk-repro-1234" -H "Content-Type: application/json" \ -d '{"model": "gpt4o", "messages": [{"role": "user", "content": "What is the capital of France?"}], "tags": ["route"]}' {"error":{"message":"Not allowed to access model due to tags configuration. Passed model=gemini-flash and tags=['route']","type":"None","param":"None","code":"401"}} ``` Proxy debug log: the semantic router had already picked the tier before the rejection ``` route_choice: name='gemini-flash' ... similarity_score=0.4735... File ".../litellm/router_strategy/tag_based_routing.py", line 286, in _resolve_or_fail_open ValueError: Not allowed to access model due to tags configuration. Passed model=gemini-flash and tags=['route'] ``` Control, the tier called directly with no tags is healthy: ``` curl -sS -X POST http://localhost:47613/v1/chat/completions \ -H "Authorization: Bearer sk-repro-1234" -H "Content-Type: application/json" \ -d '{"model": "gemini-flash", "messages": [{"role": "user", "content": "What is the capital of France?"}]}' {"id":"...","model":"gemini-flash","choices":[{"message":{"content":"The capital of France is **Paris**."... ``` Adding `tags: ["route"]` to the gemini-flash deployment removes the 401 (the request then proceeds to the tier call, where it can still hit the separate alias param-forwarding bug #36619), which confirms the rejection comes from the tier deployment lacking the marker's tag ## What a fix PR should look like When the pre-routing strategy that rewrote the model was itself selected by matching the request's tags, those tags are consumed by that selection and must not constrain deployment choice inside the routed tier's model group. Shape: in `Router.async_pre_routing_hook` (`litellm/router.py`), when the chosen strategy's tags matched the request, stamp the request metadata (next to the existing routing_decision stamp) that request-body tags were satisfied at router selection; in `litellm/router_strategy/tag_based_routing.py`, have `get_deployments_for_tag` treat that stamp as constraints already satisfied for the rewritten model group and keep the full healthy pool. Tags inherited from key or team policy should keep applying; only the request-body tags that selected the router are consumed Regression test: in the test files mapped to `litellm/router.py` and `litellm/router_strategy/tag_based_routing.py` under `tests/test_litellm/`, a tagged request through a tagged marker whose tier deployments carry no tags must reach the tier, and a tagged request sent directly to a plain model group with no matching deployments must still be rejected QA: config above on a live proxy; the tagged curl must return 200 with the debug log showing the outbound call to `generativelanguage.googleapis.com`, and a direct `{"model": "gemini-flash", "tags": ["route"]}` request must still 401 since no router selection consumed that tag
1 条评论