[Bug]: POST /reload/model_cost_map is not reliably propagated to all proxy replicas
proxy
### Check for existing issues
- [x] I have searched the existing issues and checked that my issue is not a duplicate.
### What happened?
`POST /reload/model_cost_map` does not reliably propagate a refreshed model-cost map to every LiteLLM Proxy replica.
In a three-replica deployment, calling **Reload Price Data** through the load-balanced Admin UI caused `/model/info` to alternate between old and new metadata. Twenty identical requests returned the stale value 17 times and the refreshed value 3 times.
This can make model capability metadata, token limits, and pricing depend on which replica serves a request. Clients that cache `/model/info` can then retain stale capabilities after hitting an outdated replica.
The implementation appears to use a lossy shared boolean as an event:
1. The endpoint refreshes the serving process's `litellm.model_cost` and writes `force_reload: true` to `model_cost_map_reload_config`.
2. Background polling processes observe that flag.
3. The first process that consumes it reloads its local map and writes `force_reload: false`.
4. Replicas that poll afterward see `false` and remain stale.
If DB model synchronization is disabled, the other replicas do not consume the flag at all. Redis does not distribute `litellm.model_cost`; the map remains process-local.
Expected behavior: after a successful manual reload, every running proxy replica should converge on the same model-cost-map generation.
This is separate from the custom-pricing defects tracked in #30081, although its production report independently notes the same multi-replica behavior: https://github.com/BerriAI/litellm/issues/30081#issuecomment-5030843840
The multi-pod reload feature was introduced by #13470: https://github.com/BerriAI/litellm/pull/13470
A durable implementation could store a monotonically increasing reload generation in shared storage. Each process would retain its last applied generation and reload whenever the shared generation is newer, without any process clearing the event for the others.
### Steps to Reproduce
1. Run LiteLLM Proxy with at least three replicas sharing the same database and with `STORE_MODEL_IN_DB=True`.
2. Set `LITELLM_MODEL_COST_MAP_URL` to a controllable valid model-cost-map URL and start all replicas with version A of that map.
3. Publish version B at the same URL with a recognizable metadata change.
4. Call `POST /reload/model_cost_map` once through the load-balanced Service, or use **Reload Price Data** in the Admin UI.
5. Wait longer than the background polling interval.
6. Query each pod directly using `/public/litellm_model_cost_map` or authenticated `/model/info`.
7. Observe that only the serving pod and, depending on polling order, at most one additional pod has version B. At least one replica can remain on version A.
The same problem is even more direct with `STORE_MODEL_IN_DB=False`: only the serving process reloads.
### Relevant log output
```shell
requests 1-17: supports_reasoning = null
request 18: supports_reasoning = true
request 19: supports_reasoning = null
request 20: supports_reasoning = true
```
### What part of LiteLLM is this about?
Proxy
### What LiteLLM version are you on ?
v1.95.0
### Twitter / LinkedIn details
No response
0 条评论