Feature request: add an OpenAI-compatible Fusion model on top of Auto Router

## Problem LiteLLM's Router and Auto Router are well suited to selecting one deployment/model for a request. There is a related use case where the caller wants to fan out one prompt to several candidate models in parallel and then ask an aggregator model to synthesize the final answer (Mixture-of-Agents / Fusion, similar to Together MoA and Hermes MoA). Today this requires an external orchestration service, even though LiteLLM already provides the provider translation, retries/fallbacks, callbacks, spend tracking, rate limiting, and OpenAI-compatible API surface needed by each sub-call. ## Proposed feature Add an optional composite/Fusion model which is exposed through the normal OpenAI-compatible endpoint: ```json { "model": "fusion/coding", "messages": [ {"role": "user", "content": "Review this implementation"} ] } ``` The Fusion layer would: 1. Select a configured candidate set, optionally through Auto Router. 2. Call reference model groups in parallel using the existing LiteLLM Router. 3. Drop failed/timed-out candidates according to a configurable policy. 4. Send successful candidate responses to an aggregator model, also through the existing Router. 5. Return the aggregator response in the normal Chat Completions format. ## Suggested configuration ```yaml model_list: - model_name: coding-fast litellm_params: model: openai/gpt-5-mini - model_name: coding-deep litellm_params: model: anthropic/claude-sonnet-4-5 - model_name: coding-alt litellm_params: model: gemini/gemini-2.5-pro - model_name: fusion-aggregator litellm_params: model: openai/gpt-5 fusion_models: - model_name: fusion/coding selector: type: static models: - coding-fast - coding-deep - coding-alt aggregator: model: fusion-aggregator fusion_params: min_successful_responses: 2 max_parallel_requests: 3 reference_timeout: 45 aggregator_timeout: 90 failure_mode: best_effort tool_calling: aggregator_only ``` An Auto Router selector could later be configured as: ```yaml selector: type: auto_router candidate_models: - coding-fast - coding-deep - coding-alt top_k: 3 ``` ## Important implementation boundary The existing Router currently has a useful single-deployment selection contract. It would be safer to keep that contract unchanged and add a separate Fusion Orchestrator above it: ``` Fusion Orchestrator -> existing LiteLLM Router (reference calls) -> existing LiteLLM Router (aggregator call) ``` This would avoid making every routing strategy handle a multi-deployment return value. A future selector interface could expose an optional `RoutingPlan` / Top-K API while preserving the current single-model API for backward compatibility. ## MVP scope - Chat Completions API. - Static candidate model groups. - Async parallel reference calls. - One aggregator model. - Best-effort and fail-closed policies. - Aggregator-only tool calling; reference calls do not execute tools. - Final aggregator streaming after reference calls complete. - Correlation metadata for reference/aggregator calls. - Reuse existing Router behavior for retries, fallbacks, callbacks, budgets, and spend tracking. - Reject recursive/nested Fusion configurations. ## Open questions - Should configuration live under a new `fusion_models` section or be represented as a special model type in `model_list`? - Should Top-K Auto Router selection be introduced as a new optional selector API rather than changing `async_get_available_deployment()`? - What is the preferred usage/cost representation for the N+1 underlying calls? - Should Responses API and tool-call judging be part of the initial implementation or follow-up work? ## Success criteria - Existing LiteLLM clients work by changing only the model name to a configured Fusion model. - Reference calls run concurrently and respect per-call timeout/cancellation. - A partial provider failure does not unnecessarily fail the whole request. - All underlying calls remain visible to existing callbacks and spend tracking. - Existing non-Fusion Router strategies and configurations remain backward compatible. Would this direction fit the Router/Proxy architecture? I would be happy to contribute an MVP implementation once the configuration and extension boundary are agreed.