In a LiteLLM deployment, the model string in an application request is usually a public name for a pool of candidate deployments, not the address of one fixed provider endpoint. The Router resolves that name into an eligible deployment after applying model-group matching, access controls, filters, and a routing strategy. This article traces that resolution path from model_name through the concrete backend that serves the request.
The matching mechanism turned out to be simpler than I expected, and that simplicity is exactly where its power comes from. Every request arrives at the gateway carrying a model field. LiteLLM compares that string against the model_name entries in the configuration file. On a match, the call is forwarded to the backend provider and model defined under litellm_params for that entry. The request never addresses a provider directly — it names something, and the configuration decides what that something actually is.
That indirection is what lets you swap backend models without ever touching the application's request format. When I look at how this gets used in practice, three patterns stand out:
On the proxy path, the flow is straightforward:
The response climbs back up the same path as a ModelResponse, now carrying an x-litellm-response-cost header. One design detail I appreciate here: cost is calculated asynchronously, after the response. Spend is logged to Postgres, but that bookkeeping never sits between the provider and your caller.
The SDK path is shorter. If you use the LiteLLM SDK directly, your code constructs the Router itself, and the gateway steps simply do not exist in that flow. The matching logic, though, stays identical.
The key implication is the same in both paths: the model field in the request names a model group — a public model_name — never a single deployment. The Router is the component that selects the concrete deployment inside that group. Which deployment gets chosen is a separate decision entirely, and that decision is where the rest of this article goes.
Two RouterGeneralSettings flags control how the Router behaves at this stage:
async_only_mode (default False): the Router initializes only async clients. This is a memory optimization — fewer client objects held in memory.pass_through_all_models (default False): requests for models that are not in the Router's model list pass through directly to litellm.acompletion/embedding instead of being rejected outright.
Here is the detail that finally answered my colleague's question: the router never picks a provider or a model string. It picks a deployment — one concrete backend defined by a specific combination of model, api_base, and api_key. gpt-4o served from OpenAI is one deployment; gpt-4o served from Azure, with a different base URL and different credentials, is a completely separate deployment that just happens to answer to the same name. The deployment is the smallest unit the Router can choose, and every other mechanism in the routing stack exists to answer a single question: which deployment gets this request?
In config.yaml, each deployment is exactly one entry under model_list, with everything the provider call needs packed into that entry's litellm_params:
model_list:
# Deployment 1 — Anthropic direct, my own key
- model_name: sonnet-4
litellm_params:
model: anthropic/claude-sonnet-4-20250514
api_key: os.environ/MY_ANTHROPIC_KEY
# Deployment 2 — same model, customer-managed key via proxy
- model_name: byok-sonnet-4
litellm_params:
model: anthropic/claude-sonnet-4-20250514
api_key: os.environ/CUSTOMER_KEY
api_base: https://proxy.litellm.ai/api.anthropic.com
# Deployment 3 — same public name, different provider entirely
- model_name: sonnet-4
litellm_params:
model: vertex_ai/claude-sonnet-4-20250514
vertex_project: my-project
The naming here is the part worth staring at. Deployments 1 and 3 both expose model_name: sonnet-4 while pointing at two different providers — Anthropic direct and Vertex AI. That is not a conflict; it is exactly how a multi-member model group is built. Deployment 2 gets its own public name (byok-sonnet-4) because it carries a customer-managed key through a proxy and needs to be addressable separately. Three entries, two providers, two model groups — all from one list.
model_id: how the router tells deployments apartLiteLLM assigns every deployment a unique model_id by deterministically hashing all of its litellm_params. Two consequences follow from that design choice:
sonnet-4 goes into cooldown, the Anthropic deployment with the same model_name keeps serving — behavior that looks random until you know the mechanism.litellm_params set, editing a deployment's api_base changes its identity. In practice, the router treats the edited entry as a fresh deployment with a clean health record.Throughput hints go inside each deployment's litellm_params:
rpm — requests per minute. Setting rpm: 900 on one deployment and rpm: 10 on another makes the router spread calls proportionally to that capacity.tpm — tokens per minute, used alongside RPM, for example tpm: 100000 together with rpm: 10000.weight — a bias knob for the simple-shuffle strategy: weight: 9 versus weight: 1 means the first deployment is picked roughly 90% of the time.model_info gotchaReliability settings obey a rule that bites nearly everyone once: allowed_fails and cooldown_time must live under model_info, never under litellm_params. The reason is structural — litellm_params is sent directly to the LLM provider, so the router does not look there for its own tuning values. LiteLLM also supports per-deployment overrides through allowed_fails_policy and per-deployment cooldown_time, and it will warn you when a deployment's credentials contradict its declared provider, say an OpenAI key attached to a vertex_ai/ model entry.
So a model group is nothing more exotic than "deployments that share a name." The interesting question is what happens when several of them answer to the same name — which of them actually gets the call. That is the next layer on the path from name to provider.
With single deployments covered, the next question is what glues them together. The answer is the model group: every deployment that shares the same public model_name. When my application sends model: "gpt-4o", it is addressing that pool of candidates — never an individual deployment. The group is the unit the request actually talks to; which member answers is decided later, by the router.
This detail restructured my mental model more than anything else in the config. Nothing requires a group to be single-provider:
gpt-4o on both OpenAI and Azure, and the router treats them as one pool.litellm_params.model value) is attached to each deployment entry, not to the group.So declaring a model group is really just declaring: for this public name, these backends are interchangeable. That single statement powers the three production wins routing exists for:
Here is where interchangeability hits a boundary. Cached responses are shared within a group by default, not between groups — sensible, since two groups can hide completely different models. If I want a sibling group to serve another group's cache hit, I have to opt in. With openai-gpt-3.5-turbo (backed by gpt-3.5-turbo-0613) and azure-gpt-3.5-turbo (backed by azure/chatgpt-v-2), the configuration is:
cache_responses=Truecaching_groups=[('openai-gpt-3.5-turbo','azure-gpt-3.5-turbo')]With that in place, an identical request hitting azure-gpt-3.5-turbo can be answered from a cache entry written by openai-gpt-3.5-turbo.
A wildcard entry like model_name: "azure/*" proxies every model matching the pattern from that provider — no need to enumerate them one by one. Wildcards also matter on the resolution side: they are how fallback targets such as azure/gpt-4o get matched to a real deployment when the router needs one.
Two patterns show up constantly in production configs:
model_name, and the split is adjusted in config rather than through a redeploy.At the key or team level, router_settings.model_group_alias maps an alias to a model group. Hidden aliases are still resolved when I look a model up explicitly. Ordering matters here too: model group aliases are resolved before pre-routing strategy dispatch, so by the time any routing strategy runs, the request already points at a real group. One recent fix is worth knowing as well — an empty router_settings list in the database no longer silently clobbers the YAML fallbacks, meaning my file-based alias config survives a sparse DB entry.
The similar names make these two features easy to conflate. Mapping each one onto the request pipeline resolves the ambiguity: they operate at different stages entirely.
A routing group is a named set of model_names bound to a single routing strategy. This is the mechanism that lets one Router instance run several strategies simultaneously — a detail I consider the whole point of the feature.
model_name belongs to at most one routing group. If a name isn't claimed by any group, it falls back to the Router's top-level routing_strategy — the default behavior.model: <group_name>, LiteLLM treats the union of every member's deployments as the candidate pool and applies the group's strategy to pick among them. These callable groups even appear in /v1/models, which is how Claude Code and Codex discover them during client setup.Here is the subtlety that tripped me up: a routing group assigns a strategy, not a candidate pool. Grouping gpt-4o and claude-opus together does not mean a gpt-4o request can be answered by Claude. Each name is still load-balanced strictly within its own model group — just using the shared strategy the routing group defines.
The setup that made this click for me:
router_settings:
routing_strategy: simple-shuffle
routing_groups:
- name: latency-sensitive
models: [gpt-4o]
routing_strategy: latency-based-routing
routing_strategy_args:
ttl: 3600
The result: gpt-4o requests get latency-based routing across its OpenAI + Azure deployments, while cheap-model — unclaimed by any group — stays on simple-shuffle. One Router, two strategies, zero separate processes.
I also noticed these callable groups can be granted directly on keys and teams, which gives access-control granularity at the group level, not just per-model.
An access group lives under model_info.access_groups and answers a completely different question. It bundles models under a name so I can grant an API key or team the whole set at once — but it governs only whether a call proceeds, never how it gets load-balanced.
key_ids and team_ids are synced from the key and team write paths.Once I internalized that ordering — authenticate first, route second — I stopped mixing them up. My configs got cleaner too: access groups in the key/team layer, routing groups in the router layer, and never the twain shall meet.
Once I had compared the individual routing strategies side by side, one realization changed how I read the entire subsystem: they all run the same pipeline. Every strategy reads recent statistics, removes deployments that are unhealthy or over their tpm/rpm limits, scores the survivors with its own metric, and picks the winner. A latency-based router and a usage-based router don't disagree about how to choose — only about what "best" means.
The full pipeline runs in five steps:
model_name — the whole model group.And here is the design decision I consider the most important in the whole flow: tags and budgets are filters, not selectors. Neither tag-based routing nor the budget limiter ever returns a final deployment. They can only shrink the candidate pool before a strategy sees it. That is exactly why they compose with any routing strategy instead of replacing one — I can pin a request to EU deployments and still let latency-based routing pick the fastest one among them.
Tag filtering is enabled with enable_tag_filtering under router_settings. Deployments get labeled with tags, requests can carry tags in their metadata, and get_deployments_for_tag() returns only the deployments whose tags match the request's tags. The fallback logic is worth knowing precisely:
Matching supports exact strings and tag regex patterns, plus two refinements: a required-AND prefix written with & (all listed tags must match), and an allow_fail_open flag controlling what happens when a tag matches no deployment. The typical use cases tell you how this is meant to be used in practice: free tier versus paid tier, region or compliance pinning, dev versus prod.
The budget limiter is RouterBudgetLimiting, a CustomLogger that enforces per-deployment spend budgets. When a deployment exhausts its budget, it is removed from the eligible set and fed into the health filter — so the routing strategy simply never sees it. When the budget window resets, the deployment rejoins the rotation automatically. I find this implementation choice clever: the component that observes spend is the same one that influences the candidate pool, with no separate pass required.
Step 2 actually combines two mechanisms:
allowed_fails: 3 and cooldown_time: 5s (via DEFAULT_COOLDOWN_TIME_SECONDS). Fail three times, sit out for five seconds.There is also a pre-call layer: with enable_pre_call_checks: true, I can pin region_name on a deployment (for example "eu") and filter requests by region. LiteLLM infers the region automatically for Vertex AI, Bedrock, and IBM WatsonxAI; for Azure it currently requires setting litellm.enable_preview = True.
Once I saw the pipeline as a funnel — five narrowing stages, each independently configurable — the question that started this whole investigation finally had a clean answer. The deployment that served that production request was simply whatever fell out of the last stage of the funnel. And because every stage is a separate, auditable step, I can now trace exactly why it fell out.
Once the pool has been narrowed — model group matched, routing groups respected, fallbacks accounted for — the router is left with a shortlist of deployments that could all serve the request. The routing strategy is the selector that makes the final call: one deployment, one request. LiteLLM ships a menu of strategies plus an escape hatch, and each one encodes a different opinion about what "best" means.
simple-shuffle is the default, and it is the one recommended for production. It picks a deployment based on the rpm or tpm values you provide, falls back to a random pick when neither is given, and supports a weight parameter for finer control. With weights in play, it becomes a weighted random pick proportional to each deployment's weight/rpm/tpm.
What convinces me this deserves the production recommendation is its architecture: it is the only stateless strategy in the menu. No statistics are recorded, and nothing is read from the cache. That translates directly into the best performance with minimal latency overhead — the selector does close to zero work per request.
latency-based-routing picks the deployment with the lowest average recent response time — time-to-first-token for streaming requests. Response times are cached and updated as requests return, so the picture stays current.
Two arguments tune it:
routing_strategy_args: {ttl: 10} — sets the time window the averages are computed over.lowest_latency_buffer — stops the router from dogpiling the single fastest deployment.The buffer example explains the mechanism better than any definition. Five deployments sit at 0.07s, 0.1s, 0.1s, 0.1s, and 4.66s. Without a buffer, every request would hammer the 0.07s deployment. A buffer of 0.5 treats any deployment within half a second of the lowest as eligible — so prod-2, prod-3, and prod-4 (all at 0.1s) get considered instead of overloading prod-1.
Both usage strategies first filter out deployments that have exceeded their TPM/RPM limits, then route to the one with the lowest TPM usage. The difference is the plumbing:
usage-based-routing tracks usage via Redis for the current minute.usage-based-routing-v2 is the async implementation, using async Redis calls — redis.incr and redis.mget.LiteLLM quotes Azure at 6 RPM per 1000 TPM to describe the shape of that trade-off: providers cap you on both dimensions at once, so a selector watching both is the one that keeps requests flowing instead of bouncing off limits.
least-busy picks the deployment with the fewest ongoing calls. Simple, and useful when your deployments differ more in throughput than in raw latency.
cost-based-routing selects the cheapest deployment in four explicit steps:
litellm_params["model"] against the litellm_model_cost_map (backed by model_prices_and_context_window.json) — anything missing from the map defaults to a cost of $1.It reads input/output price per token from your config first, so your own pricing overrides the global map. This is the strategy for high-volume background work — bulk summarization, batch enrichment — where equivalent-quality endpoints differ in price and nobody is measuring a single request's speed.
For anything the menu lacks, you subclass CustomRoutingStrategyBase and plug in your own selection logic. There is also one outlier: lar1 exists as an opt-in, specialized latency-aware tuning wired through its own setup path rather than the standard selector dispatch.
My take after mapping all of these: the selector is where teams over-engineer first. I start with simple-shuffle and its zero per-request cost, and I only reach for latency or usage tracking once logs show a specific problem — a rate-limit wall, a straggler deployment, or a bill that needs cutting.
Understanding how a model name becomes a pool of deployments answered half of my original question. The other half arrived as soon as two different consumers started hitting the same proxy: whose routing settings actually apply? Two keys can send the identical request through the identical gateway and still be routed differently — because router settings in LiteLLM resolve through a strict hierarchy: Keys > Teams > Global.
The lookup order is deterministic, and the most specific configuration always wins:
The history here matters. For a long time, router settings could only be configured globally: one routing strategy, one set of fallbacks, one timeout policy, one retry policy — stamped uniformly onto every request crossing the proxy instance. That works fine until it doesn't. Key-level and team-level settings change the equation:
least-busy for high-priority keys while everyone else gets latency-based-routing.When I map these two levels onto real workflows, the split is clean:
The hierarchy also doubles as the safest rollout path. I'd configure a new routing strategy on a test key first, validate a fallback chain on a small team before any global rollout, and A/B test timeout values across different keys. If something breaks, the blast radius is one key — not the entire proxy.
The mental shift for me was realizing that settings resolution is just another matching problem, same as model groups. The model string decides which pool of deployments answers; the key and the team decide how that pool gets navigated. Once I saw it that way, the gateway stopped being a black box and became something I could reason about end to end — which is exactly what that first "who served this request?" question was really asking.
Every mechanism I've walked through so far shares one quiet assumption: the model name is settled before the request arrives, and routing only decides which deployment inside that group answers it. The newest routing layer breaks that assumption. It sits on top of the classic deployment load-balancers and reads the request itself — its content, its difficulty, its quality requirements — before picking where it goes.
Before the unification, LiteLLM shipped four request-aware strategies:
Auto Router V2 folds three of these — complexity routing, semantic routing, and adaptive routing — into a single unified router with four capabilities:
At runtime, each request is classified using heuristics, the LLM classifier, lexical/semantic keyword rules, or a custom classifier plugin. The classification then routes the request to a pinned model, a random pool, or a Thompson-sampled pool per tier. On the config side, semantic keyword matching, complexity scoring, and adaptive routing collapse into a single autorouter/complexity_router block — and the old Semantic Auto Router is deprecated. If you're running standalone semantic routing, that's a migration to schedule.
The Complexity Router is the most configurable piece of this stack:
One behavioral change stands out: system prompt text is no longer scored for code/technical signals. Only the request content drives the complexity score, so a long technical system prompt no longer inflates every call into an expensive tier. After an upgrade, I'd expect cost distributions to shift noticeably — worth re-checking the dashboards.
Three features make this layer safer to adopt:
complexity_router, fixed-model baselines, and a cost-matched shuffled control on a three-model Gemini 3.x cost ladder. That cost-matched control is the detail I appreciate most: it isolates whether the router beats random selection at equal spend, not merely that cheap models are cheap.Request-aware routing decides before the call; fallbacks decide after it fails. If a call still fails after num_retries, LiteLLM falls back to another model group, and fallbacks execute in order — a list like ["gpt-3.5-turbo", "gpt-4", "gpt-4-32k"] tries gpt-3.5-turbo first.
LiteLLM separates fallbacks by error type, plus a catch-all:
fallbacks — remaining errors, like litellm.RateLimitErrorcontent_policy_fallbacks — litellm.ContentPolicyViolationErrorcontext_window_fallbacks — litellm.ContextWindowExceededErrorsdefault_fallbacks — a catch-all for when a model group is misconfiguredPrecedence is clean: a model-specific fallback always overrides the default.
Fallback chains don't have to live in static config, either. The /fallback endpoint builds them from a primary model, an ordered fallback_models array (for example ["gpt-4", "claude-3-haiku"]), and a fallback_type of "general", "context_window", or "content_policy".
Put together, this stretches the model string to its limit. The name I send is only a starting point — the request's content can steer it into a cheaper pool, and a failure can rewrite the choice entirely. The model that answers is now decided three times: when I name it, when the router classifies it, and when the first attempt dies.