Multi-provider LLM routers — 9router, OpenRouter, Portkey, LiteLLM, and a growing list of others — sell a simple pitch: one API surface, forty-plus providers behind it, automatic fallback when something breaks, and token-reduction tricks that shave the bill. For teams burned by an OpenAI outage or watching their Anthropic spend creep past forecast, the appeal is obvious. The harder question is where this abstraction belongs in a production stack, because the answer changes what your retry logic, your governance layer, and your quality monitoring can actually see.
The case for the router is real
We should be honest that the value proposition holds up. Provider outages are not rare events anymore — they are a quarterly fact of life. Pricing arbitrage between Gemini Flash, GPT-4o-mini, and Claude Haiku is genuine, often 3–10x for tasks that do not need the frontier. Token-reduction proxies that strip redundant context, compress system prompts, or cache deterministic prefixes can cut spend by 20–40% on chat-heavy workloads without touching application code. And the operational simplicity of one SDK, one auth model, one billing relationship matters more than engineers like to admit.
If you are running a single product surface with one or two LLM-backed features and your team is small, dropping a router in front of everything is probably the right call. The cost is a network hop and a vendor dependency. The benefit is that you stop writing the same provider-fallback code in three places.
Where the architectural choice actually lives
The interesting question is not whether to use a router. It is which layer owns provider selection — and what that decision costs the layers above and below it.
Think of an LLM call as passing through roughly five concerns:
- Application logic — the prompt, the task, the expected output shape.
- Governance — PII redaction, prompt-injection screening, policy enforcement, audit logging.
- Quality control — evals, output validation, fallback to a different model (not provider) when confidence is low.
- Reliability — retries, timeouts, circuit breakers, provider failover.
- Transport — auth, rate limiting, the actual HTTP call.
A router like 9router naturally wants to own layers 4 and 5, and increasingly reaches up into 3 with built-in caching and token reduction. That is a reasonable scope. The trouble starts when teams let it implicitly absorb concerns from layer 2, or when its retry behaviour interacts badly with retry logic already living in the application.
The retry-on-retry problem
This is the most common failure mode we see. The application has its own retry policy — maybe three attempts with exponential backoff, because some downstream queue requires it. The router also retries, perhaps with its own fallback chain across providers. The provider SDK underneath sometimes retries too. A single user request that should have failed fast after 4 seconds can spend 90 seconds bouncing through a fallback tree, hitting four different providers, before finally surfacing an error that the application then retries twice more.
The symptoms are tail-latency spikes that do not correlate with any single provider's status page, mystery bills from providers you thought you had deprioritised, and audit logs that show the same prompt sent to three vendors you did not approve for that data class.
The fix is not subtle but it is unglamorous: pick one layer to own retries, disable retry behaviour everywhere else, and document it. If the router owns failover, the application should fail fast on the first error. If the application owns it, the router should be configured to attempt one provider and return.
Governance gets harder, not easier
The second issue is governance. When a router transparently fails over from Provider A to Provider B, the data residency, the model card, the fine-tuning policy, and the retention defaults all change. Most routers expose this in their dashboard, but few applications surface it to the layer that needs to know — your DLP, your compliance audit trail, your customer-facing "powered by" disclosure.
A concrete example: a healthcare-adjacent product routes through a US-only provider for a specific reason. The router falls back to a provider with EU-default endpoints under load. Technically the request succeeded. Legally it may not have. The router did its job. The governance layer did not, because it was sitting upstream of the abstraction and never saw which provider actually answered.
The pattern that works is to treat the router's response metadata — actual_provider, actual_model, actual_region — as first-class fields that flow into your audit log and, where relevant, into the response your application returns. If the router does not expose this cleanly, that is a procurement signal.
Quality monitoring across providers is its own discipline
A router can keep your service available. It cannot keep your output quality consistent. GPT-4o and Claude 3.5 Sonnet do not produce equivalent outputs for the same prompt — they differ in tone, in structured-output compliance, in refusal behaviour, in how they handle long context. Falling over from one to the other during an outage is fine for a chatbot. It is potentially catastrophic for a tool-calling agent whose downstream code expects a specific JSON shape, or for a regulated workflow where output variance is a defect.
This means quality evals need to run per provider, not per logical endpoint. If you are using a router, your eval harness should be able to pin a specific provider and replay your golden set against each one in your fallback chain, not just against the primary. Otherwise you are flying blind on what the failover actually delivers.
Token-reduction features deserve the same scrutiny. Aggressive prompt compression that saves 30% on tokens but degrades instruction-following on edge cases is a bad trade if you do not measure it. The honest version is: turn on the reduction feature, re-run your evals, decide.
A reasonable default architecture
For most production systems we work on, the layering that holds up looks like this:
- Application layer owns the prompt, the task definition, and fast-fail behaviour. No retries here.
- A thin governance shim — sometimes ours, sometimes a vendor — sits between the application and the router. It handles PII redaction, policy checks, and writes the audit record. Critically, it reads provider metadata from the response and enriches the audit record after the fact.
- The router owns provider selection, failover, caching, and token reduction. One retry budget, configured deliberately.
- Quality monitoring runs out-of-band, sampling production traffic and replaying golden sets per provider, not per logical endpoint.
This is not the only valid shape. Teams with strict data-residency requirements often skip routers entirely and build a narrower abstraction over two or three vetted providers. Teams running heavy agent workloads sometimes put the router below a model-selection layer that picks the right model for the task before the router picks the right provider for the model. Both are defensible.
Our take
Multi-provider routers are a useful piece of infrastructure, not a strategy. They solve the transport and reliability problem well, and the token-reduction features are increasingly worth the integration cost on their own. What they do not do — and should not be asked to do — is replace the governance and quality layers that need to know which model actually answered a request and whether that answer was any good.
The teams that get the most out of these proxies are the ones who treat them as a clearly scoped layer with explicit boundaries, not as a magic box that makes the LLM problem go away. The teams that struggle are the ones who let the router quietly accumulate responsibilities it was never designed to own. The architecture is the choice. The router is just a tool inside it.