What is model routing? How teams cut inference cost without switching providers

Model routing sends each request to the cheapest model that can still do the job, instead of paying frontier prices for every call. It works, but not at the scale 95% noise on the internet promises.

One thing worth understanding before you build anything. The 85% figure comes from MT-Bench, an open-ended chat benchmark where a lot of the traffic is genuinely easy. Your coding agent is not that. This blog gives you the math that predicts your actual ceiling, the reason a cheaper model can bill you more than an expensive one, and the sequence to follow before you write a line of router code.

What is model routing, exactly?

A router is a decision layer that sits between your application and a pool of models. For each request, it predicts which model can handle the work acceptably, then sends the request there. Cheap model for the easy calls, frontier model for the hard ones.

The 2025 survey by Varangot-Reille and colleagues splits every approach into two families, and the split matters more than the marketing names vendors put on top of it.

Pre-generation routing

The router decides before anything is generated. It reads the prompt, estimates topic or difficulty, and picks a model. Because it never waits for a response, it adds only its own decision time. This is the cheap family, and it covers keyword rules, embedding similarity, small classifiers, and matrix factorization over past preference data.

The overhead is small enough to ignore. In the RouteLLM paper, the BERT-based router costs $3.19 per million requests, and the matrix factorization router handles 155 requests per second, so even the most expensive option adds under 0.4% on top of GPT-4 generation cost. IBM Research measured their optimization-based router at roughly 6 milliseconds and 2 kB of memory per task. Latency is not the objection people think it is.

Post-generation routing, also called cascading

The cheap model answers first. Something scores that answer, and if the score is too low the request escalates to a stronger model. FrugalGPT popularised this. It is more accurate about difficulty, because it has seen an actual attempt, and more expensive, because sometimes you pay twice for one answer.

Most production systems now blend the two: a pre-generation guess, then an escalation path when the guess turns out wrong. NVIDIA's escalation router works exactly this way, starting every session on the cheaper model and moving up when an LLM judge sees sustained difficulty.

The vocabulary worth keeping: f is the share of your calls that can leave the frontier model, and r is the cheap model's cost as a fraction of the frontier model's. Everything below is those two numbers.

Why do published savings numbers disagree so wildly?

Because they are measured on different workloads, and the workload decides the answer. We collected every routing result we could find that states both a cost figure and a quality figure, and sorted them by what they were measured on. The pattern is consistent, and nobody seems to state it out loud.

Two things fall out of this table.

The savings track how uneven your traffic is, not how clever your router is: MT-Bench contains a lot of genuinely trivial prompts sitting next to genuinely hard ones, so a router has enormous room to sort. MMLU and GSM8K are uniformly medium-hard, and the savings drop to 29% and 33% with the same routers, on the same paper, on the same day. A coding agent is closer to MMLU than to MT-Bench: almost every turn needs tool use, file context, and multi-step reasoning. There is less easy work to skim off.

There is no free lunch on offer: Six results, six quality costs. The largest saving in the table (74%) carries the largest quality drop (six accuracy points). If a vendor quotes you a savings figure with no accuracy figure beside it, the accuracy figure exists, and you are simply not being shown it.

The 80% to 95% number circulating in AI summaries is the MT-Bench result, stripped of its benchmark, restated as a general fact. Treat headline savings as ceilings rather than forecasts.

What actually sets your model routing ceiling?

Two numbers, and neither of them lives in the router:

S = f × (1 − r)

Maximum savings equals the routable fraction of your calls multiplied by the difference in cost between the destination and your home network. A perfect router with perfect foresight cannot beat this line. Cache misses, retries, escalations, and misroutes all pull you below it.

Compute the ceiling before you build the router. If it is smaller than the effort, the honest answer is not to route.

The formula holds up against real published numbers, which is a decent sign it is the right model of the problem. NVIDIA reports 74% savings while sending just 7% of calls to the frontier model. Back-solve: 0.74 = 0.93 × (1 − r) gives r ≈ 0.20, meaning their cheap tier costs about a fifth of the frontier tier.

Run the same arithmetic on Cognition's 28% coding result with r = 0.20 and you get f ≈ 0.35. Only about a third of the coding work left the frontier model. That is the real finding about coding agents, and it is not a router quality problem. It is a workload property.

Why r is smaller than you assume

The market-wide price spread is enormous. Output tokens run from $0.66 per million on DeepSeek V4 Flash off-peak to $180 per million on gpt-5.5-pro, a 273x range.

Routing only earns the gap between the two rungs you actually move between.

The spread you can use is much narrower, because you are not routing production coding work to the bottom rung. Inside Anthropic's lineup, Claude Haiku 4.5 output is $5 per million against Claude Opus 5 at $25, so r = 0.20. Fall back one rung only to Claude Sonnet 5 at $10 and r = 0.40, which halves your ceiling. At f = 0.35, that is the difference between a 28% ceiling and a 21% ceiling before anything goes wrong.

The tokenizer footnote nobody reads

Price per token is not comparable across models, and one provider says so in its own pricing page. Anthropic notes that Claude 4.7 and later models use a newer tokenizer that "produces approximately 30% more tokens for the same text."

A model priced 25% lower per token that emits 30% more tokens for the same text is not cheaper. It is slightly more expensive. Of the eleven pages ranking on page one for "model routing" in the US, none mentions this. If your router optimizes on published per-token price, it can be wrong by 30% on identical input, and your dashboards will agree with it.

Why does a cheaper model sometimes cost more?

Because switching models throws away your cache, and on an agent workload the cache is most of the bill.

Anthropic publishes the multipliers: cache reads cost 0.1x base input, and a 5-minute cache write costs 1.25x base input. Put today's prices into that and something inverts. A warm cache read on Claude Opus 5 costs $0.50 per million tokens. An uncached input token on Claude Haiku 4.5, the cheapest model in the same lineup, costs $1.00. Reading your context on the expensive model is half the price of reading it on the cheap one.

Take one agent turn carrying 100,000 tokens of shared context and producing 1,000 output tokens:

The downgrade is an investment that pays back on turn two. A router that re-decides every turn never gets there.

The break-even is turn two. That is the single most useful number in this article, because it converts a vague warning into a rule you can implement: a downgrade has to hold for at least two turns to be worth making. The formal version is that the switch pays off immediately only when output tokens exceed 3.75% of context tokens, and a real agent turn is closer to 1%.

Microsoft states the same constraint in its own router documentation: caching benefits "apply only when the same model handles consecutive requests with overlapping prompt prefixes." A router and a cache are in direct tension, and the router usually wins the argument at your expense.

The result that proves it

IBM Research ran 417 tasks from the AppWorld Test Challenge through the same CodeAct agent and expected GPT-4.1 to be cheaper than Claude Sonnet 4.6. GPT-4.1 has lower published prices on both input and output, and Sonnet took roughly three times as many reasoning steps to finish the same tasks.

Sonnet came in at $79 total, $0.19 per task. GPT-4.1 came in at $155, $0.37 per task, nearly double. Their explanation was cache-read pricing, which they describe as "something most routing discussions ignore entirely." The model with worse sticker prices and longer trajectories won on cost because it read its context more cheaply.

Their conclusion is worth quoting directly: "A router that only looks at pricing sheets is optimizing against the wrong numbers."

One more trap in the same family

Minimum cacheable prompt length is not constant across a lineup. Anthropic's caching docs list 512 tokens for Opus 5, 1,024 for Sonnet 5 and Opus 4.8, 2,048 for Opus 4.7, and 4,096 for Opus 4.6 and 4.5. Route a 900-token sub-agent prompt from Opus 5 to Opus 4.5, and it stops being cacheable at all. Nothing errors. Your bill just goes up.

How do teams cut inference cost without switching providers?

Six steps, in this order. The order matters, because three of these are free and routing is not.

1. Instrument before you optimize

You cannot route what you cannot see. Log, per call: model, input tokens, cached input tokens, cache write tokens, output tokens, latency, task type, and whether the result was accepted or retried. The cached-input field is the one teams skip and the one that decides everything above.

The practical starting point, from a hands-on router build walked through in this LLM routing tutorial: look at your logs, find the most common 20% of your prompts, and ask whether a smaller model could handle them. That single query gives you a first estimate of f in an afternoon.

2. Take the free levers first

Three discounts require no routing, no quality risk, and no new failure modes:

Nobody in the current top ten results for this keyword mentions batch or off-peak pricing. These are half-price levers hiding in public documentation.

3. Cut the tokens you send, not just the price you pay for them

Per-turn overhead is worth auditing before you touch model selection. Sending every tool's full JSON schema on every request is a fixed tax on a loop that may run fifty times. ML.ai Code ships an experimental code mode that has the model call tools by writing a short program instead of receiving every schema each turn, which its documentation measures at roughly 1,700 tokens saved per turn. On a fifty-turn session, that is 85,000 tokens you never paid to send, with no model change and no quality trade.

Reasoning effort is the same lever from the other end. If your agent runs at maximum effort on a two-line edit, you are buying reasoning tokens nobody reads.

4. Compute your ceiling and decide whether to continue

Now you have real numbers. Take f from your task-type distribution, take r from the two models you would actually move between, and multiply. If S comes out under 15%, stop here. A router is a permanent piece of infrastructure with its own failure modes, and 15% of a small bill does not justify it. Spend the time on step 2 instead.

5. Route at the boundaries of work, not the boundaries of turns

This is where most implementations leak their savings.

Sub-tasks, sub-agents, and phases are sticky. Individual turns are not.

Good routing boundaries are the ones a session does not cross back over: a whole sub-task handed to a sub-agent, a distinct phase of work, a background job. NVIDIA's stage router uses this idea directly, reading recent tool activity to judge whether the agent is still exploring (needs capability) or has settled into mechanical implementation (does not), and their LLM classifier "maintains session affinity with that model across later turns" specifically to avoid re-deciding work that has not changed.

Read-only work is the easiest win here and the safest. A search-and-explain pass over a repository writes nothing, so a wrong model choice costs a re-run and nothing else. Handing that to a cheap, read-only sub-agent is a routing decision with almost no blast radius.

ML.ai Code's Explore agent is scoped exactly this way, and its Architect and Plan agents return an order of work without touching files, so only the General agent can change code. Choosing which agent gets a job is routing, done at a boundary that sticks.

6. Gate every route with an eval you own

A router without an eval is a guess with a dashboard. Build a small golden set from your own traffic, per task class, and require the cheap model to clear a stated bar before that class becomes routable. Then keep running it, because model versions change underneath you.

DigitalOcean's conference demo of their inference router is a good model of the discipline. Running the same coding tasks through a router against Opus directly, session cost came in at 14 cents versus 44 cents.

Their engineer was explicit that the side-by-side output comparison was "a vibe check" and that "how you actually prove it is working it through evaluations," where the router scored 90% correctness against Opus at 95%. Note the honesty: a 68% saving, and five points of correctness.

Where does model routing break?

Three failure modes account for most of it, and only one is about model quality.

Difficulty is invisible at routing time

The IBM team put this way: a request like "summarize this contract" looks simple but may trigger retrieval, compliance checks, tool use and several rounds of refinement, while a highly technical prompt may be handled efficiently by a small specialist. You often don’t know how hard a task is until execution is underway.

The naive fix fails in a specific way. In a walkthrough where an engineer built a keyword-and-length classifier, the prompt "let's build a game in Python" was classified as a simple task, because it was short and the length weighting outvoted the coding keywords.

Short prompts requesting enormous work are the exact case where keyword routing gets backward, and coding traffic is full of them.

The router optimizes one variable, and production has five

Routers in production juggle cost, latency, specialization and reliability at once. Enterprise deployments pile on more, as the IBM write-up lists: compliance requirements, data residency rules, privacy constraints and approved model lists. A task that should go to one model may be legally required to go to another. IBM's summary of their own work is that they "stopped treating routing as a classification problem and started treating it as an optimization problem," which is a heavier lift than a classifier and the reason most homegrown routers plateau.

Automatic routing takes control away from the person doing the work

When OpenAI shipped GPT-5 with an automatic router over its fast and thinking modes in August 2025, the reaction from users was loud enough to become a news story. Developers in particular dislike opaque model selection, because when output quality drops they cannot tell whether the model changed, the prompt changed, or they made a mistake.

The lesson is not to avoid routing. It is that the routing decision has to be visible and overridable. A router that logs which model it chose, why, and what it cost is debuggable. One that does not is a source of mystery regressions.

What does this look like inside a coding agent?

It looks like controls built into the agent itself rather than a proxy sitting in front of it.

That framing is not ours. When a reader commented on the IBM post that "model routing is a false proposition; it should have been part of the harness all along," Yara Rizk of IBM Research replied: "Agreed, routing should disappear into the harness and just be part of the system's execution logic. Our goal is exactly that, making model choice an intrinsic capability rather than a separate concern."

The argument is straightforward once you have read the cache arithmetic. A gateway sitting outside your agent sees a stream of requests. It does not know that turns 4 through 11 are one sub-task, that the prefix is cached, that the agent is mid-exploration, or that the next tool call will invalidate everything. The agent knows all of that. It is the only layer that can route at a boundary that sticks.

ML.ai Code is built at that layer: a coding agent inside VS Code with the cost controls exposed as decisions you make, rather than hidden behind an automatic mode.

Reasoning effort is set per message across five levels, so a two-line edit does not buy reasoning tokens nobody reads. Plan mode investigates and produces a written plan with the edit and write tools switched off, so exploratory work cannot quietly become expensive execution. Explore, Architect, and Plan agents are read-only by construction, and only the General agent can change code, which fences the cheap, high-volume half of agent work off from the half where a mistake is expensive.

Delegated jobs run inline or in the background with their scope pinned for the whole job, which is the stickiness gate from the diagram above, enforced by the architecture rather than by a heuristic. Durable memory carries project context across sessions, so the same explanations are not re-sent and re-billed every time.

It is one package, one token, and it runs against whichever provider you already pay, which is the point of the title. You do not have to leave Anthropic or OpenAI to spend less with them.

If you are running high-volume LLM development work and want the cost curve to bend without the output getting worse, that is the problem this was built for. Install Ml.ai Code from the VS Code Marketplace, run step 1 above for a week, and you will have a real value for f before you commit to any router at all.

Conclusion

You now have the arithmetic that predicts your ceiling, the reason a downgrade can cost more than staying put, and the order of operations that keeps you from building a router you did not need. The first step is small: add cached-input tokens and task type to your request logs and leave it running for a week. That one field is what separates a real value for f from a guess, and its absence is why IBM's GPT-4.1 estimate came in at half the true cost.

Then take the free levers. Cache the static prefix, move any asynchronous work to a batch endpoint, and check whether your agent is using maximum reasoning effort on work that does not need it. If a ceiling worth chasing is still there afterward, route at the boundaries of work and gate it with your own evals.

If you would rather have those controls sitting in the editor where the work actually happens, ML.ai Code puts reasoning effort, plan-only mode, read-only agents, and scoped delegation in front of you on every turn, against the provider you already use.

Frequently Asked Questions

Is model routing worth it for a small team?

Usually not as the first move. Compute S = f × (1 − r) from a week of logs first. Below roughly 15%, prompt caching, batch processing and off-peak scheduling deliver more saving with no quality risk and no new infrastructure to maintain.

Does model routing hurt output quality?

Yes, measurably, in every published result that reports both numbers. The costs range from 2.8 accuracy points on Cognition's coding benchmark to 6 points on NVIDIA's agent suite. The question is not whether quality drops but whether the drop is acceptable for that task class, which is why the eval gate in step 6 is not optional.

How much latency does a router add?

Very little. Classifier-based routers add roughly 50 to 100 milliseconds against model responses that take 500 to 2,000 milliseconds; DigitalOcean measures its routing model under 200 milliseconds, and IBM measured theirs at about 6 milliseconds per task. Latency is the weakest objection to routing. Cache invalidation is the strong one.

Can I use routing without switching model providers?

Yes, and that is the common case. Most of the saving available to a coding team comes from moving between tiers inside one provider's lineup (Opus to Sonnet to Haiku, or sol to terra to luna) plus caching and batch discounts. All of those are billed on the API keys you already have.

What is the difference between model routing and model cascading?

Routing decides before generating, using the prompt alone. Cascading generates a cheap answer first, scores it, and escalates when the score is low. Cascading judges difficulty more accurately because it has seen an attempt, and costs more because some requests get answered twice. Most production systems now combine them.

Why did my cheaper model increase my bill?

Almost certainly a cache miss. Switching models forces the new model to re-process the entire shared prefix at cache-write prices, which at current Anthropic rates is 1.25x base input, against 0.1x for the read you abandoned. On a 100,000-token context, the switch turn costs 73% more than staying put, and only pays back if the new model holds the session for two or more turns.

Should I build a router or buy one?

Build the measurement, then decide. Steps 1, 2, and 4 are the same either way and produce the number that tells you whether a router is worth owning. If you do build, start with pre-generation routing on a small set of task classes, because it is cheap to run and easy to reason about, and add cascading only where you have evidence the classifier gets it wrong.