How to reduce LLM API costs without losing output quality

You reduce LLM API costs without losing output quality by sorting every lever into one of three groups first: the ones that cannot change the model's answer, the ones that change what work happens, and the ones that change what the model can do. Take them in that order.

Anthropic's Applied AI team ran exactly that sequence on a production-shaped agent and landed 13x cheaper at an unchanged 10 out of 10 pass rate, with model choice deliberately last on the list.

In this blog, you'll find a classification of every common lever by whether it can change your output, the measured quality cost of each lossy one, the eval that tells you which category you are in, and the four ways a working saving decays after you ship it.

What is the fastest way to reduce LLM API costs without losing quality?

Cache the static prefix, then batch anything nobody is waiting on. Those two moves are free of quality risk by construction, and on published production numbers they are worth more than model switching.

The reason is mechanical. Prompt caching stores the key-value tensors for a prefix the model has already processed, so a repeat request skips prefill and reads the stored state instead. The model receives the identical token sequence either way. Anthropic's caching documentation prices a cache read at 0.1x base input against a 1.25x write premium for the five-minute tier. The Batch API is the same idea applied to scheduling: Anthropic and OpenAI both discount asynchronous requests 50% for identical tokens, and DeepSeek halves its rates during defined off-peak windows.

Neither one touches the prompt, the model, or the sampling. Your output distribution is unchanged, which is why they're the right first move.

The three classes, and why the order is not arbitrary

Every cost lever falls into one of three groups, and the group decides how much evidence you need before shipping it.

Class 1: lossless. The model sees the same bytes. Caching, byte-stable prefixes, explicit breakpoints, batch, off-peak scheduling. No eval required, because there is nothing for an eval to catch.

Class 2: structural. You change what work happens: fewer steps, a read-only subagent absorbing a bulky tool result, a tighter output shape. The model's capability per step is untouched, but the information reaching it changes, so a regression check is warranted.

Class 3: lossy. You change what the model can do: a smaller model, lower reasoning effort, a compressed prompt, a semantically matched cached answer. Every published result in this class that reports both numbers shows a quality cost. These need a stated bar and a gate.

Anthropic's own summary of the same idea is worth internalizing: caching changes what you pay for tokens you were already sending, input and loop management change what the model sees, and effort and model change what the model can do. That is why the last group goes last.

Why does most cost advice skip the quality half of the promise?

The research reports both numbers. The summaries drop one. The primary literature doesn't have this problem. Microsoft Research's LLMLingua project page reports a 20x compression ratio with minimal performance loss for the original method, and a 17.1% performance improvement at 4x compression for LongLLMLingua. LLMLingua-2, published at ACL 2024 Findings, reports end-to-end latency gains of 1.6x to 2.9x at compression ratios of 2x to 5x, evaluated across MeetingBank, LongBench, ZeroScrolls, GSM8K, and BBH.

Notice the shape of those claims. A compression ratio always arrives attached to a task metric on a named benchmark. The compression ratio makes its way into blog posts. The task metric usually does not, and the 2x to 5x working range quietly becomes "up to 20x" somewhere in the retelling.

Which cost levers cannot change your output at all?

Four of them.

Prompt caching, and the one field that breaks it

The security team at ProjectDiscovery published their own before-and-after in April 2026. Their agent Neo runs 20 to 40 LLM steps per task on top of a large system prompt, and their cache hit rate was sitting at 7% because dynamic working memory lived inside that system prompt and invalidated the cacheable prefix on nearly every step.

Moving that content out of the prefix took them to 74% in a single deployment. Explicit breakpoint placement and deliberate TTLs took them to 84%, cutting total spend 59% to 70% across 9.8 billion cached tokens. Same models, same prompts, same answers.

Cache hit rate sets the effective price of tokens you were already sending. Marked points are published production figures.

The chart is the argument. At a 7% hit rate, you pay $4.68 per million input tokens on Opus 5 list rates. At 99.3%, the rate a single-developer session on one repository reached in the 28-day coding-agent study published as arXiv:2607.13080, you pay $0.53. Same model, same work, a factor of nine in price, decided entirely by where you put your variable content.

The four things that break it are all self-inflicted: anything dynamic above the breakpoint, tool definitions that change between calls, a TTL shorter than the gap between requests, and rewriting conversation history mid-session.

The tokenizer footnote that inverts a price comparison

Before you migrate anything on the strength of a rate card, run a fixed corpus through both tokenizers and compare token counts rather than prices. Anthropic's pricing documentation states that Claude 4.7 and later use a newer tokenizer producing roughly 30% more tokens for the same text.

A model priced 25% lower per token that emits 30% more tokens for identical input is not cheaper. It is slightly more expensive, and your dashboard will agree with the wrong answer.

Batch and off-peak: the discounts sitting in public documentation

Batch is a 50% discount on identical tokens for a 24-hour delivery window, and caching still applies inside a batch, so the discounts stack. It is unavailable for anything interactive, which is the entire catch. Send your evals, backfills, documentation generation, and regression sweeps there. Keep your inner loop synchronous.

Anthropic's cookbook demonstrates this cleanly: ten triage requests run synchronously came to $0.7379, and the same ten as a batch came to $0.3653, with the token columns matching exactly.

Which levers change what the model sees, but not how well it thinks?

The structural ones, and this is where the largest savings in the published data actually live.

Keep bulky tool output out of the parent context

This is the highest-value move in the whole category, and it is not a billing setting.

In Anthropic's cost optimization cookbook, a 5,000-row ledger lands in an agent's context on turn two and sits there for the rest of the run, pushing every subsequent turn past 148,000 tokens. Running the same three-claim queue with one Haiku subagent absorbing the ledger and returning a single line brought the total from $1.7945 to $0.3978, a 78% cut. A client-side prune at claim boundaries got 29%. Server-side context editing got 12% and compaction 16%.

The difference between 12% and 78% is not the technique. It is whether the bulky result ever enters the parent conversation in the first place.

Jason Zhou hit the same mechanism from the other direction while building a research agent. His web-scraping tool was returning raw page content, noise included, straight into agent memory. Adding a cheap-model summarization pass inside the tool function before the result was returned took the expensive model's input from 20,000 tokens to 4,300, roughly 70% off the run. His verdict on the output was that the new result was better than the original, because the filtering pass removed noise the expensive model had been reasoning through.

Worth sitting with. A cost reduction improved the answer, because context volume and context quality are not the same thing.

Fewer steps beat fewer tokens per step

Input tokens in an agent loop scale with the square of the step count, because step N re-sends everything from steps 1 through N-1, while output scales linearly. Our sibling piece on where LLM inference cost actually goes works the arithmetic in full, so the short version here: halving step count from 40 to 20 cuts input tokens by 65%, not 50%.

This makes agent planning behavior a cost feature. A planning pass that produces the order of work before any file is touched converts an expensive exploratory loop into a short one, and the saving compounds against the quadratic term. The 2025 survey of agentic programming techniques treats planning and context management as core architectural concerns for exactly this reason.

Shape the output instead of capping it

max_tokens is a backstop, not a tuning knob. The model never sees the value, so hitting the ceiling truncates a response mid-thought rather than producing a shorter one.

The lever that works is specifying the exact output shape in the prompt, ideally with an example. Anthropic's cookbook measures an open-ended adjudication memo at 4,096 output tokens and $0.1585, against a one-line shaped response at 61 output tokens and $0.0580, a 63% cut at the same max_tokens. Registering a stop-sequence sentinel for cases where the right answer is to bail early saved a further 56% on malformed input.

Stop paying to send pixels you don't need

Images and PDFs are tokenized by pixel area, so cost scales with resolution rather than with information content. Downscaling a claim photo from 2048x1536 to 960x720 took it from 4,088 tokens to 928 while leaving the visual question answerable. Moving a 5,000-row CSV out of context and into a code-execution sandbox, so only the computed answer returns, came in 79% cheaper than pasting the file inline.

Which levers actually trade quality for cost, and what does the trade cost?

All of them in this class, and every published result that reports both numbers puts a figure on the loss. The question is never whether the drop exists. It is whether the drop is acceptable for that task class.

Rebuilt from the trial table published in Anthropic's cost optimization cookbook. Green configurations hold 10 of 10. Coral ones do not, at any price.

That chart is the single most useful artifact in this article, because it plots the two numbers together on a real workload. Read the coral points as a catalogue of tempting mistakes.

Model downgrade

The cookbook's ladder holds a perfect pass rate down through Sonnet at medium effort, then starts slipping at Sonnet low (9 of 10, then 8 of 10). Haiku alone lands at a 55% mean pass rate across two trials. The cheaper tokens are real. So are the wrong answers.

Our piece on the routing ceiling collects six published routing results, and every one of them carries a quality cost alongside its savings, from 2.8 accuracy points on a coding benchmark to 6 points on an agent suite. If a vendor quotes a savings figure with no accuracy figure beside it, the accuracy figure exists, and you are simply not being shown it.

Semantic caching

Semantic caching returns a stored answer when a new query is close enough in vector space. The failure mode differs from prompt caching in a way worth stating, because the answer you get back was generated for a different question.

The AWS ElastiCache team demonstrated the failure live in a February 2026 walkthrough. "Recommend black shoes" gets answered and cached. "Recommend white shoes" is one word apart and lands almost on top of it in vector space. On similarity alone, the second query returns the first query's answer.

Fast, cheap, and wrong, with no error raised and no log line written. The fix is to require extracted state to match, not just the vector.

The fix they demonstrate is to extract structured state from the query before lookup (colour, size, price band, product line, turn intent) and require those to match as filters alongside the vector. They also mark anything personalized, time-sensitive, or carrying PII as never-cacheable at write time.

Their cost arithmetic deserves the same honesty. On 100,000 daily queries, their no-cache baseline came to $945 per day. A 25% hit rate brought model spend to $709, but the vector store to hold the cache added roughly $23 to $24 per day, landing the real total at $732. That is a 23% net saving, not the 86% in the video title. The vector store is a line item, and none of the 12 pages in the census above accounts for it.

Flattening rules into a condensed prompt

The cheapest configuration in Anthropic's entire sweep was five Haiku subagents gathering facts and one Sonnet call deciding against a condensed rule card, at $188 per 10,000 tasks, roughly 90% under the Opus baseline. It cleared every escalation case and failed the same routine claim in both trials, because the rule card carried an exclusion but not the exception the full manual attaches to it.

Their conclusion, in the cookbook itself, is the sentence I would put on the wall of any team running this exercise: "Cheap and slightly wrong is still not an optimization at our bar."

The lever that cost money and quality at once

Worth naming, because it breaks the assumption that a smaller prefix is always cheaper. Deferring tool schemas via tool search dropped the prefix from 2,125 tokens to 1,696, and the eval came back at 8 of 10 and 9 of 10 across two trials, at $543 and $537 per 10,000 tasks against $500 for the configuration it replaced. Worse and more expensive, because the search step added more tokens than the deferral removed on a workload with only a dozen small tools.

A smaller prefix does not always mean a cheaper task. Only the eval tells you which one you have.

How do you build the eval that makes any of this safe?

Two measurements, twenty to fifty tasks, and one rule about changing things one at a time.

Measure pass rate and cost per completed task together

Cost per token is the wrong unit. A model with a higher sticker price that finishes in fewer turns can be the cheaper option end to end, which is why IBM Research found Claude Sonnet cheaper than GPT-4.1 on the same 417 agent tasks despite worse published rates and three times as many reasoning steps.

So roll usage up per completed task and read it beside a pass rate. Anthropic's guidance on building eval suites for agents is the reference here, and the bar for getting started is low: ten claims with human-labeled correct verdicts was enough to catch every regression in their cookbook.

Set the bar before you optimize, not after

Decide the pass rate you need from baseline runs, treat it as a fixed constraint, and minimize cost underneath it. Reversing that order produces a number you cannot interpret, because you have no way to tell a saving from a regression.

Run more than one trial

Model outputs are nondeterministic, so the same configuration lands at different pass rates and different costs from run to run. In the cookbook's own sweep, Haiku scored 7 of 10 on one trial and 4 of 10 on the next with nothing changed. Their recommendation before a production decision is around fifty eval cases and at least five trials per configuration, enough that adjacent configurations stop trading places between runs.

Change one variable at a time

Steps 2 and 3 are where the money is. Step 4 is the only step that can lower your intelligence ceiling, which is why it goes last.

Before any of it, pull four numbers from a week of logs: your cache read, cache write, uncached input and output tokens per request, and your step count per completed task. The cached-input field is the one teams skip and the one that decides everything above it.

What breaks after you ship the savings?

Four things, and all of them are quiet. None raises an error, and none shows up as anything other than a slow drift in a number nobody is watching.

Cache hit rate decays: It is enabled once and treated as a setup task rather than a production metric. Someone adds a timestamp to a system prompt eight months later, the hit rate falls from 84% to 40%, and the only symptom is a bill that grew for a reason nobody can name. Track the ratio of cache-read tokens to total input tokens per workload, and alert on the ratio rather than on the total.

Semantic caches serve the wrong answer: Covered above. Cache hits are correct until the day one word carries the whole question.

Tool schemas grow: Schemas are re-sent on every request, so every tool anyone adds is a fixed tax multiplied by your step count. This is the failure mode that MCP servers produce fastest, because connecting one server can add thousands of tokens to every turn of every loop.

Runaway loops: Jason Zhou opened a video by describing an email from OpenAI on 1 December telling him he had hit his monthly usage limit, on the first day of the month. Two of his sales agents had reached each other and begun replying back and forth. In his words, "we burn $5,000 USD in a single Friday afternoon." No optimization survives an unbounded loop, which is why a per-agent spend cap belongs in the same commit as your first caching change.

What does this look like inside a coding agent?

It looks like the controls sitting where the work happens, rather than in a proxy that cannot see what your agent is doing.

A gateway in front of your agent sees a stream of requests. It does not know that turns 4 through 11 are one sub-task, that the prefix is warm, that the agent is still exploring, or that the next tool call is about to invalidate everything. The agent knows all of that. It is the only layer that can make a routing decision at a boundary that holds.

ML.ai Code is built at that layer, as a VS Code and Cursor extension that picks the most cost-efficient model for each step of a task rather than paying the frontier rate for classify, extract, draft, and verify alike. On its published per-task breakdown, that is $0.43 against $0.70 for the same completed work.

The controls map onto the three classes directly:

The read-only fencing is the part worth dwelling on, because it is a cost mechanism disguised as a safety mechanism. A search-and-explain pass writes nothing, so a wrong model choice on it costs a re-run and nothing else. That makes it the lowest-risk routing decision available to you, and on the numbers above it is also one of the largest.

If your team is running high-volume LLM development work and wants the bill to bend without the output getting worse, install ML.ai Code and run the measurement in step 1 for a week. You will have real values for your cache hit rate and your step distribution before you commit to anything architectural.

For workloads beyond the editor, ML.ai Inference applies the same ordering across your whole traffic mix, with your own evals as the acceptance test rather than a vendor benchmark. The pilot agrees the cost, latency, and quality targets in writing before anything moves.

Conclusion

You now have the three classes, the measured quality cost of the levers in the third one, and the eval that tells you which class a change belongs to.

Start with one query against a week of usage logs: cache reads over total input tokens, per workload. If that number is under 50%, dynamic content is sitting in front of your static prefix, and moving one field is worth more than every model decision you have been debating. That takes an afternoon and needs no architectural change to measure.

Then work left to right, one change at a time, re-running your eval after each. If you would rather have plan-first modes, read-only delegation, and per-turn effort sitting in the editor where the work happens, that is what ML.ai Code was built for: the same shipped answer, a different model per step, a lower bill.

Frequently Asked Questions

How much can I realistically cut my LLM API bill without any quality risk?

Enough that it is worth doing before anything else. The published production figures for prompt caching alone run from 59% to 70% of total spend, and the Batch API is a flat 50% on anything asynchronous. Both are lossless by construction, so the honest answer for your workload depends entirely on your current cache hit rate, which you can find with one query against your usage logs.

Does prompt caching change the model's output?

No. The cache stores the key-value tensors for a prefix the model has already processed, so it skips recomputation rather than approximating it. The result is identical to an uncached request with the same prompt. What changes is latency and the price of the cached portion, billed at 0.1x base input across the major providers.

Is semantic caching safe to use in production?

Only with state filters and a write-time policy about what is cacheable. Vector similarity alone will return one query's answer for a different query when the two differ by a single decisive word, and the failure raises no error. Extract structured facts from the query, require them to match, and mark anything personalized, time-sensitive, or carrying PII as never-cacheable.

Should I use a cheaper model to reduce LLM API costs?

Last, and only against an eval with a stated pass rate. Model choice is the easiest lever to pull and the only one that directly lowers your intelligence ceiling. Take caching, batching, step reduction, and output shaping first, then step down one tier at a time and re-measure, because published results consistently show a measurable accuracy cost at each step.

How many eval cases do I need before I trust a cost change?

Ten labeled tasks is enough to catch obvious regressions while you experiment. Before a production decision, aim for around fifty cases and at least five trials per configuration, because agent outputs are nondeterministic and a single run can move a pass rate by several points with nothing changed.

Why did my bill go up after I switched to a cheaper model?

Almost certainly a cache miss. Caches are per-model, so switching mid-conversation forces the new model to reprocess the entire shared prefix at cache-write rates while you abandon a warm read. On a large context, the switch turn can cost more than staying put, and only pays back if the new model holds the session for several turns.

Does prompt compression lose quality?

At high ratios, yes, and the research says so plainly. The published working range where faithfulness holds is roughly 2x to 5x compression, not the 20x figure that circulates in summaries. Treat the headline ratio as a ceiling, run your own task metric alongside it, and remember that the compressor itself costs a model call.