Three cost levers dominate LLM API spending, and they work on different parts of the bill. Prompt caching discounts the prefix you keep re-sending, Caveman-style output compression shrinks the words the model writes back, and tiered model routing sends cheap requests to small models instead of flagship ones. Each lever has a credible headline number — 90% off cached inputs, 65% off output tokens, 50–80% off routed traffic — but those numbers live on different sides of the invoice. This article compares the three against current Anthropic and OpenAI list prices, names where each one wins and where it goes negative, and lays out the order in which they actually stack. The goal is to pick a strategy that fits a workload, not to bolt every gadget onto every agent.
Each lever targets a different part of the invoice, so a compact comparison up front prevents apples-to-oranges math later.
| Lever | Token side it touches | Headline savings band | Implementation effort | Failure mode that wipes the savings |
|---|---|---|---|---|
| Prompt caching | Input prefix only (cached reads billed at ~10% of input on Anthropic) | 50–90% off cached input tokens | Low — opt in with a cache_control marker on Anthropic; automatic at ≥1,024 tokens on OpenAI | Prefix drifts even slightly (system prompt rewrites, dynamic timestamps, reordered tool schemas) and the cache miss collapses the discount to zero |
| Caveman-style output compression | Output tokens only | 22–87% off output, ~65% mean (across function explanations, error debugging, code reviews) | Low — install a skill/style rule | Most of a session is input (conversation history, file reads, tool schemas); in a typical 100k-token Claude Code session output prose is only ~6%, so overall bill cuts often land at ~4–15%, not 65% |
| Tiered model routing | Both sides — swaps the per-token rate entirely | 50–80% blended savings when70–80% of traffic routes to budget tier | Medium — needs a router (rule-based, classifier, or cascade) | Misclassification sends hard reasoning to a small model; quality collapse is silent and shows up only in evals |
The percentages above translate into real money only against current list prices. Anthropic's June 2026 rates, per million tokens (input / output / cache read), are:
Cache writes cost extra — 1.25× input for a 5-minute cache, 2× for 1-hour — so caching only pays off once the prefix is reused roughly1.4 times or more. Routing is the only lever that touches the rate itself; sending a request from Opus 4.8 to Haiku 4.5 changes the input cost from $5/MTok to $1/MTok and the output cost from $25/MTok to $5/MTok — a 5× shift on every token, cached or not.
The 90%, 65%, and 50–80% figures live on different denominators: cache reads are a subset of input tokens, Caveman savings apply to output tokens, and routing savings are blended across the whole request. A workload that achieves all three does not pay 90% × 35% × 25% of the original bill. Stacking order matters, and so does the input-to-output mix of the workload — that is what the next sections break down.
Prompt caching is the only cost lever that touches the input side of the invoice at a structural level. Both Anthropic and OpenAI store the key-value state of a stable prompt prefix server-side, so subsequent requests skip reprocessing those tokens and pay a fraction of the normal input price instead.
On Anthropic, the numbers are explicit (Anthropic cost-optimization cookbook):
OpenAI runs caching automatically with no code change: it activates once the prompt prefix exceeds 1,024 tokens and discounts cached input tokens by 50% (2026 token-usage playbook).
The savings are silent. There is no banner, no line item named "cache discount" — the only signal is a cached_tokens field inside the usage object. Teams that never inspect that field ship agents whose hit rate is 7% and never know the cache was available (NeuralTrust token-optimization guide).
ProjectDiscovery's Neo agent is the cleanest published case study: cache hit rate climbed from 7% to 84%, cumulative spend dropped 59%, and reached over 90% on fully optimized paths — from prompt structure alone (2026 token-usage playbook). For a Sonnet 4.6 bug-fix task, the same prompt cost $0.54 with caching versus $1.35 without — roughly a 2.5× difference from one config flag (AI coding costs in 2026).
Caching is prefix-match, not semantic-match. The prompt has to be assembled in a specific order:
CLAUDE.md). These rarely change.On Anthropic, you declare the boundary with a cache_control breakpoint. Auto caching places one breakpoint that slides forward as the conversation grows; explicit mode allows up to four breakpoints so different layers can have different staleness rates. On OpenAI, the1,024-token threshold plays the same role automatically (fast.io AI agent token cost optimization).
The most common failure mode is a timestamp, request ID, or username embedded in the system prompt. The prefix looks identical to a human, but the cache key diverges on every call and cache_read stays at zero (Anthropic cost-optimization cookbook).
Caching only helps when the prefix is reused. Single-turn, no-shared-context workloads — one-shot classification calls, fresh-document summarization with no system prompt — see zero benefit. For workloads that do repeat a prefix, the effort is a parameter change rather than a model swap, which is why this lever is almost always pulled first (MindStudio token-reduction strategies).
Caveman is a SKILL.md-style instruction set you drop into an agent's system prompt. It tells the model to speak in a stripped, telegraphic register: drop preambles, hedging, and structural scaffolding, but keep code blocks, shell commands, file paths, and error strings byte-for-byte exact. That last clause is the load-bearing one — the skill is not a lossy text compressor; it's a style constraint that targets only the parts of the reply no machine downstream needs to parse.
Across documented benchmarks the skill cuts output tokens by ~65% on average, with a measured range of 22% on already-tight answers up to 87% on verbose explanations like React bug walkthroughs and function overviews (JuliusBrussee/caveman, 8labs benchmarks). A March 2026 study found that constraining models to brief responses improved accuracy by 26 points on certain tasks, suggesting the filler it removes is mostly noise rather than signal (Graphify vs Caveman, Medium). Real-world MakeUseOf testing reported a more modest 52% output reduction in practice — still substantial, still less than the headline.
The project's own docs are unusually candid about where the savings evaporate (caveman HONEST-NUMBERS.md):
Worked example on Opus 4.8 at the article's stated $25/MTok output price: a 2,000-token output session trimmed by 65% saves roughly $0.022 (1,300 × $25/MTok). The 1.5k input tax at, say, $5/MTok input adds about $0.0075. Net is positive — but it's nowhere near the 65% headline, and the ratio worsens on smaller outputs where the fixed input overhead dominates.
Caveman is a workload-specific lever, not a blanket discount. It wins when the agent writes too much English: long explanations, chatty commit messages, recap-heavy debugging narration, polite transitions. It loses on tight, mechanical outputs — short refactors, yes/no answers, single-line fixes — where there is little filler to strip and the 1–1.5k input tax can flip the turn net-negative. Pair it with the proxy variant (/caveman:compress) only if your workload also has bloated memory files; the skill itself does not shrink input context.
Caching and compression both attack the volume of tokens billed. Routing changes the rate itself. With Opus 4.8 at $5/$25 per million input/output tokens and Haiku 4.5 at $1/$5, a routine formatting task sent down a tier costs a fifth as much on every single token — input and output alike (morphllm.com). On a one-million-token workload the gap can reach roughly 30x on output tokens and over 100x during promotional windows. That is why routing shows up as the largest single line item on most LLM invoices.
The headline evidence is Berkeley's FrugalGPT paper, which reported up to 98% inference cost reduction at GPT-4 Turbo parity on the HEADLINES benchmark (programstrategyhq.com). That ceiling is benchmark-shaped, not workload-shaped. The more honest real-world band sits at 50–85% blended savings, because routing a hard prompt to a small model and paying to re-run it on a big one erases the discount. Empirically, 70–80% of production traffic is routable to budget-tier models; the remainder genuinely needs frontier reasoning.
A simple if/else on prompt length or task keyword captures most of the savings. The next rung up is LiteLLM's cost-based-routing strategy, which picks the cheapest healthy deployment by per-token price after applying rpm/tpm filters (docs.litellm.ai). LiteLLM also ships request-aware routers — auto, complexity, quality — that route by request content rather than only by endpoint price (medium.com).
Stacking matters: when prompt caching and routing are combined, LiteLLM reports roughly 69% cost savings, so the two levers do not cancel (docs.litellm.ai).
The three levers stack in a specific order, and the order matters because each one pays for the next. Prompt caching first, model routing second, Caveman third.
Caching is essentially free to switch on. On Anthropic, it requires a single cache_control marker at the end of your stable prefix; on OpenAI, prompts of 1,024 tokens or more are cached automatically with no code change (Anthropic cookbook; PointFive 2026 guide). There is no quality trade-off, because cache reads are exact-match at the provider layer. Anthropic bills cache reads at 10% of the base input rate (writes at 1.25× for a 5-minute window, 2× for1-hour), breaking even after roughly 1.4 reads (2026 token-reduction playbook).
Routing second, because it changes the per-token rate from the top of the table to the bottom, not just one line of it. It needs either a gateway — LiteLLM (self-hosted, ~3–5 ms overhead), Portkey, or OpenRouter — or a simple classifier prompt that decides whether a request needs a flagship model or a small one (LiteLLM routing docs; gateway comparison).
Caveman last, because it has a narrow workload fit. It is a SKILL.md drop-in that changes register without changing capability, and benchmarks report a mean output reduction of about 65% —22% on already-tight answers, up to 87% on verbose explanations — while preserving technical accuracy (skillsllm; Testing Academy).
Take500K input tokens (40% cached) and 200K output tokens. Sonnet 4.6 list pricing: $3.00/MTok input, cache read $0.30/MTok.
The compounding caveat from the playbook literature: semantic + prefix caching can route 70–80% of tokens through a cheap layer, but mixing compression with caching naively — for instance, running caveman:compress on CLAUDE.md while the prefix is mid-cache — can change the cached bytes and bust the cache key, costing more than it saves (2026 playbook). Cache the original prefix, compress only at output time or on a stable post-cache suffix.
The three levers are not interchangeable, and the right starting point depends on what kind of traffic dominates your bill.
Multi-turn, prefix-heavy sessions on a single frontier model. A typical Claude Code or Cursor session re-sends the same system prompt, tool schemas, and CLAUDE.md content on every turn. That is exactly the shape Anthropic's prompt caching is built for: cached input tokens are billed at 10% of the base input rate on Anthropic and at 50% on OpenAI's standard caching (with newer flagship models matching the 90% discount) Anthropic cookbook, PointFive2026 guide. The write premium (1.25× for a 5-minute cache, 2× for 1-hour) breaks even after roughly 1.4 reads, so almost any agent loop pays back immediately. This is the cheapest first move: a single config flag with no quality trade-off.
Heterogeneous workloads with multi-model API access. Most products are a mix of trivial and hard calls. Routing sends classification, extraction, formatting, and short summaries to Tier-1 budget models (Claude Haiku, GPT-5-mini-class) and reserves the frontier tier for genuine reasoning. In practice 70–80% of production traffic is routable to budget tiers, and a simple classifier or even keyword rules are enough to capture most of the savings servicesground.com, neuraltrust.ai. This is the largest single per-token swing once the prefix problem is solved.
Verbose-output agent loops. Install Caveman only after profiling confirms the agent writes substantially more English than the task requires. Benchmarks show roughly 65% output-token reduction (range 22–87%) with no loss of technical accuracy, but output prose is only about 6% of a typical 100k-token session aifordevelopers.substack.com. The realistic session-level saving lands in the 4–15% range, and you must net it against the additional input tokens the caveman-style instructions add to every prompt (~1–1.5k tokens of instructions) before declaring victory.
One-shot, single-turn, low-context calls. Short customer-support replies, embedded classification, and other narrow tasks have no prefix to cache, no routing decision worth the latency, and output volumes too small for compression to matter. All three levers underperform here, and the right move is simply to push the workload to a smaller model outright.
The unifying principle: pick the lever that attacks the biggest line on your bill, not the one with the loudest percentage.