1. Route by task. Stop sending everything to the frontier model.
Most teams call GPT-4 or Claude Opus for every single request; classification, extraction, simple Q&A, complex reasoning, all of it. That's the single biggest waste in production LLM spend.
Researchers at Stanford formalized this problem back in 2023 with a framework called FrugalGPT. The idea: cascade queries through cheaper models first, and only escalate to the expensive model when the cheap one isn't confident. Their result was a 98% cost reduction while matching GPT-4's performance on the same tasks. Since then, routing has become its own research category. RouteLLM showed learned routers can deliver 2x-plus cost cuts by training on preference data instead of hand-written rules. Amazon's Bedrock Intelligent Prompt Routing gets roughly 30% cost reduction just by routing within a single model family.
The newest twist doesn't even need an LLM as the router. A recent open-source project called TRACER takes this further for classification workloads specifically. It watches your LLM's own classification traces, then trains a lightweight traditional ML model (logistic regression, gradient boosting, a small neural net) to handle the "easy" cases with a formal accuracy guarantee against the LLM it's learning from. On the Banking77 benchmark, it routed 92% of traffic away from the LLM entirely while keeping 96%+ agreement with it, cutting a $20/day workload to $1.73/day. It gets smarter as it sees more traffic, and every deferred call becomes a new training example.
The takeaway: you don't need one model for your whole product. You need a router that knows which requests are actually hard.
Sources:
- Chen, Zaharia & Zou, FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (Stanford, 2023) — arxiv.org/abs/2305.05176
- Ong et al., RouteLLM: Learning to Route LLMs with Preference Data (2024) — arxiv.org/abs/2406.18665
- Amazon Bedrock Intelligent Prompt Routing, cited in Unsolvability Ceiling in Multi-LLM Routing (2026) — arxiv.org/pdf/2605.07395
- Rida, TRACER: Trace-Based Adaptive Cost-Efficient Routing for LLM Classification (2026) — arxiv.org/abs/2604.14531 · github.com/adrida/tracer
2. Cache by meaning, not by exact string match.
Traditional caching only helps if someone asks the exact same question twice, character for character. Nobody does that. But loads of your traffic is asking the same thing in different words. "What's my balance?" and "How much money do I have?" are the same query wearing different clothes.
Semantic caching solves this by embedding the query, checking it against past queries by meaning, and returning the cached answer if there's a close enough match. GPTCache, the open-source project that popularized this, has been shown to increase response speed 2-10x on a cache hit, on top of the cost savings from skipping the LLM call entirely. A more recent implementation using Redis-backed embedding caches reported hit rates above 97% and cut costs by roughly 73% against a no-cache baseline, with a 13x drop in processing time.
The catch worth knowing: semantic caches can misfire if two genuinely different questions embed too close together. Newer research has flagged this as a real risk for anything security- or finance-adjacent, so if you're caching responses that touch money movements or access control, put a stricter similarity threshold on it and monitor for collisions.
The takeaway: if your product has any repeat-question pattern (support, FAQs, internal tools), semantic caching is close to free money.
Sources:
- Bang, GPTCache: An Open-Source Semantic Cache for LLM Applications Enabling Faster Answers and Cost Savings (NLP-OSS 2023) — aclanthology.org/2023.nlposs-1.24
- Regmi et al., GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching (2024) — arxiv.org/abs/2411.05276
- On collision/adversarial risk in semantic caches, see the SAFE-CACHE analysis of GPTCache — researchgate.net/publication/376404523
3. Trim the context. More retrieved text is not more accuracy.
This is the one most teams get backwards. The instinct is: if the model has a huge context window, feed it everything. Full documents. Every chunk that might be relevant. Just in case.
The research says the opposite. It's called the "lost in the middle" effect, first documented by Stanford and University of Washington researchers studying long-context models. LLMs are reliably good at using information at the very start or end of their context, and reliably worse at pulling out facts buried in the middle. The result is a U-shaped accuracy curve: performance drops as you stuff more context in, even though the model technically "fits" it all.
The practical fix is boring but effective: retrieve fewer, tighter chunks. Multiple analyses have found that 2-4 well-chosen chunks (roughly 2-4k tokens) consistently beats a stack of 20+ chunks (10k+ tokens), both on accuracy and on your bill. If you must send more, put your best chunk first and your second-best last, since that's where attention is strongest; simple reordering has been shown to lift multi-document QA accuracy by 10-20%.
There's also a compression layer worth knowing about. Microsoft's LLMLingua research showed prompts can be compressed 2-20x with under 2% quality loss by stripping low-information tokens before the prompt ever reaches the model. Their follow-up, LongLLMLingua, applies this specifically to long-document RAG contexts and improved accuracy by over 20% while cutting token count roughly 4x, because trimming the noise made the signal easier to find.
The takeaway: RAG isn't "give the model more." It's "give the model the right 3 things." Halving your retrieved tokens can raise accuracy, not just cut cost.
Sources:
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts (Stanford, 2023) — arxiv.org/abs/2307.03172
- Jiang et al., LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models (Microsoft, 2023) — arxiv.org/abs/2310.05736
- Jiang et al., LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (Microsoft, 2023) — arxiv.org/abs/2310.06839
4. Tighten the output. This is the most underpriced lever in the industry.
Here's a number most founders don't know: output tokens cost 2-6x more than input tokens across every major provider, and on some frontier models the gap runs as high as 10x. That's because generation happens token by token in a sequential, compute-heavy decode loop, while your input gets processed in one parallel pass.
That means a bloated system prompt is expensive. But an uncapped, rambling response is worse. If your product generates long explanations, restates the question back to the user, or pads answers with boilerplate ("I'd be happy to help with that! Here's what I found:"), you're paying the most expensive rate in the entire pricing sheet for words nobody needed.
Two fixes, both nearly free to implement:
- Strip the system prompt. Every persona instruction, every repeated formatting rule, every "you are a helpful assistant" paragraph gets billed on every single call. Trim it to what actually changes model behavior.
- Cap response length explicitly. Don't rely on the model to self-regulate length. Set a max token budget per use case and force concision in the instructions.
Reasoning models add a second trap here. Internal "thinking" tokens get billed at output rates even when you never see them, and a simple question can quietly burn thousands of hidden tokens before it produces 500 visible words. If you're using a reasoning-tier model for a task that doesn't need deep reasoning, you're paying the output premium twice over for nothing.
The takeaway: input optimization gets all the attention. Output optimization has the higher price tag per token and the least scrutiny. Fix that gap first.
Sources:
- Silicon Data, Understanding LLM Cost Per Token: A 2026 Practical Guide — siliconedata.com/blog/llm-cost-per-token
- CloudZero, LLM API Pricing Comparison In 2026: Every Major Model, Ranked By Cost — cloudzero.com/blog/llm-api-pricing-comparison
- OpsLyft, LLM Pricing Comparison: Cost Per Token Across Every Major Model (2026) — opslyft.com/blog/llm-pricing-comparison-cost-per-token
5. Go hybrid. Baseload on infrastructure you control, burst on the API.
This is the lever for teams past the prototype stage with predictable, high-volume traffic.
The economics here are genuinely context-dependent, and any blog telling you self-hosting is "always cheaper" is wrong. Research comparing on-premise deployment against commercial APIs found break-even points ranging from under 4 months for small models to multiple years for large frontier-scale ones, depending entirely on your volume and which model tier you're replacing. Below a certain volume, the fixed cost of GPUs and the engineering time to run them simply doesn't pay for itself against a pay-as-you-go API.
But at real scale, the number moves fast. Multiple analyses of hybrid deployments, where routine, high-volume, or sensitive workloads run on owned or dedicated infrastructure while burst traffic and hard reasoning tasks go to frontier APIs, report 40-70% blended cost reductions compared to a pure-API stack. One fintech case study cited in recent research cut monthly AI infra spend from $47,000 to $8,000 after moving to this model. The pattern that recurs across the research: local infrastructure wins on routine, predictable load; the API wins on the tail of hard, rare, or bursty requests where frontier capability actually matters.
Compliance can override the spreadsheet entirely here too. If you're in healthcare, finance, or anything with strict data residency requirements, on-prem or private-cloud deployment for sensitive workloads often isn't optional regardless of what the cost model says.
The takeaway: hybrid isn't a compromise. For teams with real volume, it's the architecture that lets you use expensive frontier models exactly where they're worth it, and nowhere else.
Sources:
- A Cost-Benefit Analysis of On-Premise Large Language Model Deployment: Breaking Even with Commercial LLM Services (2025) — arxiv.org/abs/2509.18101
- Alpacked, Self-Hosted LLM Guide: Costs, Architecture & Breakeven Point (2026) — alpacked.io/blog/self-hosted-llm-guide
- KKRF Group, Self-Hosted LLM vs API: Enterprise Cost & Security (2026) — kkrfgroup.com/self-hosted-llm-vs-api-enterprise
Why most companies never actually capture these savings
Here's what usually happens instead. One engineer reads about semantic caching, implements it for one feature. Another team is still sending full documents to GPT-4 because nobody told them about chunking. A third tool your company pays for has its own model choice, its own system prompt, its own retry logic, and nobody outside that team has ever looked at it.
Every lever above works. But every lever above also assumes someone can see the whole picture: which agent is calling which model, how often, on what data, at what cost, across every vendor and every tool your company runs. Most companies can't. They have an org chart for every human on payroll. They have no equivalent for the dozen AI agents and tools quietly running up the bill in parallel, each with its own default settings, none of them accountable to a single owner.
That's not an engineering problem. It's a governance problem. And it's the reason a well-intentioned "let's optimize our LLM costs" initiative usually fixes one team's spend and misses the other 80% happening somewhere else in the business.
The pattern underneath all five levers
None of this is about using a worse model. It's about not paying frontier prices for work that doesn't need a frontier model, anywhere in your stack, not just the one project someone happened to audit.
Route the easy stuff away from the expensive model. Don't pay twice for questions you've already answered. Give the model less noise, not more data. Stop paying the highest rate on your pricing sheet for words nobody reads. And once your volume is real, stop renting compute you could be routing more cheaply.
If you're the one accountable for the number, the real question isn't "which of these five levers should we pull first." It's "do we even have visibility into which of our tools and agents already need pulling." You can't route what you can't see. You can't cap output tokens on a vendor's agent if you don't know it exists.
Fix the visibility. Then fix the architecture. The bill follows.
