Start with the cost equation
Your AI bill is shaped by four variables:
AI cost = input tokens + output tokens + number of model calls + price per token
Every technique in this guide reduces at least one of them.
- Input optimization removes text the model does not need.
- Output controls stop answers from running longer than the task requires.
- Caching avoids repeated model calls.
- Routing sends each job to the right model or service tier.
- Advanced compression reduces the context or reasoning used for harder tasks.
Do not apply every technique at once. Start with the low-risk changes. Measure cost, quality, latency, and failure rate. Add more complex techniques only where traffic makes the extra infrastructure worthwhile.
1. Ask for concise output
A model often repeats the question, adds a preamble, explains its structure, and closes with an offer to help. You pay for every one of those tokens.
A strong system instruction removes that habit.
How to use it
Add a short output policy to the system prompt:
Answer directly. Skip the preamble. Do not repeat the question. Use the fewest words needed to give a complete answer.
For a support assistant, make the rule more specific:
Answer in no more than three short paragraphs. Lead with the action the customer should take. Only include background when it changes the action.
This works well for FAQs, support replies, summaries, classifications, and high-volume internal assistants.
Watch out for answers that become too terse. Test tasks where the user needs context, reasoning, safety guidance, or step-by-step instructions.
2. Set an output token budget
Concise instructions influence the model. An output cap enforces a hard ceiling.
Set a safe default whenever the caller does not specify one. Use different limits for different jobs. A classification might need 20 tokens. A support reply might need 300. A report might need 2,000.
How to use it
LIMITS = {
"classification": 50,
"support_reply": 350,
"summary": 600,
"report": 2000,
}
request.max_output_tokens = LIMITS.get(task_type, 500)
Add a finish check. If the response hits the cap, either ask the model to continue, retry with a larger limit, or return a clear partial-result state.
Never set one low global cap for every task. Truncated JSON, cut-off code, and incomplete instructions cost more to repair than the tokens they save.
3. Normalize unnecessary whitespace
Copied documents and generated prompts often contain repeated spaces, tabs, trailing spaces, and large blocks of empty lines. Those characters add noise and sometimes add tokens.
Normalize plain text before sending it to the model.
How to use it
import re
def normalize_plain_text(text):
text = re.sub(r"[ \t]+", " ", text)
text = re.sub(r"\n{3,}", "\n\n", text)
return "\n".join(line.rstrip() for line in text.splitlines()).strip()
Protect code blocks, Markdown tables, YAML, poetry, and any format where spacing carries meaning. A safe production version splits protected blocks from ordinary prose, then only cleans the prose.
This is a small saving. At high volume, small input reductions still add up.
4. Minify JSON before adding it to a prompt
Tool results, API payloads, configuration files, and analytics events often reach the model as formatted JSON. Indentation makes JSON easier for humans to read. The model rarely needs it.
Parse the JSON, then serialize it without extra spaces or line breaks.
How to use it
import json
def minify_json(raw_json):
data = json.loads(raw_json)
return json.dumps(data, separators=(",", ":"), ensure_ascii=False)
Input:
{
"customer": "Acme",
"plan": "Growth",
"seats": 42
}
Output:
{"customer":"Acme","plan":"Growth","seats":42}
Use a real parser. Do not remove spaces with a regex. A regex risks changing string values or breaking invalid JSON. Keep the formatted version in logs if engineers need it for debugging.
5. Collapse duplicate messages
Retries, webhooks, agent loops, and state bugs sometimes append the same message twice. Repeated system instructions also appear when several middleware layers build the final request.
Remove exact duplicates before the model call.
How to use it
def collapse_consecutive_duplicates(messages): cleaned = [] for message in messages: if cleaned and message == cleaned[-1]: continue cleaned.append(message) return cleaned
Start with consecutive, exact matches. They are easier to remove safely.
Do not delete every repeated sentence across the whole conversation. A user who repeats a request might be correcting the assistant or adding emphasis. Track retry IDs and event IDs when possible, since they provide a stronger signal than text alone.
6. Trim old conversation context
Long-running assistants often resend the full chat on every turn. Input cost grows with the conversation, even when the middle of the thread no longer matters.
Keep the stable instructions, the original goal, the newest turns, and any state that still affects the answer. Remove stale back-and-forth.
How to use it
def trim_history(messages, recent_turns=6): system = [m for m in messages if m["role"] == "system"][:1] users = [m for m in messages if m["role"] == "user"] first_user = users[:1] recent = messages[-recent_turns:] return deduplicate(system + first_user + recent)
A stronger version stores durable state separately:
{
"goal": "Prepare a board update",
"audience": "Investors",
"decisions": ["Use monthly cohorts", "Exclude test accounts"],
"open_questions": ["Confirm churn definition"]
}
Then send this compact state plus the latest turns.
The main risk is deleting a fact that still matters. Test memory-heavy conversations, user preferences, approvals, legal constraints, and agent tasks that span several steps.
7. Cache exact repeated requests
If the same deterministic request arrives twice, the second request should not pay for the same answer again.
An exact-match cache stores the response under a key built from every input that affects the result.
How to use it
cache_key = stable_hash({
"tenant_id": tenant_id,
"model": model,
"prompt_version": prompt_version,
"temperature": 0,
"messages": messages,
"tools": tools,
})
if cache.exists(cache_key):
return cache.get(cache_key)
response = run_model(messages)
cache.set(cache_key, response, ttl="24h")
return response
This fits FAQ answers, deterministic transformations, repeated document classification, and idempotent background jobs.
Tenant ID, permissions, tool access, model version, and prompt version belong in the key. Without them, one user might receive another user's data or an answer generated under an old policy.
8. Keep repeated prompt prefixes stable
Many model providers discount repeated prompt prefixes. The repeated section often includes the system prompt, tool definitions, schemas, policies, and reference material.
Prompt caching works best when that prefix stays identical across requests.
How to use it
Order the prompt from most stable to most variable:
1. System instructions 2. Tool definitions 3. Shared reference material 4. Session state 5. Current user message
Avoid inserting timestamps, request IDs, or user-specific values near the top. Put volatile data after the reusable prefix.
Bad structure:
Request time: 18:42:17 User: 91827 [10,000 tokens of shared instructions]
Better structure:
[10,000 tokens of shared instructions] User: 91827 Request time: 18:42:17
Track cache-hit rate by model and prompt version. A caching feature has little value if tiny prefix changes trigger misses on most requests.
9. Use a semantic cache for similar questions
An exact cache only matches identical input. A semantic cache also matches different wording with the same intent.
For example, "How do I cancel my plan?" and "Where do I end my subscription?" might share one answer.
How to use it
query_vector = embed(user_query) match = vector_store.search(query_vector, tenant_id=tenant_id, limit=1) if match.score > 0.94 and match.answer_is_fresh: return match.answer answer = run_model(user_query) vector_store.save(query_vector, answer, metadata) return answer
Similarity alone is not enough. Include product version, language, user permissions, geography, account state, and answer expiry in the eligibility rules.
Use semantic caching for stable support and knowledge-base questions. Avoid it for personal advice, live data, security-sensitive answers, or queries where one small wording change alters the required response.
10. Route simple requests to a cheaper model
Not every request needs your strongest model. Extraction, classification, formatting, and short factual transformations often work well on a cheaper model.
Route complex or high-risk tasks to a stronger model.
How to use it
Start with clear rules instead of an opaque classifier:
def choose_model(task):
if task.type in {"classification", "extraction", "formatting"}:
return "economy-model"
if task.has_code or task.requires_multi_step_reasoning:
return "frontier-model"
if task.risk in {"legal", "medical", "financial", "security"}:
return "frontier-model"
return "balanced-model"
Build a test set from real traffic. Compare quality, latency, and cost for every route. Add a fallback when the cheaper model returns invalid structure, low confidence, or a failed quality check.
Routing fails when "short" gets confused with "simple." A five-word security request might be harder than a 500-word formatting task.
11. Move non-urgent jobs to a batch API
Real-time inference is priced for immediate results. Many AI workloads do not need an answer in seconds.
Evaluations, data enrichment, tagging, embeddings, moderation backfills, and nightly summaries fit an asynchronous batch.
How to use it
jobs = [
{"id": "row_001", "task": "classify", "input": record_1},
{"id": "row_002", "task": "classify", "input": record_2},
]
batch_id = batch_service.submit(jobs)
# Later
results = batch_service.collect(batch_id)
reconcile(results, key="id")
Design for delayed results from the start. Store request IDs, prompt versions, and row IDs. Add retry rules. Make the job idempotent so a retry does not duplicate downstream work.
Do not route interactive product flows into a batch. The saving comes from accepting a slower completion window.
12. Use a lower-priority service tier
Some providers offer cheaper processing for traffic that accepts variable latency or temporary unavailability.
This fits development, evaluations, enrichment, offline analysis, and low-priority agent jobs.
How to use it
request = {
"model": selected_model,
"service_tier": "flex",
"input": prompt,
}
Treat the field name as provider-specific. The architecture matters more than the syntax.
Add a timeout and fallback policy:
try:
return run_low_priority(request, timeout=90) except CapacityUnavailable: queue_for_later(request)
Keep strict real-time flows on a service tier with predictable latency. Cheaper compute loses its value if users abandon the task before the answer arrives.
13. Prune irrelevant RAG context
RAG pipelines often retrieve several long documents, then send all of them to the generator. Much of that text does not answer the question.
A pruning step removes irrelevant passages after retrieval and before generation.
How to use it
documents = retrieve(query, top_k=20) passages = split_into_passages(documents) ranked = rerank(query, passages) selected = [p for p in ranked if p.relevance > threshold][:8] answer = generate(query=query, context=selected)
Measure answer recall, not only token reduction. Ask: did the selected context retain the evidence needed for the correct answer?
Use a "no answer in context" outcome instead of forcing the generator to answer from weak evidence. For cited answers, verify that each claim still maps to a retained passage.
14. Compress long prompts with a smaller model
For long documents, transcripts, and retrieved context, a smaller compression model identifies which tokens or passages matter. The main model receives the compressed version.
Tools such as LLMLingua-2 follow this pattern.
How to use it
compressed_context = compressor.compress( context=long_context, query=user_query, target_ratio=0.4, ) answer = run_main_model( query=user_query, context=compressed_context, )
Compression adds its own latency and cost. It makes sense when the original context is large enough to justify another model call.
Evaluate numbers, names, negations, dates, and edge cases. These details often disappear first when compression gets aggressive.
15. Optimize prompts against a real evaluation set
Prompt optimization systems such as GEPA test prompt variants, inspect failures, propose changes, and keep the versions that improve measured performance.
The saving is indirect. A better prompt reduces retries, raises first-pass success, or lets a cheaper model meet the same quality target.
How to use it
baseline = current_prompt for round in range(optimization_rounds): failures = evaluate(baseline, training_examples) candidates = propose_prompt_edits(baseline, failures) baseline = select_best(candidates, validation_examples) final_score = evaluate(baseline, held_out_test_set)
Use separate training, validation, and held-out test sets. Otherwise the optimizer might memorize the examples and fail on live traffic.
This technique fits mature, repeatable workflows with reliable graders. It is excessive for an early prototype whose task changes every week.
16. Use compact reasoning for the right tasks
Long reasoning traces consume output tokens. Sketch-of-thought asks the model to reason with short keywords, symbols, or compact conceptual links instead of full prose.
How to use it
Solve the problem using a compact reasoning sketch. Use short phrases and symbols for intermediate work. Return the final answer with the minimum explanation needed to verify it.
For a structured workflow:
{
"sketch": ["revenue - costs", "adjust for churn", "compare scenarios"],
"answer": "Scenario B has the higher twelve-month margin."
}
Use compact reasoning for math, planning, and multi-hop tasks where intermediate prose becomes expensive. Test accuracy across models and languages. Over-compression makes mistakes harder to inspect and sometimes lowers answer quality.
Which inference optimization techniques should you use first?
Start with the techniques that need little infrastructure and carry limited quality risk.
Phase 1: Apply the quick wins
- Ask for concise output.
- Set task-specific output limits.
- Normalize whitespace in plain text.
- Minify JSON.
- Collapse exact duplicate messages.
Phase 2: Reduce repeated context and calls
- Trim old conversation history.
- Add exact-match caching.
- Stabilize prompt prefixes for provider caching.
Phase 3: Optimize traffic by workload
- Route simple tasks to cheaper models.
- Move non-urgent work into batches.
- Use lower-priority service tiers for flexible jobs.
- Add semantic caching only for stable, repetitive questions.
Phase 4: Add advanced compression where volume supports it
- Prune RAG context.
- Compress long prompts.
- Optimize prompts against an evaluation set.
- Test compact reasoning on expensive reasoning tasks.
Measure savings without damaging quality
Token reduction is not the goal by itself. The goal is a lower cost per successful task.
Track these metrics before and after each change:
- Input tokens per request
- Output tokens per request
- Model calls per completed task
- Cache-hit rate
- Cost per completed task
- Valid response rate
- Human correction rate
- Task success score
- Latency at the 50th and 95th percentiles
Run one change at a time. Keep a control group. Review failures, not only averages.
A technique that cuts token use by 40% but doubles retries has failed. A routing rule that lowers cost while increasing human corrections has also failed.
The best AI cost optimization strategy is layered
The largest savings rarely come from one clever trick. They come from stacking several boring controls:
- Shorter prompts
- Shorter answers
- Fewer repeated calls
- Better caching
- Smarter routing
- Cheaper processing for work that does not need instant results
Start with output limits and prompt hygiene. Add caching when repetition appears. Add routing when traffic contains clear task classes. Use compression only after evaluations show what information the model needs.
The result is not merely a cheaper model call. It is an inference system that spends more only when the task earns it.
