How to Cut Your LLM API Cost Without Breaking Quality

Why this matters

LLM cost problems rarely arrive as a bill shock. They arrive gradually: a prompt edit adds a paragraph of context, a pipeline grows a second drafting pass, a retry policy doubles the calls on a flaky upstream, and six months later you are paying three times as much for output that has not improved. Every change was individually reasonable, so nobody can point at the commit that broke the budget.

The second failure mode is worse. Teams cut cost by swapping to a cheaper model, the bill drops by half, and quality quietly drops too — but nothing catches it, because there is no evaluation set and no before-and-after comparison. The saving is real, the regression is invisible, and the bill gets re-approved a quarter later. The rule I work by is therefore simple: measure first, change one thing at a time, and never accept a saving you cannot verify against an evaluation set. If you do not have one, building it before you optimize is worth doing first.

What I tested on

• Stack: .NET 8 API tier for tenant and billing logic, Python FastAPI tier for retrieval and generation, Playwright for the publishing layer
• Models: one large hosted model for long-form drafting, a smaller hosted model for extraction and classification, plus a locally hosted quantized open-weight model for tagging
• Scale: a corpus of roughly 1,200 articles, multi-tenant, around 40 generated articles per day at peak
• Duration: in production since early 2026, tracked over several months of real traffic
• Cost: every dollar figure in this post is an example using assumed rates — use your provider’s current list prices, which move

Step 1: put a cost ledger in front of the pipeline

Before changing anything, you need to know which stage is expensive. This is not hard to build and it is the only step people skip.

The ledger is a per-call record: tenant, stage, model, tokens in, tokens out, cached tokens if the provider reports them, latency, and whether the call succeeded or was retried. In my setup the FastAPI tier emits it as a structured log line and the .NET API tier writes it to Postgres, because billing needs the same numbers engineering does.

@dataclass
class CallLedger:
    tenant_id: str
    stage: str            # "plan" | "draft" | "extract" | "compliance" | "embed"
    model: str
    tokens_in: int
    tokens_out: int
    cached_tokens: int
    latency_ms: int
    attempt: int

    def emit(self) -> None:
        log.info("llm_call", extra=asdict(self))

Once a week I roll it up by stage, ordered by output tokens rather than by total, because output is where the money usually is:

SELECT stage, model, SUM(tokens_in) AS in_tok, SUM(tokens_out) AS out_tok, COUNT(*) AS calls
FROM llm_calls
WHERE created_at >= now() - interval '7 days'
GROUP BY stage, model
ORDER BY SUM(tokens_out) DESC;

This produced two findings I would not have guessed. A “compliance check” stage — a single cheap classification call — was responsible for about a quarter of total output tokens, because it asked the model to rewrite the entire draft instead of flagging lines. And a few tenants generated a disproportionate share of retries.

Step 2: route each stage to the cheapest model that can do it

A single large model handling everything is the default because it is the simplest thing that works, not because it is required. The routing logic I use is deliberately boring: a table, a fallback, and a reason per row.

Two rules keep routing from becoming a silent quality regression. The fallback must be more capable, not cheaper — it runs when the primary overloaded or timed out, exactly when you do not want the cheap model to also be the only option. And route on stage, not on a confidence score computed after the fact: a cheap model can emit a plausible-looking confidence number, and tuning against it is guesswork. Stage-level routing is auditable — you point at a stage, look at its evaluation set, and see whether the small model passes.

{
  "routes": [
    {
      "stage": "extract",
      "model": "small-hosted",
      "fallback": "large-hosted",
      "max_output_tokens": 800,
      "why": "Structured field extraction. The large model adds nothing here."
    },
    {
      "stage": "draft",
      "model": "large-hosted",
      "fallback": "small-hosted",
      "max_output_tokens": 2400,
      "why": "The long-form writing itself. This is the stage that does not downgrade."
    }
  ]
}

Step 3: cache the prefix that never changes

STATIC_GUIDE = load_brand_guide()      # never varies per request
DYNAMIC_BODY = format_request(payload) # varies every time

prompt = f"{STATIC_GUIDE}\n\n---\n\n{DYNAMIC_BODY}"   # static part strictly first

If your prompt has a long stable instruction block — a brand voice guide, a compliance rubric — you are re-sending the same tokens on every call, and providers that support prompt caching let you pay a reduced rate for that repeated prefix. The cache key is the exact prefix, so any change in the cached region invalidates it, the cached region has to come first, and you must read the usage fields on the response to know whether the cache hit.

The ordering constraint is what people get wrong. One template variable in the system prompt — a timestamp, a tenant name, a request ID — destroys the cache for every subsequent call in the window, silently. This is a narrow win: it matters when the static prefix is long relative to the dynamic part, and does nothing when the dynamic part dominates.

Step 4: shorten the output before you shorten the input

Output tokens are commonly priced at several times the input rate, so the cheapest saving is usually in what the model writes, not what it reads. And output length is something you control directly, unlike quality.

Ask for less, structurally. If a downstream consumer only needs a title, a summary and three tags, ask for exactly that in a defined format rather than “give me a summary” and hoping. Structured output removes the preamble, the sign-off, and the “here’s what I found” that models like to add.

SCHEMA = {
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "summary": {"type": "string"},
        "tags": {"type": "array", "items": {"type": "string"}},
    },
    "required": ["title", "summary", "tags"],
}

resp = client.create(model=route.model, messages=msgs, schema=SCHEMA, **route.params)
if resp.usage.output_tokens > route.max_output_tokens:
    raise OutputBudgetExceeded(stage=stage, used=resp.usage.output_tokens)

Step 5: batch the work that does not need an answer now

Anything not in front of a user waiting can be batched — if your provider offers a batch endpoint, use it, and if not, at least run the work concurrently with a bounded worker pool so you are not paying for idle time in the request path.

Batch pricing, where it exists, is a meaningful discount, and the only requirement is that the work can complete asynchronously. That fits a lot of real stages: embedding a newly imported corpus, generating background variants, running a nightly classification pass.

async def embed_corpus(docs: list[Document], workers: int = 8) -> list[Vector]:
    sem = asyncio.Semaphore(workers)
    async def one(doc: Document) -> Vector:
        async with sem:
            return await embed_with_retry(doc.text)
    return await asyncio.gather(*(one(d) for d in docs))

Two operational notes. Batch requests fail as a unit on some providers, so keep items small enough that one bad document does not lose you the job, and log which items were included so a partial failure is recoverable. And do not batch interactive work — the concurrency savings are lost if a user is waiting.

Step 6: move narrow tasks off the API entirely

There is a category of work where an API call is pure overhead: deterministic, high-volume, and simple. Tagging articles, detecting language, extracting a date, checking whether a string matches a pattern, deduplicating near-identical titles.

For those, a small open-weight model running locally with quantized weights costs you nothing per call. Quantization — running the model at reduced numerical precision so it needs less memory and computes faster — is what makes a modest machine able to serve these at a sensible latency. I use one for a tagging stage, the only stage in the pipeline with a per-call cost of exactly zero.

The trade-offs are real. You now own the hardware, the warm-up time, and the operational burden, and a quantized small model is not accurate on everything. Keep the scope narrow: use it where a mistake is cheap. The moment the task needs judgement, it belongs back on a hosted model.

If you are not already measuring, do that first — the ledger step above is where the savings are found, and the rest of this list is only worth applying once you can see the numbers.

What broke along the way

A retry policy was making the bill worse than the traffic justified. A timeout on the drafting stage caused a retry on the large model, and because the retry re-sent the full context, a 3 percent timeout rate turned into a much larger share of total spend. The fix was not to remove the retry but to make it cheap: retry transport failures on a small model first, escalate only when the retry is about quality.

Prompt caching hit rate collapsed to nearly zero. Cached input tokens were being counted on only a handful of calls. The cause was a request ID interpolated into the second line of the static instruction block, changing the prefix on every call. Moving the request-specific content to the end restored the hit rate, and reading the provider’s cache usage fields made the regression visible in a day. Routing extracted fields with the wrong type. The small model returned the string "42" where the schema wanted the number, which passed a naive substring check but broke the downstream analytics job. The routing was not wrong — the model can do extraction — the validation was.

Pros and cons

What it gets you

• A materially lower bill, mostly from changing which model handles which stage rather than from any single clever trick
• Predictability, because a cost ledger tells you what a new feature will cost before you ship it
• Quality that is defensible, since every saving is checked against an evaluation set instead of assumed
• Optionality: once stages are routed, you can change one model without re-architecting the pipeline

What it costs

• Time up front for the ledger and the per-stage evaluation sets — the biggest cost, and not optional
• Engineering complexity: a routing table, a cache-ordering convention, and a validation layer all need maintaining
• Some latency and debugging difficulty, since a request can now touch different models in different stages
• A hardware and operations commitment if you adopt local quantized models
• Risk of the quiet version of this exercise: cutting a stage that needed the bigger model and not noticing

Who this is for

Follow this in order if you are running a production pipeline, if a single month of provider spend made someone ask questions, or if you plan to add stages. Start with Step 1 and Step 2; they are where nearly all of the saving lives, and the ledger pays for itself the first time it reveals a stage you did not know was expensive.

Start simpler if you are building something small, or if your workload is a handful of hand-run prompts — a single model with a good prompt is fine at that scale. If you already have an evaluation set and are about to change something expensive, read the evaluation guide first, because the saving is only real if you can prove the quality held.

FAQ

Q: How much can I realistically save on LLM API costs?

A: On my pipeline the reductions stacked to roughly 55 percent without a visible change in output quality, and most of that came from routing stages to cheaper models and cutting the number of full-size calls. Savings vary enormously by workload — a pipeline that only does hard long-form generation has far less headroom than one full of classification and extraction stages — so treat any percentage you read elsewhere as a ceiling rather than a promise.

Q: Should I switch my whole pipeline to one cheap model?

A: No. Switching everything at once is the fastest way to make a cost saving you cannot measure, because a quality regression hides inside the saving. Route stage by stage, keep an evaluation set per stage, and only move a stage once the set shows the output still passes. The stages that usually cannot move are the ones doing the actual long-form writing.

Q: Does prompt caching actually save money?

A: It does when the prompt has a large static prefix and the provider supports cached input pricing, which several major providers now do. The saving applies to the cached portion only, it depends on an exact prefix match, and it disappears entirely if you change a single character near the top of the prompt. Worth doing for long shared instructions, but a narrow win.

Q: What is the safest first cost optimization for a production pipeline?

A: Instrument before you optimize. Log tokens in and out, model name, stage, latency, and outcome per call, then look at where the money actually goes. In most pipelines I have measured, one or two stages account for the majority of spend, and they are rarely the ones the team assumed. Optimizing a stage that costs two percent of the bill is effort spent on nothing.

Q: How do I stop output length from growing every month?

A: Cap it in code rather than in the prompt, because a prompt instruction is a suggestion and a parameter is a limit. Set a max output token budget per stage, assert on the returned length, and fail the stage if it exceeds the budget instead of silently truncating mid-sentence. Where the format allows, ask for structured output and cut the prose a downstream consumer never reads.