Prompt Engineering in Production Is a Systems Problem

Q: How do I know a prompt change actually helped?

You do not, without a measurement. Read the [evaluation guide](/blog/evaluate-your-llm-pipeline/) for how to build that set, and use the [evaluation guide's generator](/blog/evaluate-your-llm-pipeline/) to produce the edge cases. Subjective improvement on the examples you happened to look at is the most common way a prompt change ships a regression.

Why this matters

Prompt engineering has a bad reputation that it partially deserves. The genre is full of folklore — “add this magic phrase”, “tell it to think step by step”, “use XML tags” — and a lot of it was reverse-engineered from one model version on one task, then presented as universal truth.

The reason it feels unsatisfying is that it treats the prompt as the system. It is not. A prompt is one input to a pipeline, and in production it is usually the input you have the least control over, the part that changes most often, and the part that is hardest to test.

What actually determines whether a production LLM feature works is everything around the prompt:

What context you assembled before the call, and what you left out
What you enforce outside the model, in code, rather than asking for in prose
How failures are detected, before the user detects them for you
How you measure the result, so you know whether a change helped

The prompt is where you express intent. The system is where you make the intent true.

What I tested on

Pipeline: Python 3.13, FastAPI, multi-tenant content generation
Models: DeepSeek Chat for bulk drafts, a long-context model for the final rewrite
Stages: topic classification, knowledge-graph retrieval, draft generation, compliance check, readability rewrite
Volume: roughly forty articles a day across several tenants
Duration: the prompt layer has been in production since March 2026, and rewritten four times

The four rewrites are the useful part. Each one is a change I made after an evaluation set showed a regression, not after reading a blog post about prompt technique.

The prompt is the smallest part of the system

Here is the shape of a generation stage in my pipeline, reduced to the parts that matter:

async def generate(request: Request) -> Article:
    # 1. What the model will see
    context = await assemble_context(
        tenant=request.tenant_id,
        topic=request.topic,
        exclusions=await duplicate_check(request),
    )

    # 2. The instruction
    prompt = PROMPT_VERSION.get(request.template_kind)  # versioned, not inline
    system = SYSTEM_PROMPTS[request.template_kind]

    # 3. What happens after
    draft = await llm.generate(
        system=system,
        user=render(prompt, context),
        max_tokens=1800,
        temperature=0.7,
    )

    # 4. What the model is not allowed to decide
    article = enforce(generated=draft, rules=tenant_rules(request.tenant_id))

    # 5. What we learn from the outcome
    await record(prompt_version=PROMPT_VERSION.id, used=context.source_ids,
                 tokens=llm.last_usage, article=article)

    return article

Five stages, and the prompt is stage two. The failure modes I actually spend time on are in stages one, three, four and five — context assembly, limits, enforcement and measurement.

Context assembly is where quality is won. A model given the right three paragraphs and asked a precise question will outperform a model given thirty paragraphs and the same question, every time. This is the single highest-leverage thing in the pipeline, and it has nothing to do with the wording.

Limits are not prompt material. max_tokens is an API parameter. A required word count is a check on the output. Neither belongs in the instruction text, and putting them there means the model might comply and might not, on a schedule you cannot predict.

Enforcement belongs in code. If a term is forbidden, filter for it. If a length is required, measure it and regenerate. If a topic is out of scope, check the routing decision before the generation call, not in the prompt. Every rule you can enforce outside the model is a rule that never fails.

Measurement is what makes the rest learnable. Without a record of which prompt version produced which output, every change is a guess and every regression is invisible until a reader complains.

What actually changes output quality

Four things, in the order I would work through them.

1. The context, not the wording

If the output is wrong, the first question is whether the model had the information to be right. This is the question most teams skip, because it is less satisfying than rewriting the prompt — but it is the one that pays.

Concretely: when a generated article contradicts a fact that is in your knowledge base, the cause is nearly always retrieval, not instruction. Rewriting the prompt does not fix it; it teaches the model to be confident about a missing input.

The test is simple. Take ten outputs you believe are wrong and check whether the source material needed to answer correctly was in the context window. In my pipeline, this check was right about 70 percent of the time. The remaining 30 percent were genuine prompt or model failures.

2. Constraints the model cannot violate

This is the shift that produced the largest quality improvement in my own pipeline. Move as many rules as possible out of the prompt and into code.

Rule	As an instruction	As a check
Word count	“Write about 800 words”	Measure; regenerate if outside range
Forbidden term	“Do not mention X”	Filter the output; regenerate if present
In-scope topic	“Only discuss topics in the matrix”	Check the routing decision before calling
Brand mention	“Mention the brand once”	Count occurrences; regenerate if wrong
Duplicate content	“Write something new”	Similarity check against published articles
Structure	“Use these headings”	Validate the heading set on the parsed output

None of these are hard to check. All of them fail intermittently as instructions. Moving them to code converted a class of quality problems into a class of engineering problems, which is a strictly better place to be.

The prompt still tells the model about the important constraints — it produces better output when it knows the rules. But the code is what guarantees them. Say it in the prompt, enforce it in the pipeline.

3. Examples that disambiguate, not that repeat

Few-shot examples help when they show something the instruction cannot describe clearly — an edge case, a format detail, a judgement about tone.

They hurt when they merely demonstrate what the instruction already says. Six examples of “write a clear article” cost six times the tokens on every call and teach the model nothing the instruction did not.

The useful test: after removing the examples, would the instruction alone produce a materially different output? If not, cut them. In my pipeline, dropping two of five examples on the rewrite stage cut the token cost by about 18 percent with no measurable quality change — the evaluation set said so, which is the only reason I trusted it.

4. The output format the program can parse

If a downstream stage consumes the model’s output, make the output format explicit and machine-checkable. Ask for JSON, specify the exact keys, and parse defensively — a model that returns valid JSON 98 percent of the time still returns prose 2 percent of the time, and that 2 percent is your incident.

raw = await llm.generate(system=SYSTEM, user=prompt, max_tokens=900)
try:
    payload = ArticleSchema.model_validate_json(raw)
except ValidationError:
    # One retry with the validation error appended to the instruction.
    payload = await retry_with_error(raw, exc)

That retry is worth more than any prompt technique in this list, because it turns a hard failure into a soft one that resolves itself.

Version the prompt, or you cannot debug it

The most expensive bug I have had in this pipeline was not a model failure. It was an output that looked wrong three weeks after it was generated, with no way to know which prompt produced it, because the prompt had been edited in place twice since.

The fix is unglamorous:

Prompts live in versioned files or database records, never inline in application code
Every request logs the prompt version, the model name, the parameters and the token counts
Every output logs which sources it used, so “the model ignored the source” is distinguishable from “the source was wrong”
Prompt changes go through the same evaluation set as code changes

The last one is the discipline. A prompt edit is a code change with the same blast radius, and it deserves the same gate. When I routed prompt changes through the evaluation set, the number of production surprises dropped to roughly one per quarter — and each of those turned out to be a corpus problem rather than a prompt problem, which the trace data showed immediately.

What broke along the way

I moved a rule from the prompt into code and the outputs got worse. The forbidden-term rule was in the prompt, and moving it to a post-filter meant the model no longer knew to avoid the term in the first place — so it used it, and the filter caught it, and we regenerated. The regeneration rate tripled. The fix was to do both: keep the rule in the prompt and enforce it in code. The prompt makes it rare; the code makes it impossible.

Temperature was the wrong dial. I had been lowering temperature to reduce output variability, and it flattened the writing. The variability was not the problem; the missing context was. Once retrieval was fixed, I raised temperature back and the quality improved — the same distribution of sentences, chosen better.

A long system prompt made things worse. I consolidated instructions from four stages into one large system prompt for the whole pipeline. It cost more tokens per call and produced worse outputs, because the stage-specific rules became diluted across a large block. Splitting them back per stage improved both cost and quality. Context length is not a free resource, even inside the context window.

Few-shot examples masked a retrieval bug. I had four examples in the draft prompt that happened to cover the topic the failing articles were about, which made the outputs look fine in the sample I was checking. The evaluation set caught it — the examples had taught the model to reuse example content rather than the retrieved material. This is the clearest argument for having an evaluation set that covers cases you did not design it around.

Pros and cons

What this approach gets you

Quality improvements that survive a model update, because they are about context and constraints rather than wording
A failure rate that goes down when you fix it, rather than a rate that resets when the provider ships a new model
The ability to change prompts safely, because the evaluation set gates every change
Debuggable incidents, because every output traces back to a prompt version, a model and a set of sources

What it costs

Engineering time that looks like overhead when you are trying to ship the feature
Tokens for the context you assemble well, which is more than the tokens you would have sent with a short prompt
The discipline to write down what a good output looks like, which is harder than writing the prompt
A codebase that is bigger because the rules live in two places — the prompt and the enforcement layer

Who this is for

This is for anyone shipping an LLM feature that other people depend on. The moment a wrong output has a cost, prompt wording alone stops being sufficient, and the work moves from writing instructions to building the system around them.

Start with the prompt alone if you are prototyping, if the output is not consumed by another program, or if a wrong answer costs nothing more than a retry. That is a legitimate place to be, and most experiments should stay there.

If you are past that point, do the work in this order: fix context assembly, move rules into code, add the evaluation set, then start tuning wording. Doing it in a different order wastes time — I spent a month rewriting prompts to fix what was a retrieval bug, and that month would have bought a reranker and a better chunk size.

FAQ

Q: Is prompt engineering still worth learning if the models keep improving? A: The tricks are worth less than they were — a well-written instruction handles a model that used to need an example. What matters more now is everything around the prompt: what context you assemble, what you enforce outside the model, and how you measure the result. That work does not get absorbed by a model update.

Q: How do you version prompts in production? A: Store the prompt as a file or a database record with a version number, never inline in application code. Log the version alongside every request and every response. When an output looks wrong three weeks later, the version is the first thing you need — without it you are reconstructing history from memory.

Q: Should I use few-shot examples? A: Yes when the task has a consistent shape and the examples disambiguate the format. No when they merely restate the instruction — they cost tokens on every call and add nothing. Two or three examples that demonstrate the edge cases usually beat six that demonstrate the happy path.

Q: What is the most common production prompt mistake? A: Letting the model decide something a program should decide. Whether a length limit is respected, whether a forbidden term appears, whether a topic is in scope — these are checks, not instructions. A model that is told a rule will sometimes violate it; a regex never will.

Q: How do I know a prompt change actually helped? A: You do not, without a measurement. Read the evaluation guide for how to build that set, and use the evaluation guide’s generator to produce the edge cases. Subjective improvement on the examples you happened to look at is the most common way a prompt change ships a regression.