Build Your First RAG Pipeline That Doesn't Hallucinate

Why this matters

Retrieval-augmented generation has a reputation problem, and it deserves one. Anyone can wire a document store to a prompt and get something that answers questions — and something that also answers confidently when the answer is not in your documents at all. The failure is invisible from the outside, because the output still reads like an answer.

The pattern is consistent enough to name. When a RAG pipeline produces a wrong answer, it is almost never because the model is careless. It is because of one of three things:

The retrieval returned the wrong chunk, so the model had nothing relevant and fell back on what it already knew.
The retrieval returned the right chunk but the prompt carried too much context, so the relevant part got diluted.
The retrieval returned exactly the right chunk and the model still answered from memory rather than from it.

The first two are engineering problems with known fixes. The third is a prompting problem with one reliable fix: make the source the only acceptable answer, and make saying “not in the documents” a valid, rewarded outcome.

This guide builds a pipeline in the order that prevents each failure, and then adds the part most tutorials skip — an evaluation harness that tells you when it has broken again.

What I tested on

Pipeline: Python 3.13, FastAPI, a 1,200-article knowledge base
Models: DeepSeek Chat for generation, a hosted embedding model for indexing, the same model family for reranking
Corpus: 1,200 articles, average 1,400 tokens each
Duration: in production since March 2026, across two tenant configurations
Cost: roughly $0.86 per generated article across all stages, including the compliance check and retries

The numbers below are from the production instance, not a benchmark harness. Where a figure is an estimate rather than a measurement, I say so.

Step 1: chunk the corpus for how the questions are asked

Chunking is where most pipelines quietly go wrong, because the default is copied from an unrelated tutorial. The 512-token chunk with 50 tokens of overlap that appears in most RAG examples is tuned for prose articles. Applied to reference documentation it cuts a heading away from the section it introduces; applied to source code it produces fragments that no longer parse.

Size the chunk by what a question needs, not by what a tutorial said:

Document type	Chunk size	Overlap	Why
Prose articles	512 tokens	64	A question usually needs one passage, not a section
Reference docs	768 tokens	128	Headings must stay attached to their section
Source code	1024 tokens	192	Functions are usually longer than a small chunk
Tables and rows	384 tokens	32	The fact is the row, not the paragraph around it

You can work this out with the chunk size planner on this site, which also shows how many chunks your corpus produces at each setting. My 1,200-article corpus at 512 tokens with 64 overlap produces 3,050 chunks — a number worth knowing before you pay for the embeddings.

The rule that matters: split on semantic boundaries, not on a fixed character count. A chunk that starts halfway through a section has already lost its context, and no amount of overlap recovers it cheaply.

Step 2: retrieve wide, then rerank narrow

The default retrieval pattern — top-k straight to the model — fails because it treats all chunks as equally plausible. It is not: a chunk that contains the answer usually competes with forty chunks that contain the same topic, and a model asked to pick the three most relevant has to make that judgment itself, at temperature, without seeing the query.

Two-stage retrieval fixes this. Pull wide with a cheap embedding similarity search, then let a reranker score those candidates against the actual query:

# Stage 1 — cheap, wide. Pure vector similarity.
candidates = index.similarity_search(query, k=50)

# Stage 2 — expensive, narrow. Cross-encoder scoring.
scored = reranker.rank(query=query, documents=[c.text for c in candidates])
top = [candidates[i] for i in scored.indices[:5]]

The reranker sees the query and the passage together, which is what lets it tell “this passage discusses the topic” from “this passage discusses the topic and happens to mention it”. That distinction is the whole reason reranking exists.

In my pipeline this one change cut wrong-chunk retrievals by roughly 60 percent, and it is one extra model call on a corpus of 50 candidates instead of 50 calls on the full index. If you cannot afford a reranker, retrieve more candidates and pass five to the model instead of three — more context, lower precision, but better than three wrong chunks.

Step 3: assemble the context so the source wins

This is the stage where the third failure mode lives, and it is mostly about what you do not include.

Put the source first and label it clearly. Content that arrives in the context window without marking is indistinguishable from your instructions, and models weight recent tokens more heavily — so an unmarked passage at the end of the prompt can quietly outrank a marked one at the start.

<source id="3" title="Chunking strategies">
...the retrieved passage...
</source>

Answer using only the sources above. If they do not contain the answer,
say so explicitly. Do not use outside knowledge.

Say what to do when the answer is absent. This is the instruction most pipelines leave out, and its absence is why models fill gaps with plausible generalities. Make refusal explicit and legitimate:

If the sources do not contain the answer, reply exactly: “Not covered in the source material.”

Then check that you actually honour it. In my pipeline, a refusal is a first-class outcome that gets counted in the evaluation report, not an error to be smoothed over.

Keep the context budget honest. Pass five chunks of 768 tokens and you are asking the model to reason over 3,840 tokens of retrieved text, most of which is irrelevant. Dilution is real: relevance drops as the context grows, and the model starts finding an answer in the noise. Fewer, better chunks beat more chunks every time.

Step 4: make it observable before you tune it

You cannot improve what you cannot see. Every retrieval needs to be recorded with enough context to answer one question: was the right chunk retrieved, and did the answer use it?

trace = {
    "query": query,
    "candidates": [{"id": c.id, "score": c.score, "rerank": c.rerank_score}
                   for c in scored[:10]],
    "selected": [c.id for c in top],
    "used_source_ids": used_source_ids,   # parsed from the model output
    "refused": refused,                   # did it take the "not covered" path
    "tokens_in": tokens_in,
    "tokens_out": tokens_out,
}

The used_source_ids field is the one that pays off. If the model cites sources in its output — which you should make it do — you can check whether the cited sources actually contain the answer. That turns “the answer was wrong” into “the right chunk was retrieved but the model ignored it”, which is a completely different bug with a completely different fix.

This is also where a tracing tool like Langfuse or a similar system earns its cost. Writing your own trace store is a week of work that you will maintain forever; the alternative is a table that stops being written the first time you are busy.

Step 5: build the evaluation set first, not last

The last step is the one that turns a pipeline into something you can change safely.

An evaluation set is a list of real cases with expected properties, committed to the repository and run against every change. The property-based shape matters more than the count:

{
  "id": "case-chunk-boundary",
  "input": "What chunk size do I use for reference documentation?",
  "category": "core",
  "expect": {
    "must_include": ["768"],
    "must_not_include": ["512 tokens is the right default"],
    "min_words": 20,
    "max_words": 200
  }
}

Note what is not there: no exact reference answer. Asserting on a single expected string is why most evaluation suites get abandoned in the second week — the model has many correct phrasings, and a test that fails on wording trains you to ignore the test.

Include the negative cases. The ones that matter are off-topic requests, malformed input, injection attempts, and near-duplicates of existing content. Those are exactly the failures that reach production silently, and they are cheap to add now and expensive to debug later.

The builder on this site generates those edge cases from a list of your real requests and emits a JSON file your CI can run. Run it on every pull request. A change that drops the pass rate below your threshold fails the build instead of shipping.

What broke along the way

The first reranker made retrieval worse. I started with a reranker that scored passages by similarity to a rewritten query, and it consistently promoted chunks that shared vocabulary with the question but did not answer it. Rewriting the query is a lossy transformation and the reranker inherited the loss. Using the original query fixed it.

Chunk overlap was set too high at first. I started at 50 percent overlap “to be safe”, which doubled the index size and the embedding bill for a retrieval improvement I could not measure. Ten to twenty percent was enough. The chunk size planner flags this directly when you enter a ratio above 30 percent.

A refusal rate of zero was the warning sign. My pipeline initially refused almost nothing, and I read that as good coverage. It was the opposite: the model was answering from memory on the questions where retrieval had failed, which is exactly the case where you need a refusal. Once I made refusal explicit and counted it, the answer quality on the remaining questions improved measurably — because the model was no longer being asked to bridge a gap it could not actually bridge.

The eval set caught a change no test would have. Switching embedding models improved my own subjective impression and dropped three retrieval cases below threshold. Without the set, that change would have shipped and I would have found out from support tickets a month later.

Pros and cons

The honest summary of what this pipeline buys you and what it costs:

What it gets you

Answers grounded in your own material rather than the model’s training data
A regression suite that makes prompt and model changes safe to ship
Visibility into which stage failed, instead of guessing which one
Cost control, because retrieval-before-generation keeps the prompt small

What it costs

Roughly double the implementation work of the naive version
Two model calls per query instead of one (rerank, then generate)
A maintenance burden: embedding models, chunk sizes and prompts all drift, and the evaluation set is what tells you they did
A corpus that needs cleaning before indexing — chunking amplifies whatever mess is already there

Who this is for

Build it this way if you are shipping an AI feature to someone other than yourself, if a wrong answer has a cost beyond a lost minute, or if you intend to change the prompt more than once a month. The evaluation harness alone justifies the structure, because it is the only part that keeps paying after the pipeline is finished.

Start simpler if you are building an internal tool for yourself with a corpus of twenty documents and no users. Retrieval, one prompt, no reranker, and a handful of hand-checked cases will be fine, and the complexity here will cost you more than it saves.

FAQ

Q: Do I need a vector database to build this? A: No. You can start with a local embedding index held in memory, get the pipeline shape right, and move to a hosted store later. The chunking and reranking decisions are the ones that matter, and both are independent of where the vectors live.

Q: How do I know if my pipeline is actually working? A: You cannot tell by reading the answers. You need an evaluation set of real cases with expected properties, run against every change. Without one, a prompt edit that halves your retrieval quality still produces fluent output, and you find out when a reader complains.

Q: What is the single most common cause of bad RAG answers? A: Retrieval returning the wrong chunk, not the model misreading a good one. In my own pipeline this was 70 percent of the failures. The fix is almost always in the chunking or the query expansion, not in the prompt.

Q: Should I rerank the retrieved chunks? A: Yes if you have more than about ten chunks in the corpus and the documents are similar to each other. A cheap first-stage retrieval pulls fifty candidates, a reranker scores them against the actual query, and the top three or five go to the model. It costs one extra model call and it is the highest-leverage addition to most pipelines.

Q: How large should the context window be before it hurts? A: It depends on the model and the task, but the useful rule is that relevance falls off well before the limit does. Five well-chosen chunks of 768 tokens beat ten chunks of 768, even though the second fits. If your context window is filling up, the answer is to retrieve fewer and better, not to raise the ceiling.