Building a Production RAG Pipeline That Doesn't Hallucinate

Retrieval-Augmented Generation (RAG) looks trivial in a notebook and painful in production. The gap is almost never the model — it’s everything around the model: how you chunk, retrieve, rank, and ground the answer.

Here’s the pipeline I actually ship, and the decisions that matter.

The core problem

An LLM hallucinates when it answers from parametric memory instead of your data. RAG fixes this by putting the right context in front of the model at inference time. But “the right context” is doing a lot of work in that sentence.

If retrieval returns the wrong chunks, a better model just writes a more convincing wrong answer.

1. Chunking is a retrieval decision, not a preprocessing step

Most teams chunk by a fixed token count and move on. That destroys meaning across boundaries. Instead:

• Split on semantic structure (headings, paragraphs) first.
• Keep chunks in the 300–600 token range for dense docs.
• Store a small overlap (10–15%) so answers spanning a boundary survive.

2. Retrieve more, then re-rank

Vector search alone is noisy. The pattern that works:

# 1. Over-retrieve with vector search
candidates = vector_store.search(query, k=20)

# 2. Re-rank with a cross-encoder for precision
ranked = reranker.rank(query, candidates)
context = ranked[:5]  # keep only the best

Over-retrieving then re-ranking consistently beats a single k=5 vector search, because the embedding model optimizes for recall, not final relevance.

3. Ground the answer explicitly

Tell the model, in the system prompt, to answer only from the provided context and to say when it can’t:

1. Pass the retrieved chunks with clear source markers.
2. Ask for inline citations back to those sources.
3. Add a refusal path: “If the context doesn’t contain the answer, say so.”

This one change removes most confident-but-wrong answers.

What actually moved the needle

In order of impact: re-ranking > chunking strategy > prompt grounding > model choice. Teams obsess over the last one and ignore the first three.

Ship the boring parts well and RAG stops hallucinating.