Production RAG Pipelines: Retrieval Is a Data Problem

Retrieval-augmented generation is often introduced as “put documents in a vector database and ask a model a question.” That demo works, but production systems fail for less glamorous reasons: stale indexes, missing access filters and chunks that lose the context needed to answer.

The reliable mental model is a pipeline:

source documents -> parse -> chunk -> embed -> index
user question -> authorize -> retrieve -> rerank -> generate -> cite

#Retrieval Comes Before Generation

The model cannot correct a retriever that returns irrelevant context. Start by measuring retrieval independently:

• Recall@k: did the relevant passage appear in the top k results?
• Precision@k: how much of the returned context is useful?
• Groundedness: does the answer stay within the retrieved evidence?
• Citation accuracy: do citations actually support the claims they reference?

A small, reviewed evaluation set is more useful than a large collection of subjective examples. Keep questions, expected source documents and acceptable answer points in version control.

#Chunking Is a Product Decision

Fixed-size chunks are a good baseline, not a universal solution. Split at headings and paragraphs first, then apply a size limit. Preserve document title, section path, tenant and access-control metadata alongside every chunk.

{
  "text": "The refund window is 30 days...",
  "source": "billing-handbook.md",
  "section": "Refunds",
  "tenantId": "acme",
  "allowedRoles": ["support", "finance"]
}

At query time, authorization must be part of retrieval, not a filter applied after generation. Returning a forbidden chunk to the model is already a data leak, even if the final answer happens not to mention it.

#Hybrid Search Beats a Single Signal

Semantic search handles paraphrases well. Keyword search handles exact identifiers, error codes and product names well. Combining both, then reranking a small candidate set, is usually more robust than increasing the vector database top-k.

The generation prompt should also require the model to say when the evidence is insufficient. A confident “I do not have enough information” is a successful outcome when the index does not contain the answer.

#Operate the Index Like a Database

Give each ingestion run an index version. Build a new version, validate it, and switch an alias only after the evaluation set passes. This makes re-indexing reversible and prevents half-updated document collections.

RAG quality is not primarily a prompt trick. It is the result of good source data, explicit authorization, measurable retrieval quality and a generation step that is not allowed to invent missing evidence.