RAG That Works Starts Before the Prompt

The demo answered every question. Then somebody uploaded the real document set.

Suddenly the assistant cited superseded policies, missed answers buried in tables, and confidently combined text from two customers. The team reached for the prompt. They added instructions, examples, and increasingly stern warnings about using only the supplied context.

The prompt was innocent.

Most broken RAG systems are retrieval systems with an LLM standing at the scene of the crime. If the right evidence never reaches the model, prompt tuning cannot recover it. If the evidence is stale, incomplete, or belongs to somebody else, a fluent answer makes the failure worse.

RAG that works is an information retrieval pipeline first and an AI application second. Build it in that order.

Start by proving retrieval can find the answer

A common implementation embeds the question, retrieves the nearest five chunks, and sends them to the model. That is a prototype, not a retrieval strategy.

Dense vectors are good at semantic similarity. They are less dependable with exact identifiers, error codes, product names, dates, and internal acronyms. Keyword search has the opposite profile. Production systems usually need both.

A practical search path looks like this: apply access and tenant filters, run dense and keyword retrieval, combine the candidate sets, rerank them, then assemble context. Reciprocal Rank Fusion is a sensible way to combine rankings without pretending scores from different search methods are directly comparable. A cross-encoder reranker can then spend more compute on a small candidate set.

OpenSearch, Elasticsearch, Vespa, Weaviate, Qdrant, and PostgreSQL with pgvector can all support parts of this design. The product choice matters less than making each stage observable. For a pgvector query, the access condition belongs inside retrieval, not in application code after the search:

SELECT
 chunk_id,
 document_id,
 content,
 1 - (embedding <=> :query_embedding) AS similarity
FROM document_chunks
WHERE tenant_id = :tenant_id
 AND classification = ANY(:allowed_classifications)
 AND valid_from <= now()
 AND (valid_until IS NULL OR valid_until > now())
ORDER BY embedding <=> :query_embedding
LIMIT 30;

Fetching everything and filtering later can expose restricted text through logs, caches, rerankers, or model requests. For private AI, authorization is part of retrieval correctness. It is not middleware decoration.

Log the query, filters, candidate IDs, source versions, ranks, reranker scores, and final context IDs. Do not log sensitive document text by default. When an answer fails, you need to see where the relevant chunk disappeared.

Chunking is document engineering

The usual chunking advice, pick a token count and add overlap, treats every document like a bag of paragraphs. Real repositories contain headings, tables, code, footnotes, repeated navigation, scanned PDFs, and pages where extraction order bears little resemblance to reading order.

Chunk along semantic boundaries first. Preserve the heading path. Keep table headers with their rows. Keep a procedure together when splitting it would separate a warning from the step it governs. Store page numbers and stable source offsets so citations can be checked.

Only then apply a token limit. A configuration might start here:

chunking:
 target_tokens: 450
 max_tokens: 700
 overlap_tokens: 60
 boundaries:
 - heading
 - paragraph
 - list
 - table
 metadata:
 - document_id
 - version
 - heading_path
 - page
 - tenant_id
 - classification

Those numbers are starting assumptions, not sacred values. Product manuals may need smaller chunks for precise lookup. Policies often need more surrounding text because definitions and exceptions live in different paragraphs. Source code should be split using syntax trees or symbols, not arbitrary character windows.

Overlap can hide bad splitting during a demo, but large overlap creates duplicate candidates and wastes the LLM context window. Deduplicate adjacent results and consider expanding around a winning chunk after retrieval. Searching small units and returning a larger parent section often gives better precision without starving the model of context.

Bad extraction also poisons good chunking. Inspect samples from every source type. If a PDF parser interleaves two columns or drops table headers, fix ingestion before touching the embedding model. We have seen plenty of AI engineering time burned downstream because nobody opened the extracted text.

Stop judging the system by a few friendly questions

A RAG evaluation set should contain questions people actually ask, the evidence required to answer them, and the expected behavior when no answer exists. Include exact lookups, paraphrases, multi-part questions, stale-document traps, permission boundaries, and questions that should produce an abstention.

Generated questions help expand coverage, but they should not be the whole test set. A model generating questions from a chunk tends to produce questions that make that chunk easy to retrieve. That tests the generator's habits as much as your system.

Measure retrieval separately from generation. Recall at k tells you whether any relevant chunk appeared in the candidate set. Mean reciprocal rank shows how early the first relevant result appeared. Neither tells you whether the final answer was faithful, so evaluate groundedness, citation correctness, and answer completeness afterward.

The separation saves time. If retrieval recall is poor, work on ingestion, query rewriting, filters, hybrid search, or reranking. If retrieval is sound but answers wander, work on context assembly, instructions, model choice, and abstention behavior.

A small evaluation record can stay plain and reviewable:

{
 "question": "Which key types require annual rotation?",
 "allowed_tenant": "internal-security",
 "relevant_chunks": ["key-policy-v4#rotation"],
 "must_cite": ["key-policy-v4"],
 "forbidden_sources": ["key-policy-v3"],
 "answerable": true
}

Run these cases in CI when changing parsers, embedding models, chunk rules, retrieval parameters, or prompts. Store the index version and model versions with each result. This is where MLOps discipline earns its keep. Otherwise, a reindex can quietly change behavior while the application code remains untouched.

The model needs permission to say no

The final context builder should prefer a few strong, diverse sources over a pile of vaguely related text. Remove duplicates, label each passage with a stable citation ID, and tell the model to cite claims against those IDs. Validate citations after generation. A citation that points to an available chunk is not automatically a citation that supports the claim.

Set an abstention path for missing or conflicting evidence. Similarity thresholds can help, but a single global threshold is brittle because score distributions change across embedding models and document sets. Calibrate it against the evaluation set, and pair it with signals such as reranker confidence, source agreement, and whether required metadata filters removed all candidates.

Agents make this more serious. A weak RAG answer shown in chat is annoying. The same answer passed to an agent that edits a ticket, changes infrastructure, or emails a customer becomes an action. Retrieval evidence should be carried into tool policy, approval checks, and audit logs.

Security testing belongs here too. Documents are untrusted input. Retrieved text can contain prompt injection, malicious instructions, or data designed to redirect an agent. Treat document content as evidence, never authority. Tools should accept actions from application policy, not from prose found inside a PDF.

If you only do one thing this week

Create 30 representative questions and record the chunks that should answer them. Run the set against your current retriever, without the LLM, and inspect every miss.

That exercise usually exposes more useful work than another round of prompt tuning. It turns RAG from a persuasive demo into an engineering system you can test, secure, and improve.

If this is on your plate, TecLeads does exactly this as part of our AI & Applied ML work. If you'd like a second pair of eyes on your setup, book a 30-minute call or explore what we do.