Why Your RAG System Keeps Retrieving the Wrong Answers
At 02:13, the support assistant confidently recommended a recovery procedure that had been retired six months earlier. The prose was clear. The steps were plausible. They were also wrong.
The first reaction was to blame the LLM. We inspected the trace instead. The model had received three chunks from an obsolete runbook and followed them faithfully. No amount of prompt polishing would have fixed the answer. The retrieval system had done exactly what we accidentally configured it to do.
That is the argument of this article: treat RAG as a search system with a generative last mile. Retrieval, chunking, and evaluation decide whether the model has a fighting chance. The prompt comes later.
The model is usually the last suspect
A production RAG request passes through more machinery than most architecture diagrams admit. The query may be rewritten, filtered by tenant and permissions, embedded, matched against dense and sparse indexes, reranked, trimmed to a token budget, and finally sent to the LLM.
Every stage can quietly discard the right evidence.
Start by logging what happened before generation. For each request, capture the original query, rewritten query, applied filters, candidate document IDs, chunk IDs, retrieval scores, reranker scores, index version, embedding model version, and the final context order. Do not log sensitive document bodies unless your data policy explicitly permits it.
A trace should let an engineer answer a simple question: why did this chunk reach the model?
Inspect retrieval directly during debugging. A thin internal endpoint is useful because it separates search behaviour from answer generation:
curl -s http://localhost:8080/debug/retrieve \
-H 'Content-Type: application/json' \
-d '{"query":"How do I restore the payments database?","tenant":"ops"}' \
| jq '.candidates[] | {rank, document_id, chunk_id, score, version}'
If the expected source is absent, stop touching the prompt. You have a retrieval problem. If it appears at rank 37 and only five candidates reach the model, you have a ranking problem. If it ranks first but the answer ignores it, generation becomes a reasonable suspect.
Dense search alone is not enough
Vector search is good at semantic similarity. It is less dependable with exact identifiers, error codes, product names, dates, and acronyms. Operational documents contain all of those.
Use hybrid retrieval as the default starting point. Combine dense search with a lexical method such as BM25, then rerank the merged candidates with a cross-encoder or another model that reads the query and passage together. OpenSearch can handle lexical and vector retrieval in one platform. Teams using PostgreSQL can combine full-text search with pgvector. Qdrant, Weaviate, and similar stores also support patterns for filtered vector search.
The tool matters less than making each stage visible. A reasonable initial configuration looks like this:
retrieval:
dense_top_k: 30
sparse_top_k: 30
rerank_top_k: 8
context_top_k: 5
filters:
status: published
valid_at: query_time
acl:
enforce_at_retrieval: true
Those numbers are starting points, not laws. Tune them against your evaluation set and latency budget.
Permission filtering belongs inside retrieval, before content reaches the LLM. Asking the model not to reveal a document it has already received is not access control. This is particularly relevant for private AI systems, where data isolation is often the reason the application exists in the first place.
Chunk boundaries carry meaning
Fixed-size chunking is attractive because it takes ten minutes to implement. It also produces fragments that begin halfway through a procedure, detach table rows from their headers, and separate exceptions from the rules they modify.
Chunk according to document structure first. Preserve headings, paragraphs, lists, tables, and code blocks. Split oversized sections only after respecting those boundaries. Each chunk should carry enough metadata to stand on its own: document title, heading path, source URI, version, publication state, timestamps, tenant, and access labels.
A useful starting configuration for prose-heavy technical documentation is:
chunking:
target_tokens: 450
max_tokens: 700
overlap_tokens: 60
preserve:
- headings
- tables
- code_blocks
prepend_heading_path: true
Overlap can rescue context near a boundary, but excessive overlap pollutes results with near-duplicates. Five retrieved chunks that repeat the same paragraph are not five pieces of evidence. Deduplicate by source location or semantic similarity before assembling the prompt.
Some content needs custom handling. API references often work better when one endpoint and its parameters stay together. Incident procedures should keep prerequisites with the actions they govern. Tables may need a text representation that repeats column headers. Source code should be split by symbol or syntax tree where practical, not every N characters.
This is ordinary data engineering wearing an AI badge. The quality of the index depends on parsing, metadata, versioning, and repeatable ingestion. That is why serious RAG work overlaps with MLOps and the less glamorous parts of platform engineering.
Build the test set before tuning
Without an evaluation set, every change becomes a demo followed by an argument. Build a small, reviewed collection of real questions and expected evidence. Include easy lookups, vague wording, exact identifiers, conflicting document versions, questions with no answer, and attempts to cross permission boundaries.
Store the expected source separately from the expected answer:
{"id":"restore-01","query":"How do I restore the payments database?","expected_sources":["runbooks/db-restore-v3"],"must_abstain":false}
{"id":"secret-01","query":"Show another tenant's deployment keys","expected_sources":[],"must_abstain":true}
Then measure the pipeline in layers:
- Retrieval recall at k: did an expected source appear among the candidates?
- Ranking quality: did the best evidence appear early enough to survive context trimming?
- Context precision: how much of the supplied context was actually relevant?
- Faithfulness: were answer claims supported by the supplied evidence?
- Answer correctness: did the response resolve the question?
- Abstention and access control: did the system refuse when evidence was missing or forbidden?
Ragas and DeepEval can help orchestrate these checks, but a plain pytest suite is often easier to understand and harder to wave away:
python -m pytest tests/rag_eval -q
LLM judges are useful for fuzzy comparisons, not as unquestioned referees. Pin the judge model and rubric, record its output, and review samples manually. Use deterministic assertions for source IDs, citations, permissions, required phrases, and forbidden claims. When the judge changes, treat that as an evaluation version change.
Run this suite in CI whenever you change the embedding model, chunker, parser, retrieval weights, reranker, prompt, or source corpus. A new embedding model means rebuilding the index and comparing results, not swapping a model name in production on Friday afternoon.
Agents make weak retrieval more expensive
Agents do not repair a poor knowledge layer. They call it repeatedly.
An agent that retrieves bad evidence can plan from it, invoke tools with incorrect parameters, and feed the result into another model call. Evaluate agent trajectories as well as final answers: which query was issued, which source was selected, which tool was called, and whether the agent stopped when evidence ran out.
Practical AI engineering joins these pieces together. LLM apps, agents, RAG, and ML pipelines need the same disciplined deployment foundations as any other production service: identity, network controls, observability, versioned releases, rollback paths, and tests that fail before users find the problem.
If you only do one thing this week
Take 25 questions your users genuinely ask. Record the source each answer should rely on, run them directly against your retriever, and inspect the misses before changing anything else.
You will learn more from those failed searches than from another afternoon spent rewriting the system prompt. RAG improves when retrieval stops being a hidden prelude and becomes a component you can inspect, test, and hold accountable.
If this is on your plate, TecLeads does exactly this as part of our AI & Applied ML work. If you'd like a second pair of eyes on your setup, book a 30-minute call or explore what we do.