Troubleshooting Common RAG System Failures

Key Takeaways:

Five failure modes silently degrade production RAG systems: embedding drift, chunk-boundary loss, reranker drop, stale-corpus popularity bias, and off-topic grounding. Each leaves a different log signature.

A September 2026 study of RAG over four Indian central-government regulatory documents found an evidence-preservation ceiling above 98%, so the failures that matter come from ranking quality at query time, not missing data at ingest.

A single embedding model upgrade without re-indexing can crater recall; Drift-Adapter recovered 95-99% of the recall of a full re-embedding while adding under 10 microseconds of query latency.

Answer-level metrics hide these bugs. Recall@k looks healthy while the answers are wrong, because retrieval quality and ranking quality are different measurements.

Lexical overlap beats semantic match more often than teams expect: production embedders scored 0.0% strict Hit@1 on structurally disguised math queries while the correct item sat in the top 10 almost every time.

The Bug That Passes Every Dashboard

A September 2026 study of RAG pipelines over four structurally distinct Indian central-government regulatory documents reported an evidence-preservation ceiling above 98%. Ingestion captured nearly all the answer text, and retrieval differences between configurations were driven primarily by ranking quality, not information loss during ingestion (arXiv:2609.31660).

Your dashboard shows healthy recall and full uptime. The pipeline loses the one chunk that carried the answer, quietly. Here are five ways production retrieval-augmented generation systems degrade without firing an alert: the symptom, the diagnostics you should have logged, and the fix.

F1: Embedding drift after a model upgrade

Someone swaps the query-path embedding model for a newer one. The index still holds vectors from the old model. Nothing errors. Queries return results, and the results are subtly, consistently wrong.

What to log for F1

Stamp the embedding model version and dimension on both the indexed vector and the query vector. If those two strings differ anywhere in the pipeline, you have drift before the recall numbers move. Log the index build hash at last full build, and run a nightly canary on the frozen query set that alerts on any top-k recall change above a threshold you set from your own baseline variance.

# Nightly embedding-drift canary. Runs against a frozen gold set,
# NOT live user queries, so recall numbers stay comparable across deploys.
#
# Note: production use should cache gold embeddings, cap the comparison
# window, and alert rather than page on a single bad run.

import numpy as np
from datetime import date

GOLD_QUERIES = load_gold_queries() # (query, relevant_chunk_ids) pairs
DRIFT_ALERT_THRESHOLD = 0.05 # 5 percentage points week over week

def recall_at_k(embed_fn, chunks, queries, k=10):
 doc_vecs = {cid: embed_fn(text) for cid, text in chunks.items()}
 hits = 0
 for q, relevant in queries:
 qv = embed_fn(q)
 top_k = sorted(
 doc_vecs.items(),
 key=lambda kv: float(np.dot(qv, kv[1])),
 reverse=True,
 )[:k]
 top_ids = {cid for cid, _ in top_k}
 if top_ids & set(relevant):
 hits += 1
 return hits / len(queries)

def run_canary(embed_fn, chunks, model_version, index_version):
 if model_version != index_version:
 raise RuntimeError(
 f"DRIFT: vector was built with {index_version} "
 f"but query uses {model_version}"
 )
 score = recall_at_k(embed_fn, chunks, GOLD_QUERIES)
 baseline = load_baseline_recall()
 print(f"{date.today()} recall@10={score:.3f} baseline={baseline:.3f}")
 if baseline - score > DRIFT_ALERT_THRESHOLD:
 notify_oncall(model_version, score, baseline)
 return score

The version guard raises before the scoring runs, catching the most common form of this incident: not drift at all, but a mismatched vector space.

F2: Chunk-boundary loss for cross-section content

A fixed-size splitter slices a safety specification table between a “voltage limit” header and its “240V” value. The vector database stores the header in one chunk and the number in another. A user asks for the voltage limit. Retrieval returns the header chunk, which mentions the concept without stating a value. The LLM, handed a chunk that talks about voltage limits and never quotes one, guesses (VentureBeat, January 2026).

The symptom is wrong answers where retrieved context is topically on-point but numerically or factually incomplete. Questions that span two adjacent chunks, a definition and its application, a rule and its exception, a table header and its cells, fail far more often than single-chunk questions.

The diagnostic is a boundary-spanning evaluation set. Tag every gold question by whether its evidence lives in one chunk or spans two, then report recall separately for each group. If spanning questions score far lower, the chunker is the problem and your aggregate recall is hiding it.

The fix depends on the corpus. A February 2026 taxonomy paper found that simple structure-based methods beat LLM-guided chunkers for in-corpus retrieval, and that contextualized chunking using long-context embeddings helps in-corpus retrieval while hurting in-document retrieval. The paper is explicit that there is no universal winner, which is why the evaluation set matters more than the default you pick.

A structure-aware recursive splitter with a separator hierarchy is the right starting point for most corpora. Section markers come ahead of paragraph breaks, which keeps whole sections together for medical, legal, and policy documents.

Note: The following code is an illustrative example and has not been verified against official documentation. Please refer to the official docs for production-ready code.

from langchain_text_splitters import RecursiveCharacterTextSplitter

# Separators are tried in order. Section markers come FIRST, so the
# splitter only falls through to paragraph and space breaks when a
# section is too long for the size limit.
#
# Note: production use should set length_function to count tokens
# (tiktoken), not characters, and should log the parent section id
# onto every emitted chunk for citation and boundary debugging.

splitter = RecursiveCharacterTextSplitter(
 chunk_size=512,
 chunk_overlap=64,
 separators=[
 "\n## ", # markdown sections
 "\n### ",
 "\n\n", # paragraph break
 "\n", # line break
 ". ", # sentence
 " ",
 ],
 length_function=count_tokens,
 add_start_index=True, # lets you reconstruct the parent span
)

chunks = splitter.create_documents(
 [policy_text],
 metadatas=[{"doc_id": "ENG-SAFETY-114", "section": "voltage_limits"}],
)

# Every chunk now carries the section it came from, so a wrong answer
# can be traced to a chunker boundary instead of a retrieval miss.
for c in chunks:
 c.metadata["chunk_hash"] = stable_hash(c.page_content)

Storing the parent section id and a start index on every chunk turns a failed answer into a chunking bug report. Without those fields you cannot tell whether the retriever missed the document or the chunker split it badly.

F3: Recall@k is fine but the reranker drops the answer

Your retriever returns a wide candidate list and the answer-bearing chunk sits deep in it, well outside the top ranks. Your cross-encoder reranks down to the top 10 and the chunk is gone. Retrieval-stage recall at the candidate depth was high. Answer accuracy drops anyway. This is the failure that convinces teams reranking “does not work,” when the real problem is a mismatch between what the first stage optimizes and what the second stage rewards.

The June 2026 poisoning study named the mechanism precisely: document-level signals fragment during chunking, while rerankers favor locally coherent, answer-bearing passages over globally optimized semantic similarity (arXiv:2606.11265). The same property that makes rerankers good at picking clean passages makes them sensitive to chunk size and to whether the answer chunk is self-contained.

The symptom is a metric split. Track recall at the candidate depth before reranking and recall at the final depth after it, separately. If the first is flat and the second is falling, the reranker is the suspect. Questions whose evidence spans a boundary degrade more after reranking than cleanly bounded ones, because a fragment competes poorly against a coherent passage.

The diagnostic is a rerank-delta log per query: the gold chunk’s rank before reranking, its rank after, and the scores assigned to the top candidates. When the gold chunk falls out of the final set, you have a labeled failure case. Aggregate the deltas weekly. A reranker that helps will improve median gold rank; one that hurts will move it down while leaving pre-rerank recall untouched.

The fix is to give the reranker more coherent input, not a bigger candidate pool: larger minimum chunk sizes at ingestion, parent-document retrieval so the reranker sees the section rather than the slice, or a reranker whose training distribution matches your domain. Cross-encoder reranking of a few dozen candidates is cheap enough to run on every query; the cost is in candidate quality, not compute.

Stage	Metric to log	Healthy signal	Broken signal
ANN retrieval	recall at candidate depth vs. gold	flat or rising deploy over deploy	flat, so the first stage is not the suspect
Reranker	recall at final depth and median gold rank	median gold rank improves after rerank	pre-rerank recall flat but post-rerank recall falling
Generation	answer faithfulness vs. retrieved chunks	answer supported by cited chunk	confident answer contradicted by the chunk it cites

The second row is the one most teams skip, which is exactly where this failure hides.

F4: Stale corpus and popularity bias

A handful of high-traffic documents dominate your corpus: the onboarding handbook, the API reference, the incident postmortem template. They get edited and re-embedded constantly. A small set of deprecated documents sits untouched, still indexed, still retrievable. Over time the dense index reflects document churn rate rather than document authority, and queries return the most-edited document rather than the most-correct one.

The symptom is a specific kind of wrong answer: correct months ago, stale now. The model confidently cites a document that used to be right. Nobody files a bug, because the answer looked plausible and matched the citation.

The diagnostic is a freshness histogram over retrieved chunks, not over the corpus. For every query, log the age of each retrieved chunk (index timestamp minus document update timestamp). If median retrieved-chunk age climbs while corpus median age is stable, retrieval is drifting toward the churny documents. Log per-document retrieval frequency too. If a small set of documents accounts for most retrieved chunks, you have a concentration that has nothing to do with query intent.

The fix has three parts. Timestamp every chunk at ingestion and add recency as a ranking signal or a hard filter for time-sensitive query classes. Garbage-collect with a nightly job that re-embeds changed documents and deletes vectors whose source no longer exists. And add a staleness field to the prompt so the model can see when evidence was last updated. The facet-level diagnostics work from April 2026 found that hallucinations in these systems are driven less by retrieval accuracy and more by how retrieved evidence is integrated during generation, with “prior-driven override” as a recurring pattern. A model handed stale evidence that conflicts with its parametric knowledge frequently resolves the conflict by trusting itself.

Failure mode	Primary symptom	Key diagnostic log	Fix
F1 Embedding drift	top-k recall slides over days, no error	Embedding model version on every vector	Re-index, or a learned adapter to defer recompute
F2 Chunk-boundary loss	Topically right, factually incomplete answers	Recall split by single-chunk vs. span questions	Structure-aware chunking with parent section ids
F3 Reranker drop	Pre-rerank recall flat, post-rerank recall falls	Gold-chunk rank before vs. after rerank	Larger min chunk, parent-doc retrieval, domain-matched reranker
F4 Stale corpus	Plausible but outdated answers	Age histogram of retrieved chunks	Recency signal, vector GC, staleness in prompt
F5 Off-topic grounding	Confident answer from tangential evidence	Per-facet faithfulness and evidence sufficiency	Abstention policy and sufficient-context gate

F5: Off-topic retrieval and grounded hallucination

The last failure is the most dangerous because it looks like success. Retrieval returns something relevant-ish. The model produces a fluent, cited, confident answer. The answer is wrong because the retrieved evidence never contained the answer, only vocabulary that overlapped the question.

A September 1, 2026 study measured this directly. On structurally disguised queries in competition mathematics, strict Hit@1 at the heaviest disguise tier was 0.0% for both production embedders, while the correct item sat in the top 10 almost always. In 95.2 to 99.8% of the misses, the winner was more lexically similar to the query than the correct answer (arXiv:2609.01556). Surface form beat structure. An LLM reranker recovered 5 to 63% of the gap in mathematics and 43 to 76% in the agent-trajectory domain, but the effect sizes varied by domain, and part of the mathematics recovery concentrated on well-known competitions, which points to memorization rather than retrieval.

The symptom is a high-confidence answer to a question your corpus does not actually answer. The citation points to a real chunk that does not state the claim. Facet-level tracing breaks the question into atomic reasoning steps and scores each for evidence sufficiency; the recurring findings are evidence absence (the chunk lacks the needed fact) and evidence misalignment (the model uses a retrieved fact for the wrong step).

The diagnostic is a sufficiency label per retrieved context, computed offline over a few hundred query-context pairs. Google’s sufficient-context work found that larger models answer well when context is sufficient but often produce an incorrect answer rather than abstain when it is not, and that the mere presence of retrieved text inflates confidence and raises the tendency to answer instead of saying “I do not know” (arXiv:2411.06037). The fix follows: a selective-generation gate that decides answer-or-abstain from a sufficiency signal. In that study, adding sufficient context as an extra signal improved the share of correct answers by 2 to 10% across Gemini, GPT, and Gemma models.

The wrong choice here is chasing a better generator. If the retrieved context does not contain the answer, no amount of model quality produces a correct one, and a larger context window can make it worse by raising confidence further. Two related findings show how stubborn this is: cascading hallucination in multi-step pipelines is missed by output-level detectors, which caught 18.5% of error propagation while a stage-level framework reached 82.1% (arXiv:2606.04435), and embedding-based hallucination detection that achieved a 0% false-positive rate on synthetic hallucinations hit a 100% false-positive rate on real ones, because the hardest cases are semantically indistinguishable from faithful answers (arXiv:2512.15068). Surface similarity is not a grounding signal. Our guide to enterprise RAG architecture and costs sets answer faithfulness above 0.85 and recall@5 above 0.50 as production targets; if sufficiency labeling shows you are not reaching sufficient context, the knowledge base or the retriever is the bottleneck, not the LLM.

A forensic checklist

Every fix above comes down to the same instrumentation: version everything, log the chunk that produced each answer, and split your retrieval metrics by the failure mode you want to catch.

Version the index with the embedder. Stamp embedding model, dimension, and index build hash on every vector and query.
Split recall by question type. Single-chunk and boundary-spanning questions need separate numbers. One aggregate hides the other.
Log gold-chunk rank at every stage. Pre-rerank and post-rerank. The delta is the reranker’s report card.
Track retrieved-chunk age, not corpus age. Popularity bias shows up in what you retrieve, not what you store.
Label context sufficiency offline. If a large share of test queries lack sufficient context, fix retrieval before touching generation.
Evaluate end-to-end, not just retrieval. As we discussed in our comparison of chunking methods for document retrieval, retrieval-only metrics reward strategies that produce worse answers. Pair them with a judge model and a faithfulness check.

The pattern across all five is the same: the pipeline drifts. Nothing throws. Recall looks fine. The answers are confidently wrong. The only defense is instrumentation you set up before the incident, because by the time a user files the bug, the deploy window that caused it has scrolled off the dashboard.

Key Takeaways

Five failure modes silently degrade RAG systems: embedding drift, chunk-boundary loss, reranker drop, stale-corpus popularity bias, and off-topic grounding. Each leaves a different log signature.
A September 2026 study of RAG over four Indian central-government regulatory documents found an evidence-preservation ceiling above 98%, so the failures that matter come from ranking quality at query time, not missing data at ingest.
A single embedding model upgrade without re-indexing can crater recall; Drift-Adapter recovered 95-99% of the recall of a full re-embedding while adding under 10 microseconds of query latency.
Answer-level metrics hide these bugs. Recall@k looks healthy while the answers are wrong, because retrieval quality and ranking quality are different measurements.
Lexical match beats semantic match more often than teams expect: production embedders scored 0.0% strict Hit@1 on structurally disguised math queries while the correct item sat in the top 10 almost every time.

More in-depth coverage from this blog on closely related topics:

Sources and References

Sources cited while researching and writing this article: