First in a series on how RAG systems fail in production. This one covers the slice between retrieval and generation, where the document arrives correct and the answer comes out wrong anyway.
Your retriever pulled the right policy document. The chunk containing the answer is sitting in the context window. The model read it and said something else.
The reflex is to go and work on retrieval. Better chunking, a reranker, hybrid search, tighter similarity thresholds. That work is not wasted in general, but for this particular failure it is aimed at the wrong subsystem, and two separate mechanisms suggest parts of it make the problem more likely rather than less.
What actually fails here
Retrieval puts text into a prompt. Nothing about that compels the model to use it.
The generator is still doing what it always does: producing the most probable continuation given everything in its context, weighed against everything encoded in its weights. Retrieved text is one input to that calculation, not a constraint on it. When the retrieved passage says your return window is 14 days and the model's training data says return windows are usually 30, both numbers are in play.
This has a name in the literature, and it is one of three conflicts that matter once you put a retriever in front of a model.
| Conflict | What collides | How RAG creates it |
|---|---|---|
| Context-memory | Retrieved passage vs the model's training prior | Any retrieval over content that differs from the public web |
| Inter-context | One retrieved passage vs another | Raising top-k, adding a second index, query expansion |
| Intra-memory | The model against itself across paraphrases | Present regardless, surfaced when users rephrase |
The first row is the subject of this post, and it is the one that gets folded into the word "hallucination" and then handed to whoever owns the retriever.
Why the standard fix misses
Here is the finding that reorders the problem.
Xie et al. ran the first controlled study of what LLMs do under knowledge conflict, and found two behaviours that look contradictory until you notice what separates them:
LLMs can be highly receptive to external evidence even when that conflicts with their parametric memory, given that the external evidence is coherent and convincing. On the other hand, LLMs also demonstrate a strong confirmation bias when the external evidence contains some information that is consistent with their parametric memory, despite being presented with conflicting evidence at the same time.
What separates them is whether the evidence in the context is clean or mixed.
The operational consequence is not obvious, so it is worth being explicit about it.
The standard retrieval playbook increases how many passages you put in front of the model:
- Retrieve top-10 instead of top-3
- Add a second index, or a keyword leg alongside the vector leg
- Expand the query with rephrasings to catch more phrasings of the same question
Each of those raises the chance that the context now holds a passage agreeing with the model's prior alongside the passage that contradicts it. You have moved the system out of the regime where it follows context and into the regime where it falls back on what it already believed.
The retrieval work made retrieval better and grounding worse.
There is a companion finding that cuts the same way: larger models default to parametric knowledge more readily than smaller ones. Upgrading the model is the other reflex when a RAG system starts lying, and it is not reliably a fix either.
The second mechanism, which is about position
Evidence composition is not the only thing that changes when you retrieve more. Position changes too.
Language models use information most reliably when it sits at the beginning or the end of the context window, and least reliably when it sits in the middle. The curve is U-shaped. This is Lost in the Middle (Liu et al.), which most RAG practitioners have at least heard of, and which holds even for models built for long contexts.
Put the two together. You went from top-3 to top-10 to improve recall. The passage you needed was at rank two, so it is still there, but it now sits behind seven others and has moved from the front of the context into the middle. Recall went up. Attention on the passage that mattered went down.
The usual answer to this is semantic relevance filtering: set a minimum similarity threshold and drop the passages that are topically related but not needed. That is sound and I would take it. Notice, though, that it is a retrieval-side adjustment offered for a generator-side effect. It works by making the context smaller and cleaner, which reduces exposure without touching the mechanism. If your correct passage still lands in the middle of what survives the filter, you are still in the dip.
One qualifier worth carrying: this class of failure is task-dependent. Conflicts have little effect on tasks needing minimal knowledge from the context, and bite hardest on knowledge-intensive work. A summariser is less exposed than a policy assistant. If your product answers questions from documents, you are in the exposed category.
What the research says, including where it disagrees
This literature has not settled, and posts that flatten it are misleading you.
| Work | Claim |
|---|---|
| Longpre et al., 2021 | Models over-rely on parametric memory under conflict |
| Chen, Qian, Tan et al. | Models ground in the retrieved documents |
| Xie et al., 2024 | Both, depending on whether the evidence is mixed |
| Three-regime framework, 2026 | Parametric dominance, context dominance, and a hybrid band, predicted by conflict magnitude |
One caveat on that table. I have read Longpre and Xie directly. The Chen, Qian and Tan results I know only as they are catalogued in survey work on conflicts in text, so treat that row as secondhand and go to the sources if you are going to rely on it.
Nobody has given you a rule that says which regime your system is in. That is the honest state of it, and if you are building on this you should treat regime as something to measure on your own data rather than something to look up.
What is not in dispute is that the failure persists at the frontier. Vectara's FaithJudge leaderboard, which scores models on RAGTruth and FaithBench, currently puts the best of them near 6 percent: Gemini-2.5-Flash at 6.26, Gemini-2.5-Pro at 6.65. GPT-4.1 sits at 11.94, Claude-3.7-Sonnet at 16.05. Below 10B parameters, rates above 25 percent are ordinary.
That is your floor before you have written a line of application code. The spread between 6 and 16 also means model choice is a real lever here, just not the one that closes the gap.
The test that tells you whether you have this
Cheap, and worth running before any retrieval work:
# For each complaint, was the gold passage actually in the context
# at generation time? If it was, reranking will not save you.
grounded_but_wrong = 0
retrieval_miss = 0
for case in complaints:
ctx_ids = {c.doc_id for c in case.retrieved_chunks}
if case.gold_doc_id in ctx_ids:
grounded_but_wrong += 1 # generator-side problem
else:
retrieval_miss += 1 # retrieval-side problem
print(f"generator: {grounded_but_wrong} retrieval: {retrieval_miss}")
If the first bucket dominates, every hour spent on chunk size is an hour spent on the wrong half of the system. Most teams have never split the number, which is why the retrieval reflex survives.
What actually works
The interventions that address this live at generation and decoding, not retrieval.
| Intervention | What it does | What it costs |
|---|---|---|
| Abstention | Refuses when confidence against the context is low | Coverage, and the threshold is a product decision |
| Claim verification | Entailment-checks each claim against the passage | A second model call per response |
| Context-aware decoding | Contrastively amplifies what the context implies | Neutral regression, plus a threshold you now own |
The best studied is context-aware decoding. Rather than trusting the model to weigh context appropriately, CAD runs generation twice, with and without the retrieved passage, and forms a contrastive distribution that amplifies the difference. It is an explicit thumb on the scale toward what the context says. The original work reports a 14.3 percent improvement in factuality metrics for LLaMA on summarisation.
Claim verification is the cheapest thing that catches the nastiest variant:
for claim in extract_claims(answer):
verdict = nli(premise=retrieved_passage, hypothesis=claim)
if verdict != "entailment":
flag(claim) # citation is real, the sentence it supports is not
That variant deserves dwelling on. It is why this failure survives review: the source is real, the link works, the quote is close enough to skim past. Everything a human or a monitoring dashboard checks comes back green.
What that costs
CAD has a failure mode of its own, and it is a bad one. When the retrieved context is uninformative, amplifying it drags the model away from an answer it would have gotten right unaided. The literature calls this neutral regression, and it means naive CAD can lower accuracy on exactly the queries where retrieval added nothing.
The fixes are gates: backing off to ordinary decoding when the context carries no signal, or scaling the tilt by how far the two distributions have actually diverged, which AdaCAD measures as the Jensen-Shannon divergence between the contextual and parametric distributions. Both add a tuned threshold you own forever.
None of this is free, which is why retrieval work stays popular. It is easier, it moves metrics you already collect, and it produces visible improvement on the dashboard you happen to be looking at. It just does not touch this.
I do not have a good answer for the case where retrieved passages genuinely conflict with each other and neither is stale. Deduplicate and you might drop the correct one. Present both and you are in the regime where the model falls back on its priors. I have not found a satisfying treatment in the literature either, and if you have one I would like to read it.
