We Taught the AI to Say 'I Don't Know': The Hallucination That Almost Cost a Client (and the RAG Guardrails That Stopped It)
Three minutes before the memo went out, a senior partner at a UAE law firm caught it. The RAG assistant had cited a precedent with perfect confidence: correct formatting, a plausible case name, the works. The case did not exist. Here is why that happens, what a production guardrail stack actually looks like, and how we test for the failure you cannot see. The uncomfortable part is the conclusion. If your RAG system cannot refuse to answer, it is not a research tool. It is a liability generator with good grammar, and the math on fixing that is not close.
Why RAG Systems Hallucinate Even With Source Documents
Most people assume retrieval-augmented generation fixes hallucination by grounding the model in real documents. It doesn't. Understanding why is the whole game if you want to build something that holds up inside a law firm or a clinic.
Here is what actually happens. When you query a RAG system, the retriever returns the top-k chunks ranked by embedding similarity. If the right document isn't in the corpus, whether it was never ingested, chunked badly, or the question simply sits outside the domain, the retriever doesn't come back empty. It returns the closest thing it found. The model then receives that marginally relevant chunk and writes an answer that looks exactly like a correct one, because it learned from a world where legal citations and clinical references are produced fluently and with total confidence. The model cannot tell high-confidence retrieval from low-confidence retrieval. To the generator, both look the same.
A 2025 Google study, published at ICLR, measured how bad this gets. The finding, in the authors' own words, is that adding context "paradoxically reduces the model's ability to abstain." When the retrieved context is insufficient to answer the question, state-of-the-art models still produce a correct response only 35 to 62 percent of the time. The rest of the time they don't fall silent. They fabricate. The abstention numbers are stark. Give Claude 3.5 Sonnet no context and it declines to answer 84.1 percent of unanswerable questions; hand it loosely-related RAG context and that drops to 52 percent. Gemini 1.5 Pro falls from 100 percent abstention to 18.6 percent. More context made the models more confident, not more cautious. So the default behavior of an unguarded RAG system, asked something it cannot answer from its corpus, is to confidently invent a plausible-sounding fiction. In legal AI, that fiction is a case citation. In clinical AI, it is a dosage recommendation. Same mechanism, very different consequences.
Why Cosine Alone Leaks: The Reranking Layer
There is a failure in first-stage retrieval that a similarity threshold cannot catch on its own, and it is worth naming before we get to the gates. The embedding model most retrieval stacks ship with is a bi-encoder, something in the family of all-MiniLM-L6-v2. A bi-encoder encodes the query and each chunk independently, into separate vectors, then compares them with cosine similarity. That independence is what makes it fast enough to search millions of chunks. It is also what makes it blind to negation.
Because the query and the chunk never see each other during encoding, the bi-encoder has no way to register token-level interaction. It cannot reliably tell "the clause permits assignment" from "the clause prohibits assignment." The two sentences share almost every word. They differ by the one word that carries the entire legal meaning, and in embedding space they sit close together. The same trap waits in a clinical formulary, where "the regulator allows off-label promotion" and "the regulator prohibits off-label promotion" embed as near-neighbours. A semantically inverted chunk, the one that says the opposite of what the query needs, can score well above your 0.70 cutoff and sail through.
The fix is a cross-encoder reranker, and it runs between first-stage retrieval and the confidence check, not in place of it. A cross-encoder concatenates the query and the candidate chunk into a single sequence ([CLS] query [SEP] passage [SEP]) and pushes them through the transformer together, so every query token attends to every chunk token. It scores the pair jointly. That joint pass is exactly what lets it catch the negation a bi-encoder misses. You pay for it in latency, so account for it honestly. Reranking 30 candidates is typically 30 to 50 ms on a T4-class GPU and 100 to 150 ms on CPU. Cap it with a roughly 200 ms timeout and fall back to first-stage top-k if it blows the budget. When first-stage precision is weak, with relevant chunks scattered below the top ranks, a reranker buys a 5 to 15 point lift in nDCG. The pattern is layering, not replacement. The reranker also produces a cleaner, better-calibrated relevance score, which means it tightens Gate 1 rather than introducing a separate failure mode. You self-host an open reranker or call a managed one; either way the structure is the same.
The Guardrail Stack We Actually Deploy
No single guardrail prevents hallucination in production. It takes a layered pipeline, where each layer catches a failure mode the others miss. Four gates do most of the work.
The first is retrieval confidence thresholding. Every chunk the vector store returns carries a cosine similarity score against the query, and we reject any query whose top chunk scores below 0.70. The data backs that cutoff. At 0.70, roughly 98.6% of irrelevant queries fall below threshold while 88.9% of relevant ones clear it. Pushing to 0.80 backfires. The retriever starts rejecting valid queries, and the system quietly degrades into an ungrounded generator, which is exactly the thing you were trying to prevent.
The second gate is mandatory verbatim citation. The system prompt forces the model to quote its source chunk directly whenever it makes a factual claim. If it can't quote, because the chunk doesn't actually say what it's about to assert, the generation fails on inspection. This isn't bulletproof. A capable model can hallucinate a quote that sounds like the chunk. But it stops most low-effort fabrication and, more importantly, it makes hallucination auditable instead of invisible.
The third gate is abstention triggering. When the top similarity score drops below threshold, the system returns a structured non-answer that names the gap rather than papering over it. Short, firm, and logged every time.
The fourth gate is human escalation routing. Queries we classify as high-stakes, such as contract interpretation or treatment decision support, get flagged before the response reaches anyone, not after the damage is done.
Walking One Query Through the Stack
The four gates are easier to trust when you watch them fire in sequence, so take the query the law firm started with. A paralegal asks the assistant to find precedent for a specific contract-interpretation point, and that point, as it happens, has no precedent anywhere in the firm's indexed corpus. Nobody has litigated it. No memo covers it.
Gate 1 fires first. The retriever pulls its candidates, and the top chunk scores cosine 0.61 against the query, below the 0.70 cutoff. The system does not generate. Gate 3 takes over and returns the structured non-answer instead: "No precedent in the indexed corpus addresses this point; escalate to a partner before relying on it." In the same step it writes a row to the log, capturing the query, the top similarity score, and the abstention reason, so the refusal is auditable rather than silent. Now run the same query through an unguarded pipeline. No threshold, no abstention. The generator receives that 0.61 chunk, treats it like any other, and produces the perfectly-formatted, plausible, nonexistent case the firm caught three minutes before the memo went out. Same input, two outputs, one difference. And the logged abstention is worth more than a dead end. It is a corpus-coverage signal. A cluster of refusals on the same topic tells you exactly what to ingest next.
Testing What You Cannot See: Adversarial Corpus Evaluation
A RAG system that aces the questions it can answer tells you nothing about what it does when it can't. That second behavior is the one that gets you sanctioned, and most teams never test for it.
The test that matters is the adversarial unanswerable set: questions with no correct answer anywhere in the corpus. We build these on purpose. Cases that aren't in the legal database, drug interactions absent from the clinical formulary, regulations not yet ingested. Then we measure whether the system correctly abstains instead of guessing. Our bar before go-live is 90% correct abstention on that adversarial set, and I'll be blunt about why the number is that high. Below 90%, the system fabricates at a rate that surfaces in client-facing use within weeks, not months. Correct abstention means returning the structured non-answer, not a confident wrong one.
For the generation side, we use the RAGAS framework. Faithfulness, the share of claims in an answer that are actually supported by the retrieved chunks, is our floor metric, and in our deployments 0.75 is the line for production. Under that, users hit hallucinations or context drift often enough that it becomes operationally visible. Healthcare RAG systems we have tuned in demanding deployments have reached 0.995 faithfulness. Legal systems land more realistically between 0.80 and 0.92, depending on corpus quality and how varied the queries are. The retrieval side has its own RAGAS metric, and it is the one a reranker moves. Context precision rewards a system for ranking the genuinely relevant chunks above the noise, which is exactly what the cross-encoder reorder is for. Context recall, whether the relevant material was retrieved at all, sits with the first-stage retriever and your choice of k, not with reranking.
One thing teams forget: the test set needs maintenance. As the corpus grows, the line between answerable and unanswerable moves. A question that was unanswerable in month one may be perfectly answerable by month three. Adversarial testing is a recurring discipline, not a launch-day checkbox you tick once and retire.
The Business Case Is Not Optional: Liability, PDPL, and the Cost of One Wrong Answer
U.S. courts have been quietly generating the dataset on what happens when this guardrail stack is missing. In February 2025, Morgan & Morgan attorneys were sanctioned after their internal AI tool inserted eight fabricated case citations into court motions, with financial penalties landing on three named attorneys. In May 2025, Ellis George LLP and K&L Gates faced roughly USD 31,000 in sanctions after 9 of 27 citations in a supplemental brief turned out to be wrong, at least two of them citing cases that did not exist. Legal analytics now track more than 1,000 court cases involving AI-generated hallucinations. In Q1 2026 alone, U.S. courts imposed over USD 145,000 in sanctions for AI hallucinations in legal filings.
The UAE adds a regulatory layer on top of the liability one, and it pays to scope it precisely rather than wave at it. Under the federal PDPL, Federal Decree-Law No. 45 of 2021, Article 18 gives a data subject the right to object to decisions made solely by automated processing where those decisions carry legal consequences or seriously affect them. That is the objection right, and it is the limit of what Article 18 does. The duty to disclose information about automated decision-making sits separately, in Article 13, the right to obtain information. And there is no express statutory right to human intervention in the PDPL at all. This matters for design. The verbatim-citation gate is how you actually satisfy the Article 13 disclosure duty in practice, and the human-escalation gate is the conservative posture that keeps a high-stakes answer from ever being a solely-automated decision in the first place. The guardrails are the compliance implementation, not just engineering hygiene. One honest caveat: the federal PDPL Executive Regulations remain unpublished as of 2026, so the enforcement machinery is mid-transition. The obligations are binding now regardless, and on-premise guardrails are the posture that holds however the regulations land.
Healthcare carries a stricter layer than "keep a human in the loop." The Dubai Health Authority's Policy for Use of Artificial Intelligence in Healthcare, first issued in August 2021 and reinforced by a 2025 DHA circular on AI and data protection, requires that AI support clinical decision-making rather than replace it, and it goes further on failure behavior. As legal commentary on the policy frames it, the system must degrade gracefully: raise automatic alerts and cease operation in a controlled way on malfunction, never fail silently. An abstaining RAG assistant is that requirement expressed in software. There is also a residency reason the inference server lives on local hardware for a clinic. Patient records flowing through the model intersect the national health information exchanges, NABIDH in Dubai, Malaffi in Abu Dhabi, and Riayati in the Northern Emirates, now interconnected under the National Unified Medical Record. By the end of June 2025, NABIDH alone held more than 10.41 million medical records across 1,888 licensed facilities, with 53,659 professionals and 91 EMR systems integrated. This is regulated health data, and the residency expectation around it is a composite rather than one statutory line: PDPL cross-border transfer gateways that are not operationally switched on, DHA governance over Dubai health data, and a clear vendor best practice of UAE hosting. On-premise is what satisfies all three at once.
Now weigh that against the cost of the fix. The guardrail stack described above is four to six hours of engineering configuration per deployment. On the other side of the ledger sits unlimited professional liability and live regulatory exposure. The engineering hours have never once been the expensive option.
Questions about your setup?
We help UAE SMEs build AI systems that are compliant, on-premise, and actually useful. Free initial conversation.