RAG hallucination guardrails: a five-layer design
Five guardrails that stop a RAG assistant inventing answers: refusal before retrieval, a similarity floor, the prompt, a sentence gate and voice rules.
Retrieval-augmented generation (RAG) does not stop a model inventing answers, and weak context can make it more confident rather than more careful. RAG hallucination guardrails belong mostly in code around the model. Build them as five layers in a fixed order, the way this site's assistant works:
Refuse questions the assistant must never answer, before retrieval.
Give the model only evidence it can be held to: passages that clear a similarity floor, or the site's own records for questions about the work as a whole. With neither, refuse without calling it.
Treat the system prompt as the one layer the model can ignore.
Release the answer sentence by sentence, only after deterministic checks.
Never let the assistant speak as the person behind it.
A spend check before the model call completes the design.
I built and maintain the site, assistant included. I publish no accuracy figure for it, and the section on measurement explains why.
Why RAG still hallucinates
In a RAG system, a retriever returns passages from a corpus and the language model writes its answer from them. A hallucination is a false or unsupported statement delivered as if it were true.
Retrieval narrows what a model is likely to say. It does not bound it.
Retrieval can make a model more confident, not more correct
Google researchers separated two cases: retrieved context that is sufficient to answer the question, and context that is not. Summarising their paper, which appeared at ICLR 2025, the authors are blunt: "while RAG generally improves overall performance, it paradoxically reduces the model's ability to abstain from answering when appropriate." In one example, Gemma went from incorrect answers on 10.2% of questions with no context to 66.1% with insufficient context.
Knowing when not to answer pays: the authors' selective generation method improved the fraction of correct answers, among the times the model responded, by 2–10% for Gemini, GPT and Gemma.
A top-k retriever returns k passages whether or not any is relevant. If the model alone judges whether they are enough, abstaining rests with the component this research shows often gets it wrong.
Models are rewarded for guessing
Kalai and colleagues argue that "language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty." When grading gives no credit for "I don't know", guessing raises the score. An instruction to admit uncertainty pushes against that incentive. It helps. It is not a control.
Commercial legal tools still hallucinated
Some legal research providers promoted RAG as "eliminating" hallucinations. The first preregistered evaluation of AI legal research tools found that the products from LexisNexis and Thomson Reuters each hallucinated between 17% and 33% of the time. What that means for legal products is in legal AI hallucinations: what to build.
The five layers at a glance
Layer | What it stops | Runs in | When it fires |
|---|---|---|---|
1. Refusal patterns | Answers the assistant must never give | Code | Fixed refusal before retrieval |
2. Similarity floor | Answers built on weak evidence | Code | Fixed refusal, no model call |
3. System prompt | Rule-breaking, when the model complies | The model | Nothing enforces it on its own |
4. Sentence gate | Unsourced claims, invented values, first person | Code | Stream stops, the rest is withheld |
5. Voice rules | Commitments made in someone else's name | Code and prompt | Through layers 1, 3 and 4 |
The order matters as much as the layers. Both refusal layers run before the language model is called, and the prompt, the only layer that depends on the model's cooperation, is followed by a deterministic check. The spend check between the floor and the prompt does not stop fabrication, but it belongs to the same failure family.
Layer 1: refuse before you retrieve
The first failure is not a missing answer. It is an answer that should not exist.
Questions your corpus can answer but your assistant must not
On this site, a question about fees would retrieve the services page. The text is relevant, and answering from it is exactly the mistake: any figure, date or commitment the model composes from that page is a representation nobody made. Relevance is not permission.
So some questions are declined before anything is embedded or retrieved:
Engagement terms: fees, availability and hiring.
Private details the site deliberately does not publish, such as a phone number, home address, marital or military status, or date of birth.
Speculation: predictions, and comparisons with named people.
Meta requests that try to extract or override the instructions.
Task requests: writing code or text, translating or solving something, rather than asking about the published work.
The refusal is fixed copy, not model output: a refusal generated by the model is a refusal the model can talk itself out of. The clients for the embedder (the model that turns text into vectors) and the language model are built lazily, so a refusal here makes no external call.
For your own system, list the questions whose correct answer is "not from this assistant", whatever the corpus holds.
Patterns, not a classifier
The refusal set is a list of regular expressions for English and Persian, and of phrase lists for the site's other eight languages, all run against a match-folded question: case-folded, with Arabic letter forms folded to Persian ones, Persian and Arabic digits folded to Latin, Arabic short-vowel marks removed, and the zero-width non-joiner inside Persian words turned into a space. The question is also tried with my name read as a pronoun, so "what are Mohammad Panahi's rates?" is declined like "what are his rates?", and a follow-up is checked together with the question before it.
It is not a classifier. A classifier that is correct most of the time is still a promise this site cannot make, because its misses are commitments in someone's name.
The limit is plain: patterns miss paraphrases. Persian spelling is not one of the gaps. Because the fold turns a zero-width non-joiner into a space, a word the standard orthography joins arrives at the patterns split in two, and every Persian pattern accepts both that form and the unspaced one, so چقدر میگیرد and چقدر میگیرد are both declined. A question phrased in words the list does not hold still gets through, and the later layers are there for it: the prompt forbids rates and availability, and the gate stops a sentence with a first-person word it recognises or a figure the sources do not contain. That narrows the gap. It does not close it. The lists for languages other than English and Persian are shorter, and paraphrases in them get through more often.
Layer 2: a similarity floor that skips the model call
After retrieval, the assistant takes the best similarity between the question and the passages it would hand to the model. Below a floor, a question that is not about me is refused with fixed copy, and the language model is never called. The refusal is not free, since in production a hosted API has already embedded the question, but it removes the generation call, so weak evidence never becomes a fluent answer. Questions about me take a different path, described below, because a similarity score cannot tell a broad question about someone's work from an off-topic one.
Why you cannot threshold a reciprocal rank fusion score
The retriever is hybrid: dense vector search and whole-word keyword search, fused by reciprocal rank fusion (RRF). How that copes with Persian and English side by side is covered in hybrid search for Persian RAG.
RRF is a good way to combine rankings and a bad number to threshold. In Cormack, Clarke and Büttcher's formulation, a document's score is the sum of 1/(k + r(d)) over the rankings, where r(d) is its rank in each list and k = 60 "was fixed during a pilot investigation". The score is built from positions alone, so a corpus of pure noise still produces a first-ranked document with the best fused score on offer. A floor on that number measures rank, not relevance.
What the floor compares instead
Each retriever that can produce a bounded similarity reports one:
Vector search reports cosine similarity, clamped to the range 0 to 1. The floor uses the similarity to the question as asked and, for a follow-up, to the question joined with the one before it. A follow-up worded as a position in a list, such as "the second one", is joined with the item it points at as well: the position is read from the question, the list from the previous answer by the site's own titles and names or by what it quotes, and the model is told which item it is. The item is joined only when the position and the list can both be read without doubt; otherwise the start of the previous answer is joined instead. The test reads the wording, not the answer, so an unrelated follow-up that happens to say "the second" is joined with the start of the answer too and can clear the floor; the prompt and the gate still stand behind it. When a question is about me, retrieval also searches with my name attached. That helps order the passages but never counts toward the floor: a name makes any question about me look close to the CV, including ones the site cannot answer.
Keyword search reports coverage: the share of the question's content words, three characters or longer (two in Chinese), found as whole words in the passage. Common words are removed first, in each of the site's languages, and coverage counts toward the floor only when at least two different content words match. One matched word is a coincidence. For a follow-up that says nothing but which item it means, such as "Tell me more about the second one", the content words are those of the site's own title or name for that item.
Each passage keeps the higher of the two, and the floor is compared with the best across the passages returned; the passage that sets it is always among them. The fused score only decides the order.
Whole words matter. Match substrings instead, and "art" is found inside "start", so unrelated questions begin to look like evidence. If the embedding service is unreachable, retrieval falls back to keyword search alone, and the floor still applies.
Questions a similarity score cannot answer
"What has he done recently?" has no nearest neighbour. Its answer is a list sorted by date, and no passage is similar to that: on this site's embedder it scored below a question about the weather. So questions about the work as a whole are recognised by patterns in each of the site's ten languages, like the refusal set. That covers what I do now or did recently, my career, projects, research, writing, education, skills and services. Those questions are answered from the site's records: entries from the CV, the case studies, the articles and the services, read from the database for each question. The records come in the reader's language where a translation exists, and the dated ones are ordered by their own dates, most recent first.
Records are chosen by rule, not by similarity, so the floor does not apply to them. They are exempt from nothing else. The refusal patterns run first, every dated record carries its dates at the precision the site shows, and every sentence built on one passes the same gate.
A question about me that the patterns do not route to the records, and that no passage supports, is given a short overview of those records rather than refused, because a wrong refusal there tells the reader the site does not cover something it does. If the records do not answer it either, the model is told to reply with a fixed token instead of prose, and the reader sees the same fixed refusal. That makes the model's judgement one more route to a refusal. It never turns a refusal into an answer.
How to choose a RAG similarity threshold
There is no portable number. Cosine similarity depends on the embedding model, the language mix and the corpus, so a floor that suits one embedder can refuse everything, or nothing, on another.
The design default on this site is 0.35, and the test suite runs a surrogate embedder, a stand-in model used only in tests, at that value. It has not yet been calibrated against the production embedder. A retrieval-only command sends the site's golden and suggested questions through the deployed embedder without calling the model, and the production floor changes only when that run passes. It is not a value to copy. To choose yours:
Run questions the assistant must answer and questions it must refuse through the embedder you deploy, and compare the two distributions of best similarity.
Decide how many answerable questions you will give up to stop the must-refuse ones. Where a confident wrong answer is worse than no answer, set the floor higher.
Record which embedder and corpus the number belongs to. Every page added to the corpus changes what an off-topic question can match, this article included.
The RAG vs raw demo is built around this boundary. It has no ungrounded route: a question that is unrelated to me, and that no passage supports, stops before the language model is reached.
Layer 3: the system prompt is the only layer a model can ignore
The prompt still does real work. It tells the model to use only the numbered sources supplied with the question, write about me in the third person in the reader's language, never state availability, rates or timelines, never invent or compute a figure or a date, say "currently" only of what a record marks as current, end every factual sentence with a source marker, and answer with a fixed token when the sources do not cover the question.
It cannot enforce any of it. OWASP's LLM07:2025 System Prompt Leakage entry recommends guardrails outside the model, because an independent system that inspects the output "is preferable to system prompt instructions." Anthropic's guidance on reducing hallucinations, which includes letting the model say it does not know, is equally candid: "while these techniques significantly reduce hallucinations, they don't eliminate them entirely."
So the prompt is written on the assumption that it will sometimes be ignored. Its job is to make compliance the easiest path, and every high-risk instruction has a deterministic counterpart:
Prompt instruction | Deterministic counterpart |
|---|---|
A source marker on every factual sentence | The gate checks the marker is present and in range |
Never invent a number | Every number, URL and email must appear in a source the sentence cites |
Write in the third person | The gate's first-person check |
No fees or availability | Layer 1 patterns, then the gate's value and first-person checks |
One instruction has no full counterpart: "use only what the sources say", for a claim with no number in it. The gate can require a valid marker on that sentence. It cannot decide whether the source says it. Nor do the rules on time. A sentence that puts two dated events in the wrong order, or calls an old role recent, carries no value for the gate to check. Records reduce that risk by arriving in date order with the current entries marked; they do not remove it.
Layer 4: gate the stream one sentence at a time
You cannot un-say a sentence the reader has already read. NVIDIA's NeMo Guardrails documentation names the trade-off: with stream_first on, the default, "if a rail blocks the content, the user has already received the tokens." This site takes the other side. Text is held until a sentence ends, checked, and only then released, at the cost of about one sentence of delay.
Each sentence faces four checks, each a fact about strings rather than a judgement about meaning:
Marker in range: it points at a source that was supplied.
Marker present on every substantive sentence; a short connective one with no values passes. Length is counted in words separated by spaces, except in Chinese, which does not use them: there every three characters count as two words.
Values in the cited evidence: every number, URL and email address appears in a source the sentence cites, with Persian and Arabic digits folded to Latin.
No first person, which would mean the assistant has started speaking as me.
On a violation, released sentences stay, the failing one is never sent, and the stream stops, which also cancels the upstream generation. The reader sees fixed text saying the rest of the answer could not be supported by the sources on this site and has been withheld. Markers are checked, then removed, so readers see prose, not a bibliography, and a truncated answer gets a different notice: an interrupted sentence is not an unsupported one.
Supported is not true. Values are matched against the sources a sentence cites, not against the claim: a sentence that cites two sources can take a value from either, and a sentence with no values passes with any in-range marker. The gate cannot show that a claim is correct, or even that the cited passage makes it. OWASP's LLM09:2025 Misinformation entry asks teams to "automatically validate key outputs"; this is a narrow, deterministic version. The mechanics are in streaming LLM guardrails.
Layer 5: voice and representation rules
An assistant that speaks for someone makes representations
In Moffatt v. Air Canada (2024 BCCRT 149), reported by McCarthy Tétrault and Bristows, British Columbia's Civil Resolution Tribunal dealt with an airline chatbot that suggested a customer could claim a bereavement fare retroactively. The airline's position, as the tribunal put it, suggested "the chatbot is a separate legal entity that is responsible for its own actions". The tribunal disagreed: "It makes no difference whether the information comes from a static page or a chatbot."
This is engineering context, not legal advice, and one tribunal decision is not a general rule. The engineering lesson is narrower: an assistant on your site speaks with your site's authority. A rate or a start date written in my voice would be a representation I never made. A confident wrong answer is worse than no answer; a confident wrong commitment is worse still.
Third person, fixed refusal copy and a first-person check
The assistant is never me. It writes about me in the third person, and every refusal and notice is fixed copy. The voice rule is enforced in three places:
Layer 1 declines questions about fees, availability, hiring and private details before retrieval, in each of the site's languages.
Layer 3 forbids the first person and any commitment on my behalf.
Layer 4 stops the stream when a first-person word appears.
The check is weaker in Persian. The English first-person list covers the pronouns. The Persian list is curated, not exhaustive: Persian marks person in verb endings, and a rule broad enough to catch them all would also catch ordinary words that end the same way. The list holds the pronouns and a few high-risk verbs, such as هستم and دارم, each in both the forms the fold can produce: the check folds before it splits words, so میکنم reaches it as two tokens and میکنم as one. What remains uncovered is the suffix, not the spelling. Persian leans more on the prompt, and that is written down rather than hidden. For the site's other languages the list holds the first-person pronouns and possessives, and for Spanish, Portuguese, Russian, Arabic and Hindi, which usually say "I" with the verb alone, a few first-person verbs as well. In Chinese, which does not separate words with spaces, the check looks for 我 anywhere in the sentence except in a compound of 自我, "self-", such as 自我修正, "self-correction". In Arabic one first-person word is left out on purpose, because the letter folding that makes Persian and Arabic spellings comparable makes it identical to a common preposition.
Cost abuse and denial of wallet: the same failure family
A hallucination is output nobody authorised. An unbounded bill is spend nobody authorised. OWASP files the second under LLM10:2025 Unbounded Consumption, which includes Denial of Wallet: attackers "exploit the cost-per-use model of cloud-based AI services, leading to unsustainable financial burdens on the provider". Its mitigations include input size limits, rate limits and user quotas, timeouts and throttling, and graceful degradation.
A pre-flight budget, not a post-flight alert
A check after the model call has already spent the money. On this site a monthly spend check runs after the floor and before the model call. It adds spend so far to an upper bound on this turn's cost: the whole prompt counted at one token per UTF-8 byte, plus the full output cap, reasoning allowance included. Checking against the expected cost instead is how a cap gets overshot.
If the total would pass the cap, no model is called, and the reader gets a fixed notice that the monthly limit has been reached, not an error.
The byte count is a ceiling, not a generous guess. Every vocabulary this endpoint can be pointed at is a byte-level BPE, and in one of those a token is a non-empty sequence of bytes, so no text can produce more tokens than it has bytes, in any language. The only way past the bound is a tokenizer that expands the text before it splits it, and none of these does. The bound replaced a flat two-characters-per-token rule, the rule this site retired: the Persian token tax measured that rule coming in below the true count on every Persian sentence of a parallel corpus on cl100k_base. The ceiling is loose — for Persian it reserves several times what the prompt really costs — and it is loose in the one direction a spend cap is allowed to be wrong.
Two limits remain. The check is not a reservation: spend is read and compared, not held, so requests arriving together can each pass and overshoot the cap; the per-IP limits narrow that window without closing it. And the counter is not a full ledger: it holds the cost of this endpoint's model calls, so the hosted embedding behind every question sits outside it, and a write lost while the counter store is unreachable is never retried. The cap is a guardrail, not an invoice.
Fail closed on spend, fail open on rate limits
If the counter store is unreachable, spend is unknown, and unknown spend against a hard cap has one safe reading: refuse as if the cap were reached.
The rate limiter fails open instead, because a cache outage should not take the site down. The worst case for a failed-open rate limit is some extra requests; for a failed-open spend cap it is an unbounded bill. There are three per-IP limits, refilling over a minute, an hour and a day, keyed on the address the edge proxy observed rather than a header the client can type. They slow scripts down; the monthly cap is the backstop on spend, not a guarantee.
Stop paying when the reader leaves
A closed browser tab does not stop an upstream model request. The server checks for a disconnect inside the streaming loop and closes the provider stream, which cancels the request. The gate's stop does the same: nobody should pay for sentences nobody will see.
What was generated before the break was still generated, and is still on the invoice, so the month is charged for it. Both provider protocols report usage in a final event that a stream cut short never reaches, so the service keeps every character it received and fills only the gaps the provider left, at one token per UTF-8 byte. Where the provider did report, its own figures win: a turn that runs to completion records exactly what it said, not an approximation of it.
How to measure it: refusal accuracy, recall@k and groundedness
Three properties matter. Refusal accuracy measures whether the assistant declines what it must. Recall@k measures whether the evidence reaches the model. Groundedness measures whether the answer stays within that evidence. On this site a golden set of questions in all ten of the site's languages, most of them English and Persian, is scored deterministically on every CI run, with no model call.
Refusal accuracy: two sets, no partial credit
Refusal is scored on two sets, because the two refusal layers fail differently. Pattern refusals must be caught by layer 1 before retrieval runs; no-evidence questions, which have nothing to do with the site, must fall below the floor and must not be recognised as questions about me. A third set checks that questions about the work as a whole reach the site's records.
Both sets must score perfectly for the build to pass. One missed refusal is an invented commitment on a real person's site, and an average would hide it. The same run checks the opposite failure: answerable questions refused by the patterns or the floor. Hamel Husain and Shreya Shankar's evals FAQ prefers pass/fail to rating scales because "binary evaluations force clearer thinking and more consistent labeling."
I do not quote the must-refuse questions: this article will enter the assistant's corpus, and quoting them would give those exact questions a strong keyword match on this page.
Recall@k below the default k
In the golden set, recall@k asks whether at least one passage that answers the question is in the top k, which makes it a hit rate. Anthropic's Contextual Retrieval write-up uses the stricter share of relevant documents retrieved, reporting "1 minus recall@20" as its failure measure.
The golden set measures recall at k = 5, below the default top-k of 8, because on a test corpus of a few dozen chunks the top eight is a large share of everything there is, and measuring there would flatter the retriever.
Groundedness: enforced per sentence, not scored offline
Offline, groundedness is usually scored as faithfulness. Ragas defines it as the claims in a response that the retrieved context supports, divided by all its claims, from 0 to 1; in its standard form, an LLM identifies and checks the claims.
LLM judges are useful and biased. Zheng and colleagues found "position, verbosity, and self-enhancement biases", while strong judges still reached over 80% agreement with human preferences. Husain and Shankar advise validating a judge's true positive and true negative rates on a held-out labelled set, and starting with cheap code-based checks wherever a rule can decide.
This site takes the code-based route: the gate enforces groundedness at runtime, and no faithfulness metric or LLM judge runs in CI.
What my golden set does not prove
The embedder is a surrogate. Retrieval in CI uses a bag-of-tokens hashing model with no learned meaning. Routing, fusion, ranking, top-k selection and the evidence decision, floor comparison included, are the production code; only candidate scoring, which production does in the database, is mirrored in the test. So CI catches regressions in the routing and ranking machinery, not in the database query or the live assistant's recall, and the surrogate even exercises a sparse list that production does not use.
The sets are small. There are twenty-one pattern refusals and eleven no-evidence questions. "Must be perfect" on eleven questions is a regression guard, not a refusal rate.
The live checks are commands, not CI jobs. The test that would send the golden set to the deployed retriever still skips. A retrieval-only command runs the golden and suggested questions through the deployed embedder without calling the model, and a script, run by hand after each import, puts the must-refuse questions to the live assistant.
Record answers are checked only by the gate. CI checks which questions reach the site's records and which records they get, against a fixture, with a scripted stand-in for the model. How a real model words an answer from them is checked at runtime, by the gate.
So I publish no accuracy, refusal or recall figure for the assistant: the automated run uses a surrogate embedder, the live checks are small and run by hand, and this site does not estimate numbers it has not measured.
Structure over trust in a contract assistant
I worked as an AI engineer on Kartez Contract, a trilingual contract workspace that the client halted one month from completion for war-related funding reasons. There, keeping the model anchored was handled by structure rather than trust: answers draw only on the selected clauses, fixes must quote the existing wording exactly, outputs follow fixed schemas, and the model runs with low randomness when drafting. These are design constraints, not a measured guarantee. The floor and gate above belong to this site's assistant, not to Kartez. The Kartez Contract case study covers the project. What those patterns look like in contract AI, and why the evaluation set comes first, is in legal AI hallucinations: what to build.
Checklist: guardrails for a production RAG assistant
Refuse by pattern first. Decline fees, availability, private data and instruction extraction before retrieval.
Threshold a similarity, not a rank score. Cosine can carry a floor, and so can whole-word coverage once common words are removed and at least two content words match; an RRF score cannot. Below the floor, refuse with fixed copy and skip the model.
Answer overview questions from records, not similarity. A list sorted by date has no nearest neighbour.
Calibrate the floor on the embedder you deploy. A floor checked on a surrogate is a default.
Mirror every prompt rule with a check outside the model.
Gate the stream by sentence. Require a marker, check numbers, URLs and emails verbatim, withhold everything from the failing sentence on, and give truncation its own notice.
Ban the first person in assistants that speak about a person or firm, and document where the check is weaker.
Reserve an upper bound before the call, not a generous guess. For a byte-level BPE, the prompt's UTF-8 byte count is one. Make the reservation atomic so concurrent requests cannot all pass, fail closed, and cancel upstream work on disconnect.
Record a cost for every call, including ones cut short: keep the text as it streams, and fill only what the provider's final usage event never reported.
Measure refusal separately from recall. State the embedder, the set sizes and what the run does not prove.
FAQ
Does RAG eliminate hallucinations?
No. RAG-based legal research tools from LexisNexis and Thomson Reuters still hallucinated between 17% and 33% of the time in a preregistered study, and insufficient context can make a model more confident rather than more careful. Retrieval improves the odds; guarantees come from checks outside the model.
What similarity threshold should a RAG system use?
There is no portable number, because similarity distributions depend on the embedding model and the corpus. Threshold a bounded similarity such as cosine, never a reciprocal rank fusion score, and set the value from answerable and must-refuse questions run on the embedder you deploy. A floor checked only on a surrogate is a default, not a calibration.
Is a system prompt enough to stop hallucinations?
No. OWASP recommends avoiding "using system prompts to control the model behavior where possible" and relying "on systems outside of the LLM" instead, and Anthropic's guidance says prompting techniques reduce hallucinations without eliminating them. Keep the prompt, and give every instruction that matters a deterministic check outside the model.
Can an LLM judge replace deterministic checks?
Not for rules a string check can decide, such as whether a marker points at a real source or a number appears in the evidence. Judges show position, verbosity and self-enhancement biases. Use them for questions code cannot answer, with binary labels and a judge validated against human labels.
How do you stop an LLM chatbot running up a bill?
Before the model call, reserve an upper bound on its cost atomically — for a byte-level BPE tokenizer, the prompt's UTF-8 byte count is one, in any language — and refuse if the counter is unavailable. Record a cost for every call, including ones cut short, by keeping the streamed text and filling what the provider's final usage event never reported. Degrade to fixed copy instead of erroring, and cancel upstream generation when the client disconnects. OWASP lists this risk as LLM10:2025 Unbounded Consumption, which includes Denial of Wallet.
Sources
If you are designing an assistant that has to refuse as reliably as it answers, scope your project.