Legal AI hallucination cases pass 2,000: what to build

Court decisions on hallucinated AI content have passed 2,000. Why RAG alone did not fix it, and design patterns from my work on contract AI.

As of 14 September 2026, Damien Charlotin's AI Hallucination Cases database lists 2,041 court and tribunal decisions on hallucinated AI content. In most, a court found or implied that a party relied on such content; as an exception, the database also includes some where AI use was alleged but not confirmed. Retrieval alone is not the fix: three leading legal research tools, tested in a 2024 Stanford-led study, each hallucinated 17–33% of the time. What to build is structure that makes output checkable, a person who decides, and an evaluation set built first.

I was an AI engineer on Kartez Contract; this is what I would keep from that work and what I would do first.

What the AI hallucination cases database tells builders, and what it does not

Charlotin's count has passed 2,000. Three things matter before anyone quotes it in a pitch or a risk memo.

It counts decisions, not errors. The database "does not track the (necessarily wider) universe of all fake citations or use of AI in court filings". It is a record of failures that reached a judge, not a hallucination rate.

It does not blame tools. Its own note says a tool's mention "does not necessarily mean that tool was responsible for the hallucinations in question", so nothing here ties a decision in the database to a tool. The tools named below come from the Stanford study, not from the database.

The categories matter more than the total. In Charlotin's database, 1,690 decisions are tagged Fabricated, 854 Misrepresented and 548 False Quotes, and one decision can carry several tags. A check that a cited source exists can catch most of the first kind, the invented authorities. It lets the other two through whenever the source is real but the quotation is invented or the reading of it is wrong. A contract tool can make both mistakes while pointing at a clause that exists.

Charlotin also lists 1,174 decisions where the party using AI was a pro se litigant (someone representing themselves) and 811 where it was a lawyer, so a legal AI tool can assume neither a lawyer in the loop nor that one will catch its errors. Contract, with 449 decisions, is the largest legal field in his database. Those are contract disputes in which hallucinated AI content was at issue, not contracts drafted by AI.

No. The Stanford study below concludes that legal retrieval-augmented generation (RAG) "can reduce hallucinations compared to general-purpose AI systems (here, GPT-4), but hallucinations remain substantial, wide-ranging, and potentially insidious." Insidious is the right word: a misgrounded answer arrives with a real source attached. The general guardrail design is in RAG hallucination guardrails.

The Stanford-led study: 17–33%

In a paper posted to arXiv on 30 May 2024, researchers at Stanford and Yale reported "the first preregistered empirical evaluation of AI-driven legal research tools". Preregistered means the study and its dataset of over 200 legal queries were registered before the evaluation was run, a guard against choosing queries to fit the results. Some providers had marketed retrieval as "eliminating" or "avoid[ing]" hallucinations. The study found that Lexis+ AI, Westlaw AI-Assisted Research and Ask Practical Law AI each hallucinated 17–33% of the time, and that "the providers' claims are overstated".

The definition matters more than the range. A response counts as hallucinated "if it is either incorrect or misgrounded", and misgrounded means key propositions are "cited but misinterpret the source or reference an inapplicable source". The tools were tested in 2024, on legal research rather than contracts, so the range is not a figure for legal AI today. The category carries over: in a contract, a misgrounded answer sounds right and points at the wrong clause.

Google's sufficient-context finding: more context, fewer abstentions

The "Sufficient Context" paper separates errors where the model misuses good context from errors where the context was never enough to answer. The larger models it tested, from Google, OpenAI and Anthropic, "often output incorrect answers instead of abstaining when the context is not" sufficient.

Google Research's write-up of the work (14 May 2025) says that while RAG "generally improves overall performance, it paradoxically reduces the model's ability to abstain from answering when appropriate". In Google's example, Gemma gave incorrect answers on 10.2% of questions with no context and on 66.1% with insufficient context.

Retrieval changes what the model sees, not whether it knows it has enough. A question about termination may turn on a definition or a notice clause that was never retrieved. The design has to make "I need another section" a normal output, and the evaluation has to test for it.

AI contract review: the failures that matter

An invented clause becomes a liability for whoever signs, so a model that makes one up is worse than no model at all. These are the failures I would design against first. No pattern below proves an answer right; most make a wrong answer catchable.

Failure

In a contract

Pattern that makes it checkable

Invented content

A clause or deadline the document does not contain

Answers from selected clauses; the user accepts every edit

Misquoted wording

A fix that edits text that is not there

Every fix quotes the current wording exactly

Misgrounded answer

An answer from the wrong clause, or from a section never shown

Clause selection; an instruction to ask for missing sections

Broken output

A cut-off risk list shown as if complete

Fixed schemas, never shown raw; a completeness check on repaired replies (recommended)

Partial edits

A party renamed in one section only

Whole-contract edits planned, applied everywhere, highlighted

Changed values

An amount, date or name altered in a rewrite

A verbatim value check (recommended below)

Contract AI design patterns from Kartez Contract

From November 2025 to July 2026 I worked as an AI engineer on Kartez Contract, a trilingual, right-to-left contract workspace with a drafting wizard, a review journey and a contract studio. I rebuilt and extended the AI engine behind those workflows, starting from the client's first basic AI chains, and built the copilot that classifies each request, finds the relevant clauses and plans edits across the whole contract.

The hardest problems were retrieval and anchoring, and anchoring was handled by structure rather than trust. None of the six patterns below is a measured guarantee; the first four are the anchoring constraints themselves.

Answers only from the clauses the copilot selected

The user's contract was split along its own structure, including the articles, clauses and notes (ماده، بند، تبصره) that Persian contracts number in their own way. Only the sections relevant to a question were used to answer it, and the copilot was instructed to say when it needed sections it had not been shown.

A clause is a unit a lawyer already works in, so a retrieval set made of clauses is one an evaluator can label and check. The instruction offers an alternative to guessing, but it is not an enforced gate, so the evaluation set needs questions the selected clauses cannot answer. Persian retrieval has its own traps, covered for a different system in hybrid search for Persian RAG.

Fixes that quote the contract exactly

Each suggested fix was required to quote the current wording word for word, so it could be found and replaced, and it came with ready-to-use replacement text. That requirement is what makes exact-quote validation possible: a program can confirm the quoted text exists before the fix is applied.

Whether a rewrite is wise is a legal judgement. Whether the sentence it replaces exists is a string match. OWASP's guidance on LLM misinformation makes the general case: "automatically validate key outputs, especially output from high-stakes environments". Building it again, I would drop any fix whose quote is not found, never apply it approximately, and count the drops as an early warning.

These fixes sit in the review journey, which had not been fully tested when work stopped, so the pattern stands on its logic, not on a measurement.

Fixed output schemas, never shown raw

Every analysis returned a fixed structure, so timelines, risk registers and fixes were laid out the same way each time. A badly formed response went through a multi-stage parse-and-repair step, including repair of cut-off replies, and was retried with more room when needed. It was never shown to the user raw.

This targets a quiet failure: a truncated risk list looks complete to a reader. Repair alone does not solve it, because a cut-off list that is closed off still parses, so I would retry or mark a repaired truncation rather than display it as whole. A schema also gives the other checks a known place to look.

Low randomness when drafting

When drafting, the model ran with low randomness. That buys repeatability, not correctness: the same input tends to give the same draft, so failures can be reproduced and evaluation runs compared.

Guided workflows, not one chat box for every job

Drafting, reviewing and editing are different jobs, and each needs its own guided workflow, not one chat window. The review, for example, is written from the user's side, so its first step asks for the user's role in the deal.

Anthropic's "Building effective agents" uses "workflow" in the orchestration sense: "systems where LLMs and tools are orchestrated through predefined code paths". It says they "offer predictability and consistency for well-defined tasks". Each fixed step has a known input and output shape that can be checked. A blank chat box accepts anything and returns anything.

Highlighted edits the user accepts or rejects

The rule across all three workflows was that the AI proposes and the user decides. Copilot edits were highlighted in the editor for the user to accept or reject. A whole-contract instruction, such as renaming a party, was planned first, applied to every affected section and highlighted.

The user judges marked changes instead of rereading the document, and a rename is highlighted everywhere it landed, not found later as an inconsistency.

A different system: the gates on this site's assistant

Nothing described in this section was part of Kartez Contract. The assistant on this site, which I built and maintain, answers only from the site's own content, behind deterministic gates.

Some questions are declined by pattern before anything is retrieved: rates and availability, private details, speculation, attempts on the system prompt, and requests to write, translate or solve something rather than ask about the work. If a question is not about me and nothing retrieved clears a retrieval-score floor, the assistant refuses and the model is never called. Questions about me are also recognised by pattern, from cues such as my name or a third-person pronoun, and they are not refused just because no passage clears the floor. Instead, the model is given the site's own records, entries from its CV, case studies, articles and services, and told to reply with a fixed token if they do not cover the question; the reader then sees the same refusal. While an answer streams, a sentence gate checks each complete sentence before it is shown:

  • Citation range. Every citation marker points to a source the model was given.

  • Citation required. Every substantive sentence carries a marker.

  • Value support. Every number, URL and email address is present in a source the sentence cites.

  • Voice. No sentence uses first-person pronouns such as I, me or we.

One failed sentence stops the answer, and that sentence is never shown. Citation markers are checked, then removed, so readers never see them.

The mechanism is in streaming LLM guardrails. The RAG vs raw demo streams a real grounded answer next to a raw column that stays empty by design; ask something that is not about me and that the site does not cover, and the run stops before any model is reached. If I were building a contract tool, I would add the same kind of check for amounts, dates and party names, with each value required to match the source word for word.

The usual order is to build, then measure. For legal AI I would reverse it.

  1. A design constraint is a hypothesis. "Fixes must quote the contract exactly" is a rule. How often the model breaks it is a number, and only an evaluation set produces it.

  2. Every change needs a regression check. Without a fixed set, each model, prompt or retrieval change is judged by feel.

  3. The set outlives the project. A labelled set and its last run are evidence someone else can use. A list of design principles is not.

What happened on Kartez Contract

The client halted the engagement one month before completion, when war-related financial difficulties stopped the funding. It was a client funding failure, not a delivery failure. By then the drafting wizard, the contract studio with its copilot and the trilingual interface were built and wired end to end. The review journey was built across all five steps, but it had not been fully tested and was still being refined when work stopped.

No accuracy figure exists, because the engagement stopped before a formal evaluation set was built. The design constraints were real; the number that would have turned them into a claim was never produced.

A first-week evaluation set for contract AI

This is advice, not a description of anything Kartez Contract had. Each item is a binary pass or fail.

  • Answerable questions, labelled with their clause. The answer must be right and draw on that clause.

  • Questions the contract cannot answer. Expect "I need section X" or "the contract does not say", not an answer.

  • Edit requests. Every quoted span in a proposed fix must string-match the source document.

  • Whole-contract edits. A party rename must change every occurrence and nothing else.

  • Values that must not move. Amounts, dates, notice periods and party names, compared verbatim before and after each rewrite.

  • Broken outputs. Truncated and malformed replies must never reach the screen raw, and a repaired truncation must never be shown as complete.

  • Persian structure. Numbering by ماده, بند and تبصره, mixed-direction lines and Persian digits.

  • Both sides of the deal. The same contract reviewed from each party's position.

EU AI Act deadlines after the Digital Omnibus: an engineering view

  • Parliament adopted the final text on 16 June 2026 and the Council on 29 June 2026.

  • Marking of AI-generated content is due by 2 December 2026 for systems placed on the market before 2 August 2026.

  • Other Article 50 transparency duties, such as telling users they are dealing with an AI system, apply from 2 August 2026.

  • Stand-alone Annex III high-risk obligations move to 2 December 2027. For AI embedded in products covered by Annex I safety legislation, it is 2 August 2028.

Scope is a question for counsel. None of the patterns above is a compliance measure.

  • What is each answer grounded in? A clause or a section, not "the document".

  • Can every edit be located? Each change quotes the text it replaces, checked against the document.

  • Can it say the answer is not there? Ask for a demonstration.

  • Is output validated before display? Broken replies are repaired or withheld.

  • Does anything change without acceptance? Nothing should.

  • Is there one workflow per job? Drafting, review and editing should not share one chat box.

  • Are values checked verbatim? Amounts, dates and party names, against the source.

  • Can I see the evaluation set? What is in it, which items test abstention, and when it last ran.

If you are scoping a contract or legal AI system, Project Diagnostics turns five questions into a first-pass architecture.

Frequently asked questions

How many AI hallucination cases have reached the courts?

As of 14 September 2026, Damien Charlotin's AI Hallucination Cases database lists 2,041 court and tribunal decisions on hallucinated AI content. In most, a court found or implied that a party relied on such content; as an exception, some where AI use was alleged but not confirmed are included. It records decisions, not every fake citation or use of AI in court filings.

It reduces them but does not stop them. In a preregistered Stanford-led study, three retrieval-based legal research tools hallucinated less than GPT-4, yet each still hallucinated 17–33% of the time when tested in 2024. Google's sufficient-context research found that larger models often answer incorrectly instead of abstaining when the retrieved context is not enough.

Can AI review contracts reliably?

Not on trust. What can be built is a review that makes errors checkable: answers from selected clauses, fixes that quote the contract exactly, fixed output schemas and a person who accepts or rejects every change. Whether a tool actually catches errors is a measurement, so ask for its evaluation set.

What is exact-quote validation?

It is a check that the text a proposed fix quotes exists, word for word, in the current document before the fix is applied. It needs a rule that every fix quotes the wording it replaces, and it turns part of a legal edit into a string match.

Why is there no accuracy figure for Kartez Contract?

The client halted the engagement one month before completion for war-related funding reasons, before a formal evaluation set was built. This site does not estimate numbers it has not measured. That gap is why this article argues for building the evaluation set first.

Sources