Why one AI can't be trusted to check your documents

No, a single AI model can't be trusted as the final check on a document that matters. The same model that drafts a clause has no independent reason to doubt it, and a growing body of research shows that models are bad at catching their own mistakes. Dependable assurance comes from independence: more than one reviewer, and a human who makes the final call. This piece explains why, with the case law and the research to back it up.

The $5,000 mistake that turned into a daily occurrence

In 2023, two New York lawyers filed a brief in Mata v. Avianca that cited six court decisions. None of them existed. A chatbot had invented the cases, complete with quotations and citations in correct Bluebook format, and the lawyers had filed them without checking. The court sanctioned them $5,000 and issued a public reprimand. At the time, the legal world treated it as a freak event, the kind of thing a careful professional would never do.

That assumption aged badly. Damien Charlotin, a researcher at HEC Paris, now maintains a public database of court decisions involving AI-fabricated citations. It held fewer than a hundred entries in the spring of 2025. By mid-2026 it had passed 1,600, and roughly nine in ten of those decisions were written in a single year. Bloomberg Law quoted a Stanford policy fellow describing the problem as metastasizing. Charlotin himself told reporters he was getting tired of keeping up with it, because new cases were arriving several times a day.

The sanctions climbed with the volume:

Case	When	Penalty
Mata v. Avianca (S.D.N.Y.)	2023	$5,000 + public reprimand
Lacey v. State Farm (C.D. Cal.)	2025	~$31,000 in fees against two firms
Whiting v. City of Athens (6th Cir.)	2026	$15,000 per attorney
Sullivan & Cromwell (bankruptcy filing)	2026	Formal apology; ~28 bad citations

In the California case, the special master wrote that the AI had "affirmatively misled" him. Big firm, small firm, government lawyer: the pattern did not discriminate.

The hallucination was never the real failure. The failure was reliance, a person treating a single model's confident output as if it had already been checked.

Every sanction in that database traces back to the same missing step.

---

Why a model can't grade its own homework

The intuitive fix is to ask the model to check itself. Have it draft, then have it review. It sounds reasonable, and it mostly does not work. The research points in one direction:

• Self-review makes things worse. A widely cited 2023 paper, "Large Language Models Cannot Self-Correct Reasoning Yet," found that when a model reviewed its own answers without outside information, accuracy went down. It flipped correct answers to wrong ones more often than it fixed real errors.
• There's no reliable internal fix. A 2024 survey of the self-correction literature reached a blunt conclusion: no convincing evidence that a model can reliably correct its own reasoning using only its own feedback.
• The blind spot is specific. Researchers documented a "self-correction blind spot," where a model misses an error in its own output but catches the very same error when it's presented as someone else's writing. The wrongness was detectable. The model just couldn't see it in its own work.
• Models lean toward agreement. This behavior, called sycophancy, gets worse when the content looks self-generated. One benchmark found models will endorse and try to "prove" false statements that resemble their own prior output.

A model reviewing its own draft is not a skeptic. It is the most motivated reasoner in the room.

This is exactly the principle that keeps an auditor separate from the books they examine. The person who wrote the ledger is the worst-positioned person to find the error in it.

Independence is not a nice-to-have in review work. It is the whole point of review work.

---

"It looked right" is the entire problem

The danger with AI output isn't that it looks obviously broken. It's that it looks finished. A fabricated case citation arrives in perfect format. An invented statistic arrives with two decimal places. A clause that contradicts the one three pages earlier reads just as smoothly as a correct one. Fluency is what the technology is built to produce, and fluency is precisely what a tired human reviewer mistakes for correctness on a final read.

This holds even for tools built specifically for the job. A Stanford study of AI legal research products found that purpose-built tools, the ones marketed as grounded in real law, still produced unsupported or incorrect statements somewhere between roughly one in six and one in three of the time, depending on the product. General-purpose chatbots were far worse. The researchers' practical takeaway was uncomfortable: if you have to verify every proposition and every citation anyway, the efficiency the tool promised largely evaporates.

Courts have started sorting the errors into recognizable types:

1. Citations to cases that do not exist. Pure invention, dressed in correct format.
2. Real-looking citations attached to real cases that say nothing of the kind. The case is real; the proposition it supposedly supports is not.
3. Accurate quotations used to support a proposition the case actually contradicts. The quiet killer, because nothing about it looks fake.

That third category is the dangerous one. Every individual piece checks out. Only the conclusion is wrong.

---

One reviewer is one point of failure, human or machine

None of this is unique to AI. Ask any editor why writers can't reliably proofread their own work. You know what the sentence is supposed to say, so that's what you read, even when the words on the page say something else. A single reviewer, of any kind, is a single point of failure. Add a second independent reader and the math changes, because the errors one misses are rarely the same errors the other misses.

The legal sanctions data makes this concrete in a way that's easy to overlook. Roughly nine in ten documented incidents involve solo practitioners or small firms. That isn't because large firms avoid AI. They use it heavily. It's because large firms built a checking layer around it: review workflows, verification steps, someone whose job is to confirm every citation before anything is filed. They institutionalized the second set of eyes. The firms getting sanctioned are the ones that let a single confident output go straight out the door.

So the real question for anyone relying on AI is not "which model is most accurate?" It's "what catches the model when it's wrong?" A more accurate model lowers the error rate. It does not change the structure of the risk, because the structure of the risk is that one reviewer signed off alone.

---

This was never only a legal problem

Legal filings get the headlines because judges write sanctions orders and those orders are public. But the same failure lives in every document that becomes binding the moment someone relies on it.

• Financial documents. The error is usually a number. A figure stated one way in the summary and another way in the schedule. A total that doesn't sum. A percentage that drifts between sections. None of it announces itself, and all of it can move money.
• Contracts and agreements. The error is usually a relationship between parts of the document. A term defined on page two and used inconsistently on page nine. A cross-reference pointing to the wrong exhibit. A party label that's correct in the preamble and wrong in the signature block. An obligation that no longer matches what the deal intends. The contradiction only exists across the whole document, which is exactly the view a single fast read tends to miss.
• Proposals, reports, and client-facing work. The error is the one that embarrasses you later: the stale figure copied from last quarter, the claim that doesn't survive a careful reading, the inconsistency a counterparty notices before you do.

One missed clause can cost a great deal more than the time it would have taken to catch it.

---

What actually earns trust

If a single model can't be trusted to check itself, and a single reviewer of any kind is a single point of failure, then trust has to be built structurally rather than bought from a better model. Four things do the work.

1. Independence. The thing that checks the document should have no hand in writing it and no stake in approving it. That's the auditor-and-books principle, applied to documents.

2. More than one reviewer. Independent reviewers built on different underlying systems fail in different ways, so the errors that slip past one tend to get caught by another. Agreement among independent reviewers is a meaningful signal. A single confident voice is not.

3. A human with the final call. The reviewers surface what to look at, ranked, with the reasoning attached. A person decides what's real, what matters, and what changes. The research community studying self-correction has largely concluded the same thing: the path to reliable AI output runs through external verification, not through teaching models to introspect better. The fix lives outside the model.

4. A record you can show. Not a record that claims the document is correct, because no software can promise that. A record that proves an independent check happened and captures what the reviewers flagged, so the people downstream can see the work and reconcile it.

A proof of integrity confirms that a review occurred and hasn't been altered since. It does not certify that every edit is right. Correctness comes from human judgment. The proof comes from the process.

Keeping those two claims separate is not a technicality. It's the difference between a tool you can trust and a tool that's quietly lying to you about what it guarantees.

---

What to do before you sign, send, or file

The professional duty here is already settling into place. The American Bar Association's formal guidance on generative AI lands on a simple obligation: verify the output before you rely on it. That duty doesn't care whether the draft came from a junior associate, a research tool, or a chatbot. The signature is yours.

In practice, that means three things:

• Treat any AI-generated document as a draft to be checked, never as a finished one.
• Build or borrow a second set of eyes — the checking layer large firms built in-house and small firms too often skipped.
• Be honest about where a model's confidence ends and real verification begins. The gap between those two is exactly where the sanctions, the lost money, and the embarrassment live.

The technology will keep getting better at sounding right. That's the one thing it has always been good at, and it's the precise reason a final, independent check matters more as the drafts get more convincing, not less.

---

FAQ

Can you trust AI to review a legal document? Not as the final word. A single model has no independent reason to doubt its own output, and studies show even purpose-built legal AI tools produce incorrect or unsupported statements a meaningful share of the time. Use AI output as a draft, then verify it independently before relying on it.

Why can't an AI just check its own work? Research on self-correction has repeatedly found that models asked to review their own answers, without outside information, often make accuracy worse rather than better. There's even a documented "blind spot" where a model misses an error in its own output but catches the identical error when it's presented as someone else's. Models also tend to agree with content that looks self-generated.

What is an AI hallucination in a document? It's confident, fluent output that is false: an invented citation, a fabricated figure, or an accurate-looking quotation used to support a claim it doesn't actually support. The problem isn't that it looks broken. It's that it looks finished.

How common are AI hallucinations in real filings? A public database maintained by HEC Paris researcher Damien Charlotin grew from fewer than 100 documented court cases in spring 2025 to more than 1,600 by mid-2026, with new cases arriving several times a day. Sanctions have reached tens of thousands of dollars per attorney.

What makes document review trustworthy if one AI isn't enough? Four things: independence from whoever wrote the document, more than one independent reviewer, a human who makes the final decision, and a record that proves the check happened. Trust is built into the process, not bought from a single more-accurate model.

Does a proof of review mean my document is correct? No. A proof of integrity confirms that an independent review took place and hasn't been altered since. Correctness still depends on a human reviewing and accepting the findings. Those are two separate claims, and any tool that blurs them is overpromising.

---

Recensa runs an independent review pass on your finished Word or PDF document, gives you a ranked Issue Ledger with the reasoning behind each finding, and produces a Proof Report you can hand to your team. It had no part in writing your document, so it has no stake in approving it. You decide what ships. See what it catches.