Stepping Up to the Plate: AI Hallucinations and the Lawyer’s Ethical Duty to Verify

In our previous post in this series, we discussed lawyers’ duties in the age of generative AI through six common duty themes associated with the various proposed and issued rules and guidelines, along with examples of rules associated with each of those themes.

There’s a simple reason that state bars and courts continue to issue proposals for new or changed rules related to the use of generative AI: the cautionary tales associated with its use in the court system continue to make headlines. Just when you think you’ve seen it all, there is a new cautionary tale that makes you wonder “what were they thinking?” In this blog post, we’re going to review a few of those cautionary tales to discuss what we can learn from them. We’re going to discuss what hallucinations really are and how to manage them. And we’re going to discuss why the use of AI differs for general purpose AI models like ChatGPT vs. purpose-built models for eDiscovery.

Cautionary Tales with AI in Case Filings

It’s easy to assume at this point that everyone understands the pitfalls and risks associated with the use of generative AI. Based on trends in the courts, that’s far from the case. Here are examples of a few recent cautionary tales with AI in case filings:

• A personal injury firm was ordered to pay more than $45,000 in attorney fees to opposing counsel for filing a motion with faulty citations likely generated by AI and then continuing to enter filings with the same errors after being notified of the mistakes.
• In this Mississippi case where lawyers on both sides of a case cited fake hallucinated cases in their filings, pro hac vice admission in the case was revoked for the out-of-state lawyers and they both were barred from entering an appearance in any case before the U.S. District Court for the Northern District of Mississippi for two years. The local counsel attorneys were disqualified from the case. All counsel received fines.
• Here, the Court imposed “career-altering sanctions” on one attorney, including a public reprimand, disqualification from the case, referral to licensing authorities, and a six-month suspension from practice for “intentionally delet[ing] his ChatGPT account in order to destroy evidence of AI misuse”.
• After finding “over two dozen fake citations and misrepresentations of fact in” briefs filed by two attorneys for their client, those attorneys accused the Court of “engaging in a vast conspiracy to harass them”. They received numerous sanctions, including reimbursement of reasonable attorneys’ fees, paying double costs to appellees for costs incurred, paying $15,000 each to the registry of the court as punitive sanctions, and consideration of disciplinary proceedings.
• “[M]ultiple fabricated quotations” in the declaration and “manufactured citations to deposition transcripts” led to $4,000 in attorney fees and the order to attend a minimum 3-hour CLE course “on the topic of hallucinatory citations generated by AI in the legal field” in this case.

These are just a few examples. With so many cautionary tales out there, you would think that most legal professionals would understand by now the importance of verifying outputs from AI models in their case filings. Instead, the problem seems to be getting worse – a lot worse.

According to the AI Hallucinations Cases site created by Damien Charlotin (and reported by eDiscovery Today), there were 16 reported cases with hallucinations in 2023. That nearly quadrupled to 59 cases in 2024. In 2025, the number of cases grew fourteen-fold to 827. And, as of July 8 of this year, we had already exceeded last year’s total at 828.

Over half of hallucinations cases (1,018) as of that date were caused by pro se parties, so the issue isn’t just a lawyer ethics problem. Many pro se parties are relying on public large language models (LLMs) like ChatGPT to draft legal filings, not fully understanding their propensity to hallucinate.

What Hallucinations Really Are and How to Manage Them

These numbers illustrate two important facts: 1) more people than ever – including lawyers and pro se parties – are using AI models to assist with generating case filings, and 2) many of those people don’t realize these models can hallucinate regularly.

Other legal professionals are choosing to stay on the sidelines for now, waiting until AI companies “fix” hallucinations before they embrace the tools.

Neither approach is the correct way to deal with hallucinations. Hallucinations are not “bugs” in the software that need to be fixed, they are a byproduct of how the technology works. LLMs are fundamentally probability-based prediction systems trained on literally trillions of words from various sources to learn how language is structured and how ideas are commonly connected.

As a result, they are optimized to generate language (i.e., “generative” AI) that sounds natural and plausible and is correct a lot of the time. But the language they generate is not necessarily guaranteed to be true. Sometimes, the highest probability in the probabilistic system turns out to be a hallucination, i.e., incorrect information that the model states with confidence. This is because these models don’t actually “know” anything, they’re just good at predicting the right thing to say most of the time.

Even as the models improve over time, hallucinations are here to stay, and there will never be any guarantees of the total accuracy of the content being generated as long as they remain probabilistic. Does that mean lawyers and legal professionals should avoid using these solutions? Absolutely not, the efficiency gains of using this technology are tremendous. What it does mean is that, as is the case with any other technology, it’s important to check and verify the outputs for accuracy. No approach is perfect.  Remember, humans make mistakes too!

General Purpose AI vs. Purpose-Built AI

If probabilistic generative AI models hallucinate, how does that impact the use of GenAI technology in eDiscovery solutions? General-purpose public AI models (like ChatGPT and Claude) and purpose-built AI solutions for eDiscovery both rely on large language model technology, so neither is immune from hallucinations. However, the risk profile is significantly different for purpose-built eDiscovery solutions because they typically constrain how the model operates.

As discussed above, public AI models are designed to answer virtually any question by drawing on broad training data and probabilistic language generation. This increases the potential for hallucinations. In contrast, leading eDiscovery AI solutions are generally designed to work within a defined corpus of evidence, limiting the model’s ability to invent information outside the available documents.

Rather than relying solely on the model’s internal knowledge, these systems commonly use retrieval-augmented generation (RAG) or similar techniques to retrieve relevant documents first and then generate responses based on that evidence. Many platforms also provide citations or links back to the source documents, making it easier for legal professionals to verify the AI’s conclusions and support a more defensible workflow.

That said, “purpose-built” does not mean “hallucination-proof.” Hallucinations can still occur in eDiscovery AI solutions. However, purpose-built eDiscovery solutions (and the processes built around them) typically incorporate multiple safeguards that reduce both the likelihood and impact of hallucinations. These safeguards may include restricting responses to retrieved evidence, requiring source citations, limiting outputs to specific legal workflows, applying confidence thresholds, logging prompts and responses for auditability, and keeping humans firmly in the loop for review and validation.

That last part is key, as human review and validation is a necessary component of any approach that uses technology to assist with review. The “Staples-easy button” for eDiscovery review does not exist.

Conclusion

Lawyers and legal professionals are using, and in many cases misusing generative AI technology at an unprecedented rate. As the usage of GenAI goes up, so will the incidents involving hallucinations, until users develop a broader understanding of how hallucinations occur and the importance of checking and verifying outputs from these models.

We need to keep those same principles in mind with purpose-built eDiscovery solutions. Hallucinations can occur there as well, but the guardrails in place and best practices for applying the technology can enable us to use these systems with confidence.

Next time, we’ll discuss confidentiality risks when lawyers or staff upload privileged or sensitive information into public AI systems.

For more regarding Cimplifi specialized expertise regarding AI & machine learning, click here.