You cannot assess reliability from a demo. Demos are run on documents the vendor selected, in a category the vendor tuned, with a presenter who knows the answer. The only assessments that predict live performance are the ones you run on your own documents, where you already know what is in them — and the only vendor answers worth anything are the ones that survive a follow-up question.
Start from the base rate, not the pitch
Before evaluating any individual tool, calibrate on what the category actually achieves. Stanford researchers tested the AI legal research assistants sold by LexisNexis and Thomson Reuters and found they “each hallucinate between 17% and 33% of the time.”[1] The paper's conclusion was that providers' claims of hallucination-free output are overstated.
Those are retrieval-grounded products, built on proprietary curated databases, sold at enterprise prices by the two incumbents of legal information. If that is the measured performance of the best-resourced entrants in an adjacent task, then a startup's assurance that its system “doesn't hallucinate because we use RAG” is not a technical claim. It is a description of the architecture that was measured at 17–33%.
The reliability question that actually matters
Most buyers ask “how accurate is it?” This is the wrong question, and vendors love it, because accuracy is a single number computed on a test set of the vendor's choosing and it can be made to say almost anything.
The question that predicts whether a tool is safe to build a process on is different: when this system is wrong, how long does it take me to find out?
A tool that is wrong 10% of the time but shows you the exact source sentence behind every assertion is operationally safe — errors are caught in the seconds it takes to read the quote against the claim. A tool that is wrong 3% of the time and outputs fluent unattributed prose is operationally dangerous, because the only way to detect the 3% is to redo the work you bought the tool to avoid. The second tool has better accuracy and worse reliability, and it will win every bake-off scored on accuracy alone.
A test protocol you can run in a week
1. Build a seeded set from a closed deal
Take 40–60 documents from a transaction that has already closed, where your team knows every material issue. You are not testing whether the tool finds things. You are testing whether it finds the things you already know are there, and what else it claims.
2. Seed known negatives deliberately
Include several documents you are confident are clean. A tool that returns findings on genuinely unremarkable documents is telling you its false-positive rate, and false positives are not harmless — they consume the reviewer attention the tool was bought to conserve, and they train your team to skim output.
3. Score recall on the issues you seeded, not on total findings
Count how many of your known material issues appeared. This is the only number in the exercise that maps to the risk you actually care about. A tool returning 200 findings that misses two of your five known deal-relevant issues has failed, regardless of how comprehensive the output looks.
4. Trace every high-severity finding to source
For each one, open the cited document and read the cited text. You are looking specifically for misgrounded citations — a real clause cited for a proposition it does not support. These pass a click-the-link check and fail a read-the-clause check, and they are the most common failure that survives casual review.
5. Run the same document twice
Submit an identical document in two separate sessions and compare. Material divergence between runs tells you the output is not reproducible, which has a specific consequence: you cannot defend a decision you cannot reconstruct, and you cannot audit a process whose outputs move.
6. Test the long document
Most tools degrade at length, and most do so silently. Submit something genuinely large — a full credit agreement, a 10-K — and check whether findings from the final third appear at all. Silent truncation is common and rarely disclosed.
Nine vendor questions and the answers that should worry you
| Ask | Answer that should concern you | What good looks like |
|---|---|---|
| What is your measured error rate, and on what set? | A number with no described test set | A described methodology, even if the number is unflattering |
| Does every finding carry source text? | “It links to the document” | The specific sentence, quoted, not just a page reference |
| What happens when the model is uncertain? | “It's very accurate” | Uncertain output is visibly marked or demoted |
| How do I tell “nothing found” from “not examined”? | Blank space in the report | Explicit coverage reporting per category |
| Is the same document scored the same way twice? | “Broadly, yes” | A stated determinism policy and a way to test it |
| What happens above your length limit? | Nothing — it isn't mentioned | An explicit error or a documented routing path, not silent truncation |
| Is our data used for training? | “Not by default” | Contractual prohibition, in writing, no default to negotiate away |
| Can I reconstruct last quarter's output? | “We're always improving the model” | Versioned models with retained run records |
| What do you indemnify? | Fees paid, capped, with an AI-output carve-out | Read carefully — see the caveat below |
Why output shape predicts reliability better than model choice
Buyers spend evaluation time on which model sits underneath. It is close to irrelevant to your risk, because it is the layer you cannot inspect, cannot control, and cannot audit. What you can inspect is the shape of what comes out.
Structured, scored output — a value per defined category, each carrying the source text that produced it, measured against a stated threshold — is more reliable than narrative output for reasons that have nothing to do with model quality:
- It exposes coverage. If every category returns a score, a category that found nothing is visible as a low score with no supporting quote. In a narrative report, the same situation is indistinguishable from the topic never having been considered.
- It is disagreeable in the useful sense. A reviewer can say “that clause is market-standard, this should score 30, not 82” — a specific, recordable correction. Nobody argues with a paragraph; they either accept it or rewrite it.
- It is comparable. The same rubric across 900 documents produces something you can rank and triage. Narrative summaries of 900 documents produce 900 things to read, which is the problem you started with.
- It is reconstructible. A score, a threshold, a quote and a model version is a complete record of why a document was flagged. That is what makes the decision defensible later, when someone asks why this was or was not escalated.
This is worth stating precisely, because the overclaim is tempting: structured scoring does not make the underlying model more accurate. It makes the model's errors cheap to find. Given a measured base rate that nobody has eliminated, cheap-to-find is the property with real operational value.
The obligation behind all of this
ABA Formal Opinion 512, issued 29 July 2024, is the profession's first formal ethics guidance on generative AI. It holds that lawyers using these tools must “fully consider their applicable ethical obligations,” including competence, confidentiality, client communication, supervisory responsibility, and fees reflecting time actually spent.[2]
The practical reading: the duty attaches to the work product that leaves your desk, not to the procurement decision that preceded it. Choosing a well-regarded vendor does not discharge it. A verification path that you actually follow does.
Bottom line
Reliability is not a property of the model. It is a property of the loop: output you can check quickly, a way to tell silence from coverage, reproducibility across runs, and a record you can reconstruct months later.
Test on documents where you know the answer. Trace the high-severity findings to source text. Run the same file twice. Then ask what happens when it is wrong — and if the honest answer is “we would probably not notice,” the accuracy number does not matter.
Sources
- Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. arXiv:2405.20362; published in the Journal of Empirical Legal Studies (2025). arxiv.org/abs/2405.20362
- American Bar Association Standing Committee on Ethics and Professional Responsibility, Formal Opinion 512: Generative Artificial Intelligence Tools, 29 July 2024. americanbar.org
We cite only sources we have retrieved and read. Where a figure could not be verified, we changed the figure rather than the citation — see our methodology.