Put two vendors' feature lists side by side and they are nearly identical — both extract clauses, flag risks, support the major formats, integrate with data rooms. The features have converged. What has not converged is a set of properties that never appear on a feature list, because they describe how a tool behaves when it is wrong, and that is the axis on which these products actually differ.
Why feature parity is real
The capabilities that differentiated tools three years ago are now table stakes, for a straightforward reason: most vendors build on the same handful of foundation models. The model layer is largely commoditised. What a vendor adds on top — the rubric, the output format, the coverage accounting, the record — is where the actual product lives, and almost none of it shows up as a bullet point.
This has a practical consequence for evaluation. Time spent comparing feature matrices is time spent comparing the commoditised layer. The differentiators are all in the second column below.
| On the feature list | What actually differs |
|---|---|
| “Extracts key provisions” | Whether the extraction quotes its source sentence |
| “Identifies risks” | Whether risks are ranked, and against whose rubric |
| “Processes any document type” | What happens to the ones it cannot process |
| “Comprehensive analysis” | Whether you can tell a clean result from an unexamined one |
| “Enterprise security” | Whether the training prohibition is contractual and flows down |
| “Continuously improving” | Whether last quarter's output is still reproducible |
First, which question was the tool built to answer?
Before comparing quality, check that two tools are even doing the same job. Contract review tools fall into four families, each built for a different question, and a tool optimised for one is genuinely poor at another.
| Family | Built to answer | Strong at | Weak at |
|---|---|---|---|
| Playbook comparison | Does this deviate from our standard position? | Repeat contracting, your own paper | Buy-side diligence on someone else's paper |
| Clause extraction | Where is provision X across these documents? | Building a structured record at volume | Telling you which provisions matter |
| Conversational Q&A | What does this document say about Y? | Ad-hoc exploration of a known document | Coverage — it only answers what you ask |
| Risk screening | Which documents carry the most risk? | Triage across a large unread set | Deep single-document analysis |
Playbook comparison flags deviations from your standard positions. In M&A diligence you are reading a target's contracts with a hundred different counterparties, none drafted against your playbook, so everything deviates and everything flags. Clause extraction answers “where is X?” but not “does X matter?” — 340 change-of-control provisions with no ordering relocates the triage problem rather than solving it. Q&A demos superbly and only tells you about things you thought to ask, which makes it a better search box, not a coverage instrument. Risk screening is shallower per document; it exists to tell you which forty of twelve hundred deserve a lawyer's afternoon, and to leave a record of the rest.
For buy-side diligence the binding constraint is coverage rather than depth, so the usual answer is screening first with extraction alongside, and Q&A as a supplement, never the coverage layer. Most disappointment in this category is a family mismatch, not a quality problem. Once the family fits, the properties below decide between products.
The six properties that separate tools
1. Cost of verifying a single finding
The highest-leverage difference, and the one buyers most often fail to measure.
Grounded commercial legal AI has been measured hallucinating between 17% and 33% of the time in an adjacent task, with providers' hallucination-free claims judged overstated.[1] Nobody has eliminated the error rate, so verification is permanent. Its per-finding cost therefore multiplies across every finding, every deal, forever.
A finding that carries its source sentence verifies in seconds. A finding that links to a 90-page agreement verifies by re-reading it. Two tools with identical recall can differ by an order of magnitude in total cost of ownership on this one property alone — and it never appears in a pitch.
2. Whether silence is distinguishable from coverage
Ask: if a category returns no findings, how do I know it was assessed?
Most reports render “examined and clean” identically to “never examined” — as blank space or a comfortable zero. Tools that assign blocking categories an explicit floor when nothing is found, and report coverage as a first-class output, are in a different reliability class. This is the failure mode that reaches closing, precisely because there is nothing on the page to check.
3. Behaviour at the edges
Every system has limits. The difference is whether it tells you when it hits one.
- Document past the length ceiling — explicit error, documented routing, or silent truncation?
- Scanned document with no text layer — reported as a failure, or absent from the output?
- Unsupported format — flagged, or quietly dropped from the count?
Silent degradation is the single most common source of real misses, and it is invisible in a demo because demos use clean documents.
4. Reproducibility and versioning
Run the same document twice, live, during evaluation. If findings move materially, you cannot reconstruct a decision — and a decision you cannot reconstruct is one you cannot defend.
Related: “we're always improving the model” is offered as a virtue and means the system that produced last quarter's report no longer exists. Versioned models with retained run records are a genuine differentiator and cost the vendor almost nothing, which is why their absence tells you about engineering priorities.
5. Whose rubric the severity reflects
A score is only meaningful relative to a definition of serious. If severity is a fixed global property of a clause type, the tool cannot represent that a provision is routine in your sector and material in another — or vice versa.
Ask whether categories, weights and thresholds are configurable, and whether your reviewers' overrides feed back into calibration. A tool that improves through your use is worth substantially more over three years than one that improves only when the vendor ships.
The cost of getting this wrong is over-flagging, and it compounds. By the fortieth “critical” item that turns out to be market-standard, reviewers are skimming — and the forty-first, which is real, gets the same treatment. Measure precision in the top 20 findings, not overall: reviewers under pressure read the top of the list, and standard precision metrics hide this entirely.
6. Whether anything operates across the set
Customer concentration, aggregate indemnity exposure, and how many employment agreements lack an IP assignment are not properties of any single document. Many tools are strictly per-document, and it is not obvious from a demo, because demos run one document at a time.
The contract terms that belong on the list
Capability is half the decision. The other half is where your deal documents go, and it is settled in the contract, not the demo:
- A contractual training prohibition that flows down to subprocessors and the underlying model provider, with no carve-out for telemetry, evaluation sets or human review of outputs.
- A named subprocessor list with change notification.
- Stated retention with client-triggered deletion, so a document does not persist two years after the deal died.
- Read-level access logging, not only write logging. In deal work the sensitive event is someone seeing a document, not changing it.
On price, essentially no vendor in this category publishes one, so you cannot benchmark the headline. You can control the shape: avoid per-seat models, which tax exactly the reviewers who create value; get volume carry-forward if your deal flow is lumpy; ask for the standalone per-deal price even if you intend to buy annually, because it reveals the real unit economics; and insist on a paid pilot on your own closed deal, treating refusal as a finding.
Behind all of it sits professional responsibility. ABA Formal Opinion 512 (29 July 2024) holds that lawyers using generative AI must “fully consider their applicable ethical obligations,” including competence and supervisory responsibility.[2] That duty attaches to the output leaving your desk, not to the procurement decision — which is why every property on this page is about verifiability, coverage and reconstructability rather than raw capability.
Two things that matter less than buyers think
Which model is underneath. It is the layer you cannot inspect, control or audit, and it is largely shared across the category. Evaluation time spent here is close to wasted.
Processing speed. Your reviewers were never the bottleneck at reading speed — they were the bottleneck at deciding what to read, and now at verifying what comes back. A tool that processes ten times faster delivers results into a queue that moves at exactly the same pace.
The test that separates them in an afternoon
Take 40–60 documents from a closed deal where your team already knows every material issue. Then:
- Reconcile counts first. Submitted versus processed versus failed. A tool that silently drops documents fails here and the rest of the evaluation is moot.
- Score recall against your known issues — not against total findings returned.
- Time the verification of ten findings with a stopwatch. This number will surprise you and it is the one that compounds.
- Read the quotes against the claims, hunting misgrounded citations — a real clause cited for something it does not say.
- Run one document twice.
- Submit the credit agreement and check whether findings from the final third appear at all.
The question that outranks the test
Every real product has this story. A vendor who cannot produce one has no live deployments, is not being straight, or has no feedback path by which errors return to them — and the third is the most concerning, because it means their error rate is unmeasured rather than low.
What you want is specific and slightly uncomfortable: the class of error, how it surfaced, what changed. What you do not want is a pivot to a case study.
Bottom line
Features have converged; behaviour under error has not. The better tool is the one whose mistakes are cheap to find, whose gaps are visible rather than silent, whose edges fail loudly, and whose output you can still reconstruct in eighteen months.
None of that is on a feature list, all of it is testable in an afternoon, and it is what you will actually live with.
Sources
- Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. arXiv:2405.20362; Journal of Empirical Legal Studies (2025). Measures legal research, not document review — see the scoping note above. arxiv.org/abs/2405.20362
- ABA Standing Committee on Ethics and Professional Responsibility, Formal Opinion 512: Generative Artificial Intelligence Tools, 29 July 2024. americanbar.org
We name no competitors and publish no comparison scores, because no independent benchmark exists and we will not invent one. We cite only sources we have retrieved and read — see our methodology.