Any single accuracy figure for this task is meaningless, and not because the measurement is hard. It is because “anomaly” is defined relative to a baseline — unusual compared to what? — and every vendor picks their own. Change the baseline and the same system's accuracy moves by tens of points without a line of code changing. Before you compare numbers, you have to compare baselines, and almost nobody does.
Why the number moves so much
An anomaly detector answers “is this clause unusual?” That question has no answer without a reference set. Four are in common use, and they disagree constantly:
| Baseline | “Unusual” means | Fails when |
|---|---|---|
| Vendor's training corpus | Rare across contracts generally | Your sector's norms differ from the corpus |
| Your own playbook | Deviates from your standard positions | You are the buyer reviewing someone else's paper |
| The deal's own set | Inconsistent with sibling contracts | The whole set is non-standard |
| Legal-risk rubric | Carries material risk, common or not | Common-but-dangerous vs rare-but-harmless get conflated |
These produce genuinely different outputs on the same document. A 60-day termination notice is unremarkable against a general corpus, a deviation against a playbook demanding 90, perfectly consistent within a deal where every contract says 60, and — depending on the counterparty's importance — either trivial or serious on a risk rubric.
The trade-off nobody puts in a datasheet
Detection systems trade recall against precision, and you cannot maximise both. What matters is that the two failures cost you very different amounts in diligence.
- A false negative — a real anomaly not flagged — is a provision that reaches closing unexamined. It surfaces post-close, when leverage is gone. Cost: potentially the deal economics.
- A false positive — a routine clause flagged — costs reviewer minutes. Individually trivial.
That asymmetry argues for tuning toward recall, and most vendors do. But the reasoning has a limit that gets ignored: false positives are not independently cheap, they compound. At forty spurious “critical” findings, reviewers stop reading carefully, and the forty-first — which is real — gets the same skim as the forty before it. High recall bought with unusable precision converts a detection problem into an attention problem, and attention was the scarce resource you were trying to protect.
The practical answer is not a better ratio. It is ranking: keep recall high so nothing is silently dropped, but order findings by severity against a defined rubric so reviewer attention flows down a list rather than across an undifferentiated pile. A flat list of 400 flags and a ranked list of the same 400 have identical accuracy and completely different operational value.
What the category actually achieves
There is no independent published benchmark for anomaly detection in M&A contract review — no peer-reviewed measurement, and no vendor publishing test methodology. We looked; if you find one, it is worth more than anything on this page.
The closest rigorous evidence in an adjacent task is Stanford's evaluation of commercial legal AI research tools, which found the products sold by LexisNexis and Thomson Reuters “each hallucinate between 17% and 33% of the time,” and concluded providers' hallucination-free claims were overstated.[1]
Measuring it yourself, which is the only real option
Build a set where you know the answer
Take 40–60 documents from a closed deal. Have someone who worked it list every anomaly they consider material. That list is your ground truth — imperfect, but yours, and calibrated to your firm's actual standard rather than a vendor's.
Measure recall against that list, not against total findings
Of the anomalies you know are there, how many surfaced? This is the only number that maps to the risk you care about. A system returning 300 findings while missing two of your five material ones has failed, regardless of how thorough the output looks.
Measure precision at the top of the ranking
Not overall precision — precision in the top 20 findings, because that is what a reviewer actually reads under time pressure. A system with mediocre overall precision and excellent top-20 precision is more useful than the reverse, and standard metrics hide this entirely.
Include clean documents
Seed documents you are confident are unremarkable. Findings returned on those are your false-positive rate on the material most likely to be waved through.
Check severity calibration separately from detection
A system can find the right clause and rank it wrongly. Track these as distinct failures — detection misses and severity misses need different fixes, and conflating them makes the output uninterpretable.
Test the same document twice
If findings differ between runs, the accuracy figure is a sample from a distribution rather than a property of the system, and you cannot reconstruct a decision you cannot reproduce.
The failure that no accuracy metric captures
Precision and recall are computed over findings. Both are blind to the document that was never processed — a parse failure, a silent truncation past a length limit, an unsupported format. That document contributes nothing to either metric and appears in the report as clean.
This is the largest practical source of missed anomalies in real deployments, and it is invisible to every accuracy discussion. Which is why the control that matters most is not a better detector but coverage reporting: how many documents were processed, how many failed, which categories returned results, and whether anything was truncated. A system that scores every document against every defined category makes a gap visible as a low score with no supporting quote, rather than as silence.
Questions worth asking
- Unusual compared to what? Name the baseline. Can we substitute our own?
- Recall and precision on what test set, built by whom? A number without a described set is decoration.
- Is precision reported overall or at the top of the ranking? Only the second predicts reviewer experience.
- Are detection and severity errors tracked separately?
- What is the document length ceiling, and what happens above it? Listen for silent truncation.
- Does every finding carry the source sentence? This decides whether verification costs seconds or costs re-reading.
- How do I distinguish a clean document from an unprocessed one?
Bottom line
Accuracy in anomaly detection is not a property of the system; it is a property of the system plus a baseline plus a severity rubric, and vendors control all three when they quote you a figure. Ignore the number and test on documents where you know the answer.
Then optimise for the two things that determine operational value rather than benchmark value: ranking, so high recall does not drown the reviewer, and coverage reporting, so the document that never got processed cannot masquerade as the document that came back clean.
Sources
- Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. arXiv:2405.20362; Journal of Empirical Legal Studies (2025). Measures legal research, not contract anomaly detection — see the scoping note above. arxiv.org/abs/2405.20362
We publish no accuracy figure for this task because no independently verifiable one exists. We cite only sources we have retrieved and read — see our methodology.