What Should I Look For in an AI Due Diligence Solution?

Updated August 2026 · 6 min read · Deal Room Intelligence Series

Failure 1 — The document that was never processed

What happens: a scanned agreement with no text layer, a file past a length ceiling, an unsupported format. It produces zero findings, and zero findings looks identical to a clean document in almost every report format.

Requirement: coverage reporting as a first-class output — submitted, processed in full, partially processed, failed, with reasons. Plus a stated document length ceiling and explicit behaviour above it.

How to test: submit a scan, an over-length credit agreement, and a spreadsheet in your pilot set. Check whether failures are reported or simply absent from the count. This is the single most common source of real misses and it never appears in a demo.

Failure 2 — The finding that cost four minutes to check

What happens: a finding says “change-of-control provision identified” and links to a 90-page agreement. Verifying it means reading the agreement, which is the work the tool was bought to reduce. Under deadline, verification quietly stops happening.

Requirement: every finding carries the source sentence, quoted verbatim, beside the claim.

Why it dominates the economics: grounded commercial legal AI has been measured hallucinating between 17% and 33% of the time in an adjacent task, with providers' hallucination-free claims judged overstated.^[1] Verification is permanent, so its per-finding cost multiplies across every finding you will ever review. Two tools with identical accuracy can differ by an order of magnitude in real cost on this property alone.

Failure 3 — The clean-looking report on an unexamined category

What happens: a category returns nothing and renders as blank space or a comfortable zero. Nobody can tell whether it was assessed and found clean, or never assessed at all. This is the failure that reaches closing, because there is nothing on the page to check.

Requirement: absence of a finding must be visibly different from absence of a search. In practice: every defined category reports a value, and blocking categories carry an explicit non-zero floor plus a stated diligence request when nothing was found.

How to test: ask directly — if a category returns no findings, how do I know it was examined? If the answer is that the report omits it, the tool cannot tell you what it missed.

Failure 4 — The output that moved

What happens: the same document scores differently on two runs, or the vendor updates the model and prior results can no longer be reproduced. Eighteen months later, when someone asks why a document was not escalated, there is no answer — the system that produced the report no longer exists.

Requirement: versioned models and rulesets, with the version stamped on every stored result; run records retained for a stated period and exportable; thresholds recorded as they stood at run time.

How to test: ask them to run the same file twice, live, during evaluation. And ask what “we're always improving the model” means for last quarter's report.

Failure 5 — The forty-first critical finding nobody read

What happens: the system over-flags. By the fortieth “critical” item that turns out to be market-standard, reviewers are skimming — and the forty-first, which is real, gets the same treatment. False positives are not independently cheap; they compound, and what they consume is the attention the tool exists to direct.

Requirement: configurable categories, weights and thresholds, so severity reflects your sector and your risk appetite rather than a global default. Plus override capture that feeds back into calibration.

How to test: measure precision in the top 20 findings, not overall. Reviewers under pressure read the top of the list; standard precision metrics hide this entirely.

Failure 6 — The deal document that went somewhere unexpected

What happens: content is used for training, or reaches a subprocessor nobody enumerated, or persists two years after the deal died.

Requirement: contractual training prohibition with flow-down to subprocessors and the underlying model provider, with no carve-out for telemetry, evaluation sets or human review of outputs; a named subprocessor list with change notification; stated retention with client-triggered deletion; and read-level access logging, not only write logging.

Why the last one: in deal work the sensitive event is someone seeing a document, not changing it. Most systems log the wrong thing, and clean-team arrangements are unprovable without read logs.

Requirements that matter less than you would expect

• Which foundation model is underneath. Largely commoditised, and the layer you cannot inspect, control or audit.
• Processing throughput. Your reviewers were never the bottleneck at reading speed. Delivering results faster into a queue that moves at the same pace changes nothing.
• Feature count. Converged across the category. The differentiators are all behavioural.
• Accuracy percentages. Produced on a test set the vendor designed and does not disclose. There is no independent benchmark in this category.

The two commercial requirements worth holding firm on

A paid pilot on your own closed deal. A demo on vendor-selected documents in a vendor-tuned category, presented by someone who knows the answer, measures nothing. A pilot on a transaction where you already know every material issue measures everything. Vendors resist; push anyway, and treat refusal as a finding.

Pricing structure over headline price. Essentially no vendor in this category publishes prices, so you cannot benchmark. What you can control is the shape: avoid per-seat models, which tax exactly the reviewers who create value; get volume carry-forward if your deal flow is lumpy; and ask for the standalone per-deal price even if you intend to buy annually, because it reveals the real unit economics.

The obligation that sits behind the whole list

ABA Formal Opinion 512 (29 July 2024) holds that lawyers using generative AI must “fully consider their applicable ethical obligations,” including competence and supervisory responsibility.^[2] That duty attaches to the output leaving your desk, not to the procurement decision.

Which is why the requirements above cluster around verifiability, coverage and reconstructability rather than around capability. You are not buying a system that is never wrong — the evidence says none exists. You are buying one whose errors are cheap to find and whose work you can still evidence in eighteen months.

Bottom line

Write your requirements from the six failures, not from the demo. Coverage reporting, quoted source text, visible gaps, versioned reproducibility, configurable severity, and contractual data terms.

Then test all six in an afternoon on a closed deal. Every one of them is checkable before you sign, and every one of them is expensive to discover afterwards.

Test the six failures against our output →