Version Control and Audit Trails in AI Review Tools

Updated August 2026 · 6 min read · Deal Room Intelligence Series

The moment this matters

Eighteen months after close, a provision surfaces that nobody escalated. An insurer, a claimant, or your own committee asks a simple question: what did you know, and when?

There are only two possible answers, and the gap between them is enormous.

The first describes a managed process where a judgment turned out wrong — an ordinary professional risk. The second is indistinguishable from having no process. Note that both can arise from identical system behaviour. What separates them is entirely what was recorded.

What a complete record contains

Two rows deserve attention because they are the least common and among the most valuable.

Threshold in force. A pre-committed escalation line turns “we missed it” into “it ranked below our documented threshold” — a risk-appetite decision rather than an oversight. But that only works if the threshold is recorded as it stood at the time. A threshold you can describe afterwards but not evidence looks like a rationalisation, because there is no way to distinguish it from one.

Human overrides. This is the record that a professional was actually in the loop. ABA Formal Opinion 512 (29 July 2024) holds that lawyers using generative AI must “fully consider their applicable ethical obligations,” including competence and supervisory responsibility.^[1] An override log is the most direct evidence that supervision happened rather than being asserted.

Why model versioning is not a nicety

“We're always improving the model” is offered as a virtue and is, for audit purposes, a serious problem. It means the system that produced your report no longer exists.

Consider what that does to the questions you will be asked:

• Would the system flag this today? Possibly — but that says nothing about what it did then.
• Can you re-run the original analysis? No.
• Was this a system limitation or a configuration choice? Unanswerable.

This matters more given the measured error rates in the category. Grounded commercial legal AI has been found to hallucinate between 17% and 33% of the time in an adjacent task, with providers' hallucination-free claims judged overstated.^[2] If errors are expected, the ability to demonstrate which error occurred, under which version, is the difference between a bounded problem and an open one.

Determinism, and the honest version of it

Ask a vendor to run the same document twice, live, and compare. Language models are not deterministic by default, and a system that returns materially different findings across runs cannot support a reconstructible record.

There are legitimate engineering answers — fixed sampling parameters, cached results per document version, pinned model snapshots. There is also an illegitimate one, which is silence. The test is cheap and you should insist on running it during evaluation rather than taking a description.

A related question worth asking: if the document is re-uploaded unchanged, does the system re-analyse it or return the stored result? Re-analysis is more honest but introduces drift; caching is stable but can hide a model change. Either is fine if it is disclosed. Neither is fine if you cannot tell which is happening.

The audit gap nobody closes: read access

Most systems log writes — uploads, edits, deletions — and not reads. That is exactly backwards for deal work, where the sensitive event is someone seeing a document, not changing it.

This bites hardest in clean-team arrangements. If designated material is walled off from a subset of your own people, you need to be able to demonstrate the wall held. A system that indexes a whole data room and answers questions across it can quietly become the most effective way to read around a wall — and without read logging, there is no way to prove either that it happened or that it did not.

Insist on: per-deal-room isolation, role-based access enforced server-side rather than by hiding a button, and a read-level log you can export.

Retention, and the tension with security

Audit trails argue for keeping things. Confidentiality argues for deleting them. Both are right, and the resolution is to separate the two classes of data.

• Delete the documents. They stop being needed at close and become pure exposure afterwards.
• Retain the metadata record — inventory, scores, thresholds, versions, overrides, and the quoted source sentences. This is a fraction of the volume, carries far less sensitivity than the full documents, and it is what reconstruction actually requires.

That split gives you a defensible record without a standing archive of confidential deal material, and it is worth specifying in the contract rather than accepting a single blanket retention setting.

What to require

1. Versioned models and rulesets, with the version stamped on every stored result
2. Run records retained for a stated period, exportable in a portable format
3. Thresholds recorded as they stood at run time, not merely as current configuration
4. Source quotes stored alongside findings, not regenerated on demand
5. Override capture with rationale, attributable to a named person
6. Coverage record: submitted, processed, failed, truncated
7. Read-level access logging, exportable
8. Document deletion decoupled from metadata retention

Bottom line

Accuracy is what you hope for; reconstructability is what you can prove. Since the measured evidence says errors are permanent in this category, design for the second — because when a miss is investigated, the question is never whether the system was good. It is what the record shows.

Version the models, stamp the threshold, store the quote, capture the override, log the reads, and keep the metadata after you delete the documents.

See a scored run with its source text →