Processing speed is the least important number in this decision, and vendors quote it because it is the most flattering. A system that scores 1,200 documents in an hour has not saved you an hour — your reviewers were never going to read them in an hour anyway. What matters is time-to-first-useful-answer, and that is governed by things other than throughput.
What actually determines elapsed time
| Factor | Effect | Usually disclosed? |
|---|---|---|
| Text extraction | Scanned documents need OCR — often the slowest stage | No |
| Document length | Long documents chunk into multiple passes | Rarely |
| Depth of analysis | Single-pass scoring vs chunked deep analysis differ by an order of magnitude | Sometimes |
| Concurrency limits | How many documents process simultaneously | No |
| Upstream rate limits | The model provider's throughput ceiling, shared across all customers | Almost never |
| Retries | Failed calls re-run, silently extending elapsed time | No |
The last three are the ones that turn a quoted figure into a real one. Throughput benchmarks are measured on a quiet system with clean text; a Monday morning with three deals running concurrently is a different machine.
The number that matters instead
Reframe from “how fast can it process the room” to “how long until I know whether to keep working on this deal?”
That question has a completely different answer, because it does not require the whole room. It requires the blocking categories scored on whatever documents exist right now. A pass that returns transaction-ending issues on 200 documents within the hour is worth vastly more than one that returns a complete analysis of 1,200 documents overnight — because the first can stop the workstream before it runs, and the second arrives after the day is committed.
Why “overnight is fine” is usually wrong
Teams often conclude that batch processing overnight is adequate, since nobody reads results at 2am. That reasoning holds for the full analysis and fails for the first pass, for a specific operational reason: data rooms fill incrementally.
Documents arrive in batches across days or weeks, and the ones that arrive late are disproportionately the ones a seller was slow to disclose. A system that only produces value on a complete room produces its first useful output at the point where most of the budget is already committed.
What you want instead is three passes on different clocks — a blocking-category pass on day one against whatever exists, a full pass as the room fills, and a delta pass on every new upload batch. The delta pass is the one teams skip and the one that catches late surprises.
Where the real delay sits, and it is not the model
In most deployments, elapsed wall-clock time is dominated by two things that have nothing to do with analysis speed.
Extraction failures. Scanned documents without a text layer, unsupported formats, password-protected files, corrupted uploads. These consume time, and worse, they frequently fail silently — producing no findings, which is indistinguishable from a clean document in most report formats. A fast run that skipped 8% of the room is not a fast run; it is an incomplete one that finished early.
Verification. Someone must check escalated findings against source text before anything reaches a committee, and that is human time that begins after processing ends. Grounded commercial legal AI has been measured hallucinating between 17% and 33% of the time in an adjacent task, with providers' hallucination-free claims found to be overstated.[1] So verification is permanent, not transitional.
This is why quoted throughput is close to irrelevant to your actual timeline. If a system processes 1,200 documents in 40 minutes and produces findings that each take four minutes to verify, the binding constraint is not the 40 minutes.
The question that predicts your real elapsed time
Not “how many documents per hour?” but “how long does it take one of my people to confirm one of your findings?”
If every finding carries the source sentence beside the claim, verification is seconds and the throughput number roughly holds. If findings link to a document without quoting it, verification means re-reading — and a system that processes ten times faster delivers results ten times faster into a queue that moves at the same speed as before.
Questions worth asking a vendor
- What is your time to first blocking-category result on a partial upload? This is the number that maps to decision value.
- What happens to documents that fail extraction — do they appear in the output? Listen for whether failures are reported or silent.
- What is the document length ceiling, and what happens above it? Silent truncation is common and rarely disclosed.
- Is throughput affected by other customers' load? Shared rate limits are the norm; the honest answer is yes.
- Can we re-run only new and amended documents? Without a delta mode, every incremental upload means reprocessing the whole room.
- Does every finding carry the source sentence? This decides your real timeline more than any throughput figure.
What to measure once you are running
- Submitted versus processed count. Reconcile these before reading a single finding. This is the highest-value two minutes in the whole workflow.
- Time from upload to first escalation-worthy finding — not time to completion.
- Verification minutes per escalated finding. Multiply by your escalation volume; that is your real bottleneck.
- Re-processing time on a delta batch. If it is the same as a full run, incremental disclosure will be painful.
Bottom line
Throughput is a vanity metric in this category. Your reviewers were never the bottleneck at reading speed — they were the bottleneck at deciding what to read, and now at verifying what comes back.
Optimise for time-to-first-blocking-answer on partial data, for extraction failures being loudly reported rather than silently skipped, and for findings that can be confirmed in seconds. Get those three right and the documents-per-hour figure stops mattering, which is roughly the correct amount for it to matter.
Sources
- Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. arXiv:2405.20362; Journal of Empirical Legal Studies (2025). Measures legal research, not document review — see the scoping note above. arxiv.org/abs/2405.20362
We publish no throughput benchmark here because a figure measured on our own chosen documents would not predict yours. We cite only sources we have retrieved and read — see our methodology.