BigDocs: the benchmark

Over the years I have adopted a handful of patterns, mostly borrowed from data architecture and engineering. This piece benchmarks them on a hard case, contract review, against the one big prompt.

The patterns

Each has its own piece. Here is the short version.

The Agentic Wordhouse. A dimensional model for your words, building on Andrej Karpathy’s LLM Wiki with ideas from data warehousing. Documents are usually stored one per instance, this contract and that contract. Customer contracts accumulate alongside your employment agreements, and the versions drift. To read one is to read a big document. To read many is to wade through a lot of content that has nothing to do with the question. The Wordhouse turns that around: one document per topic (limitation of liability, payment terms, data protection), with the variants as instances inside it. Each position records what we accept or reject, the fallback we can offer, when to escalate, and the exact source wording it came from. It sounds like rewriting all your content, but building and maintaining the Wordhouse can be automated too.

Structure first. Documents are written for human eyes. Numbering, indentation, tables and schedules carry meaning that a model only half sees, and if you ask it to find the structure for itself it will find a slightly different one each time. So before any judgement, code splits the incoming document using the structure it already carries: its numbering, headings and schedules. A second, independent reader confirms every word was captured, and anything left out is left out on purpose, with a reason. A person confirms the structure in a plain text file or a review page. Everything after that works through the confirmed structure as many small tasks instead of one giant one.

The instruction file. Left to itself, an agent decides as it goes what to read and what to skip, and you only find out afterwards. Better to write the job down first. The model reads a short index of the Wordhouse and proposes which topics each part of the document needs. A person approves that plan, and it becomes a fully mapped task list: every part, the topics it will be checked against, and the output each task must return. Only then does anything run. No embeddings or vector store, and you can see exactly what each task was given and why.

Never retype. Most people ask for the whole document back with the changes made. Every time a model copies text it has a chance to change it: 30 days becomes 60, “must” becomes “should”, a sub-clause quietly merges into its neighbour. That is where the errors nobody notices live. So the model points at text that already exists, and code copies it. Edits are operations on exact pieces of the source, placed into the customer’s own Word file by code. If the model can’t retype it, it can’t mistype it.

Output built for oversight. A tidy redline hides its reasons, so the reviewer is left to trust it or redo it. Every clause gets a row: the decision, the positions it relied on, the reasoning, a line for the customer. Escalations and gaps are set aside for a person to decide, not quietly resolved.

Maintaining the Wordhouse. A library goes stale the day after you build it, and maintenance is where most knowledge bases quietly die. The library is built and kept current by its own skill, with its own checks, so a team can look after it without a developer.

Build the tests in. Software teams stopped trusting “it looks right” a long time ago. This is borrowed straight from engineering. Every stage checks its own completeness, and the work loops until the checks pass. The same tests become the monitoring once it is in use.

The benchmark

The setup. A software provider reviews a customer’s contract bundle against its own standard paper and negotiating policy. Both sides use public, open-licensed agreements so anyone can repeat the test.

• Our paper is Common Paper (cloud service agreement, AI addendum, data protection, SLA, professional services and cover page). The Wordhouse skill built a 22-topic library of 174 positions from it, and I added a policy layer: filled-in values, fallbacks, terms we reject and escalation triggers.
• The customer’s paper is the Bonterms bundle: six documents, 585 paragraphs, about 25,000 words.
• I planted 15 departures in each test case, so there is a known right answer. They range from a one-word change to a whole new section, and from minor to serious, and include new terms no policy covers, which a good reviewer should send to a person rather than decide.

What was compared.

Treatment	What it is	Runs
Structured pipeline	The patterns above, as Claude skills	11 on the Bonterms cases, 3 identical repeats on one case
Single prompt, changes only	Fresh Claude, no tools, one prompt holding our paper, the policy note and the customer contract. Asked for the changes it wants	1
Single prompt, every paragraph	Same inputs. Must return every paragraph with its decision and a reason, and is told that silence is a failure	3

Same model throughout.

The inputs are deliberately unequal. The single prompt gets the raw documents and the policy; the pipeline gets the Wordhouse built from them. The Wordhouse is part of what is being tested. This is method against method, not model against model.

	Structured pipeline	One prompt, changes only	One prompt, every paragraph
Knowing you have it all
Stopped early without saying so	0 of 11	Can't tell	3 of 3
Paragraphs accounted for	100%	Not reported	86 to 89%
New term no policy covers, sent to a person	3 of 3	No	Never reached
Getting it right
Planted departures classified correctly (of 15)	14	10	10 to 12
Same answer next time
Same classification in all 3 runs	14 of 15	Not tested	7 of 15
Agreement on how serious each one is	96%	Not tested	64%
Showing why
Decisions citing a checkable source	85 to 87%	Not reported	1 to 17%
The redline
Words deleted per changed paragraph	6.5	About 22	About 10
Section deleted without a marker	n/a	n/a	3 of 3 runs
Effort
Cost per run	US$7 to 10	US$0.52	US$4 to 4.70
Time per run	5 to 6 min	3 min	About 17 min

What I found

It stops without telling you. In all three full runs the single prompt reached its output limit and stopped, between 11 and 14 percent short of the end. The last part of the services agreement and the whole SLA were missing, and the reply simply ended. On a larger Commonwealth government contract of about 38,000 words, both single-prompt runs stopped after 22 to 30 percent. The pipeline accounted for every paragraph in every run, and listed what it set aside on purpose, with a reason.

It is a lottery on what matters. Both approaches notice much the same problems: across three runs each agreed with itself on whether a planted departure was a problem 14 times out of 15. The difference is in how serious they say it is. Only 7 of the 15 got the same classification in all three single-prompt runs, against 14 of 15 for the pipeline. Agreement on severity was 64 percent against 96 percent. One single-prompt run scored as well as the pipeline. You can’t tell in advance which run that will be.

You don’t know why. The pipeline names a checkable library position for 85 to 87 percent of the clauses it reviews, and every position it names exists. The single prompt cited a source in 1 percent of its reasons in one run and 17 percent in another. It can cite section numbers in our standard paper, but not an internal policy with no numbering, so the decisions that matter most come back as assertions.

It edits with a heavier hand. The pipeline deleted about 6.5 words per changed paragraph. The single prompt deleted about 10 when working paragraph by paragraph, and about 22 when asked only for its changes. All three full runs deleted a whole section without marking it “[Intentionally omitted]”, and none fixed the cross-references.

It doesn’t know what it doesn’t know. When a new term appeared that no policy covers, the pipeline sent it to a person in all three runs. The single prompt invented a negotiating position for it in all three, and redlined it, so a customer could receive changes nobody had approved.

It is cheaper per run, not faster. A single-prompt run cost about US$4 to US$4.70 and took about 17 minutes, and more than 30 minutes on the government contract once the tool started resuming it. A pipeline run cost US$7 to US$10 and took 5 to 6 minutes, because the small tasks run side by side.

Anecdotally, big documents run as one task often fail outright and have to be started again. Compartmentalised work fails too, but in contained pieces, and you rarely start from scratch.

What the benchmark found in my own tools

A benchmark that only confirms what you hoped isn’t worth much. This one found faults in mine:

• The library skill’s own map check passed while 14 substantive clauses had no position. A targeted second run fixed it, and clause-level coverage is now an explicit test.
• On the government contract, the contents page was treated as part of the contract. The human structure check caught it, which is what that check is there for.
• My answer key was too strict in places: I had missed escalation rules the library skill had written. Corrections were allowed only when backed by library text, applied to both approaches, and checked by an independent reviewer who hadn’t seen the scores.

Knowing where you stand

The pipeline doesn’t win by being smarter. Both approaches notice most of the same problems. It wins because you know where you stand.

With one big prompt you are left with two questions the output can’t answer: have I got it all, and should I run it again in case it changes its mind? With the structured approach every clause is accounted for, each decision points to a position you can check, a re-run gives the same answer on what matters, and what it doesn’t know is handed to a person rather than guessed.

This will get better, as it has been. The single prompt copied 585 paragraphs back character for character and, on one run, matched the pipeline. Output limits will rise and models will steady. But today, where the business transaction rests on the answer, agents still need steering with some strictness: a confirmed structure to work through, a Wordhouse to point at, small tasks with clean context, and tests that don’t take the model’s word for it. That is how a team gets the benefit of better models now without betting the outcome on the run it happened to keep.

The test set doesn’t retire once the results are in. It becomes the regression suite, rerun on every model, library or skill change, which is the ongoing monitoring an AI management system needs under ISO 42001 (Clause 9.1, and verification and validation under Annex A.6.2.4). Build the tests in

Limits

One domain, one model, a small number of runs, and an answer key built by me and corrected after the runs under the rule described above. There are good vendor tools in this space, among them LegalOn, Spellbook, Harvey, Luminance, Kira and Ironclad. I haven’t evaluated them, and this isn’t an alternative to them. The point is the method: patterns for working with big documents, how to test the results, and how to keep watching them once they’re in use.

If your team has a document job that comes back from AI looking right and turns out not to be, bring it to a BigDocs session.