Use a buyer-controlled, representative test with a concurrent or carefully matched comparator.

Treat the pilot as an experiment in the complete workflow, not a demonstration of attractive outputs.

1. Freeze the protocol. Agree scope, weights, acceptance standard, thresholds, severity rules, cohort dates, exclusions and stopping conditions.
2. Choose files independently. Stratify by peril, complexity, geography, inspection source, input quality and other workflow-relevant factors. Keep known difficult-input cases visible.
3. Separate configuration from evaluation. Permit tuning on a development set, then use an unseen holdout. Record model, prompt, rule and price-list versions.
4. Control the comparison. Use matched historical packages or randomised live allocation where operationally and legally appropriate. Give routes equivalent evidence; prevent leakage from the incumbent’s completed answer.
5. Calibrate reviewers. Have two reviewers independently assess a subset, adjudicate disagreements and measure agreement on severity. Blind supplier identity where practical.
6. Capture all work. Log checking, email queries, duplicate entry, corrections, escalation and fallback activity—not just time spent inside the supplier interface.
7. Follow files after acceptance. Use an agreed observation window for discovered production defects. Separate later evidence of additional damage from errors present in the original submission.

NIST’s generative AI profile recommends evaluating systems in real-world conditions and documenting human overrides. Its treatment of confabulation also supports testing factual support, rather than accepting a fluent explanation as evidence. (nvlpubs.nist.gov)

How many files are enough?

There is no universal pilot sample size. Set it from the error rate, improvement or service risk the buyer needs to distinguish, with enough observations in important segments.

For illustration, under an independent, constant-probability binomial model, zero material errors in 100 tested files gives a one-sided 95% upper bound of approximately 2.95%; zero in 300 gives approximately 0.99%. These are calculations using exact binomial reasoning, not a guarantee that the next file is safe. NIST documents exact and one-sided binomial confidence-limit methods. (itl.nist.gov)

Files sharing an inspection team, catastrophe or rule defect may not behave as independent observations. Have the analysis account for clustering and loss mix; do not represent an overall clean sample as proof for every subgroup. (itl.nist.gov)