The following protocol is a proposed design, not an industry-prescribed sample size or duration.
Phase 1: establish the baseline and acceptance authority
Choose a defined routine residential cohort, such as water or wind losses. Separate complex losses and unresolved coverage questions rather than letting either supplier selection or reviewer discretion move them in and out of the comparison.
Name:
- An economic sponsor responsible for the business case.
- A technical claims owner responsible for acceptance.
- An operations owner responsible for retained work and exceptions.
- Technology, security, procurement and legal approvers.
Record the incumbent’s actual touch time and queue time. Agree material-error definitions and calibrate reviewers on a shared sample before scoring suppliers.
Phase 2: compare outputs on controlled inputs
Use identical, frozen evidence packages for a replay comparison where appropriate. Prevent access to another route’s completed answer, use equivalent instructions, and blind reviewers to producer identity where practical.
Include incomplete and contradictory evidence deliberately. Score whether the producer identifies the problem and escalates appropriately—not whether it invents a complete-looking answer.
Phase 3: validate the live operating model
A replay does not test real intake queries, hand-offs or queue behaviour. Follow it with controlled live allocation across comparable strata: peril, complexity, inspection source and other relevant operating conditions.
Keep a frozen evaluation set separate from examples used to configure rules. Record any changes made during the pilot and retest affected cases.
Which pilot metrics should decide the result?
| Metric | Proposed definition | Protection against misleading results |
|---|---|---|
| First-pass acceptance | First submissions accepted without a required return ÷ first submissions reviewed | Show the unreviewed queue and reason-coded returns |
| Accepted yield | Accepted files ÷ all assigned in-scope files | Retain declines, escalations and unresolved work |
| Material-defect rate | Independently reviewed files with at least one material defect ÷ files audited | Use an independent sample; report count and denominator |
| Escaped material defects | Material defects found after production acceptance ÷ accepted files subsequently audited | Keep follow-up periods comparable |
| Retained touch time | All carrier intake, review, correction, query and exception minutes | Include repeated handling and rescue work |
| Time to usable file | Initial assignment or receipt to defined acceptance | Report median and tail time; show incomplete-input pauses separately |
| Revision burden | Revision cycles and correction effort by cause | Separate new evidence from original defects |
| Cost per accepted file | Cohort costs ÷ accepted files in the same cohort | Label immature cohorts provisional |
| Export success | Required records successfully retrieved and reconstructed | Test a real exit package, not a contractual assertion |
Do not adopt an arbitrary universal acceptance percentage or pilot size. Set thresholds from the current baseline, the minimum improvement worth buying and the consequences of missed defects. Report uncertainty and subgroup results; a small pilot without observed serious errors does not establish that rare errors are absent.