Use carrier-selected files, independent acceptance decisions and pre-agreed thresholds.

Separate configuration and learning from the scored evaluation.

NIST’s generative AI profile recommends empirical evaluation of capability claims and verification of generated sources. It also warns that laboratory or benchmark performance may not represent deployment conditions. The following protocol applies that principle to a carrier’s production workflow; it is not a NIST-prescribed claims test. (nvlpubs.nist.gov)

Pilot checklist

• ☐ Freeze the acceptance rubric, cost definitions, loss mix and success criteria before scoring.
• ☐ Use a calibration set to configure instructions and resolve reviewer disagreements.
• ☐ Keep a separate, previously unseen evaluation set.
• ☐ Include the inspection channels, complexity bands and incomplete-input conditions expected in use.
• ☐ Use matched assignments or balanced allocation; prevent the supplier from selecting only easy files.
• ☐ Prevent access to historical final estimates when they would reveal the expected answer.
• ☐ Have carrier QA score outputs independently; conceal producer identity where practical.
• ☐ Record all human assistance, including supplier experts and internal staff repairing outputs.
• ☐ Test missing evidence, contradictory inputs, late supplements, revisions and system failures.
• ☐ Keep experimental outputs from influencing live claims until authorised review is complete.
• ☐ Preserve failed, declined and unfinished assignments in the results.
• ☐ Monitor later corrections during a defined follow-up period.

For a replay of historical files, do not automatically treat the historical final estimate as unquestionable truth. Require reviewers to assess support against the evidence and approved requirements.

How many pilot files are enough?

There is no universal number for this buying decision. Set sample size from the error tolerance, desired precision, workflow segments and variability in time and cost.

For perspective, using an exact one-sided binomial calculation, zero observed errors in 100 files gives an approximately 2.95% upper 95% confidence bound; zero in 300 gives approximately 0.99%. These are desk calculations using the method described by NIST, assuming independent trials with a common error probability and reliable error detection—not proof of those rates across different claim populations. (itl.nist.gov)

Do not combine unrelated loss types into one reassuring average. Ask a statistical specialist to address clustering by inspector, producer, event or rule configuration when relevant.

What makes the pilot a go, revise or stop?

A route that costs more but provides needed surge capacity may still be worth buying. Present it as a capacity or resilience purchase—not as a cost-saving result.

Working document: Claims Automation Pilot Designer and acceptance scorecard.