Ask what was counted, what was excluded and whether the measure corresponds to the carrier’s acceptance risk. Field-level extraction accuracy, line-item agreement and complete-file acceptability are not interchangeable procurement measures.
Request a substantiation schedule covering:
| Question | Evidence to request |
|---|---|
| What is the unit? | Field, document, line item, estimate or complete file |
| What is the denominator? | All attempted units, including failures and abstentions |
| Who selected the sample? | Selection method and excluded populations |
| What is the reference answer? | Independently reviewed evidence and adjudication method |
| What does an error mean? | Severity definitions, omissions and unsupported additions |
| How much human work occurred? | Supplier and carrier edits before scoring |
| Does it generalise? | Results by loss, complexity, inspection source and guideline version |
| How uncertain is the result? | Sample size and an appropriate interval estimate |
As a hypothetical illustration, a system can correctly extract numerous administrative fields while omitting one materially affected room. A high field-level score would not answer whether that file is acceptable.
Require the supplier to report abstentions and failure-to-deliver separately. For automated QA, test both missed material defects and unnecessary alerts; otherwise, the carrier can receive a large alert workload without a corresponding improvement in accepted-file quality.