Before the pilot, convert “target” in the table into a numeric threshold or a binary test approved by the relevant carrier owner.
| Criterion | Weak: typically 0–1 | Acceptable: rating 3 | Strong: rating 5 |
|---|---|---|---|
| 1. Accepted-file economics — 20 | Compares invoice price only; excludes carrier corrections, implementation or failed assignments | Reconciled cohort costs; agreed cost ceiling met; retained work and unresolved files visible | Stretch cost target sustained; downside scenarios remain affordable; capacity release and cash savings separately evidenced |
| 2. Technical quality — 20 | Undefined “accuracy”; supplier-selected samples; material omissions hidden in average estimate values | Independently checked files meet acceptance, material-error and first-pass targets by required segment | Results sustained across periods and inspection sources; post-acceptance defects monitored; corrective action demonstrably prevents recurrence |
| 3. Traceability — 12 | Reviewer must reconstruct support manually; original observations overwritten; unexplained quantities | Material estimate elements trace to evidence, calculation or explicit assumption; changes and approvals retrievable | Reconstruction tests repeatedly succeed after revisions, exports and system changes; reviewer effort meets the stretch target |
| 4. Authority and governance — 10 | Unclear decision rights; silent rule/model changes; human approval is nominal | Carrier owns approval and escalation; rules and automation versions recorded; change control and override tests pass | Repeated regression, override and rollback tests demonstrate effective control without bypassing accountable review |
| 5. Integration and export — 10 | PDF-only hand-off where editable output is required; rekeying uncosted; conflicting versions | Agreed formats round-trip successfully; identifiers, attachments, status and versions reconcile; export test passes | Repeated failure recovery and migration rehearsals succeed with low measured manual handling |
| 6. Security and privacy — 8 | Unknown subprocessors or data destinations; shared accounts; vague retention or training rights | Security team approves the actual service boundary, access, data use, retention, subprocessors and incident arrangements | Ongoing assurance, access reviews, incident exercises and deletion/export evidence remain satisfactory through change |
| 7. Usable-file time — 7 | Measures draft creation only; indefinite paused clocks; exception queue omitted | Assignment-to-acceptance and production clocks are visible; tail-time and exception targets met | Tail performance sustained through difficult-input cohorts; root-cause fixes reduce delays without suppressing escalation |
| 8. Surge and continuity — 6 | Headcount claims without available reviewer capacity; unspecified overflow | Tested intake, production, QA and carrier-review capacities support the agreed surge scenario | Repeated exercises or relevant events demonstrate quality-preserving surge, recovery and credible capacity allocation |
| 9. Commercial clarity — 4 | Unpriced revisions, unilateral scope changes, opaque minimums or punitive exit | Normalised pricing, revision definitions, service measures, audit access and exit obligations documented | Terms have worked through revisions, volume changes and an exit rehearsal; no material unpriced dependency remains |
| 10. Implementation and people — 3 | No accountable owner; reviewer workload assumed away; training or rule maintenance unfunded | Named owners, funded implementation, trained reviewers, rollout stages and rollback plan | Repeated delivery against milestones; effective backup ownership, calibration and benefit tracking after launch |
What evidence should accompany the ten ratings?
Request a compact evidence pack rather than ten separate presentations:
- Economics: invoices, production and review time records, loaded labour assumptions, correction logs and allocation schedule.
- Quality: blinded review results, severity rubric, disagreements, final adjudications and first-submission artefacts.
- Traceability: a buyer-selected file reconstructed from source evidence through revisions to approval.
- Control: responsibility matrix, rule/model inventory, change approvals, overrides and rollback records.
- Integration: round-trip results, failed-transfer logs, reconciliation reports and an independently opened export.
- Security: actual service and subprocessor boundary, assurance reports, access tests and incident exercises.
- Time: event-level timestamps, blocked-time reasons and aged exception queues.
- Capacity: committed staffing by role, competing demand assumptions and stress-test results.
- Commercial: completed price schedule, exclusions, revision terms and transition obligations.
- Implementation: funded plan, named owners, training results and operating acceptance sign-offs.