Require demonstrations of failure handling, not just the happy path.

The following are proposed buyer tests, not claims that every supplier currently supports them.

Can a reviewer reconstruct the evidence?

Choose material estimate elements and ask the producer to show:

• The original photograph, note, measurement or other supporting input.
• The calculation or rule used to produce the quantity or description.
• Any assumption, uncertainty or conflicting evidence.
• The relevant version and subsequent edits.
• The person who approved the final package.

Repeat the exercise after export. A generated rationale is not a substitute for the underlying evidence; NIST specifically identifies the risk of confidently incorrect content and fabricated supporting logic or citations. (nvlpubs.nist.gov)

Does the integration preserve the claim correctly?

Run a representative round trip covering assignment identifiers, estimates, photographs, attachments, notes, status, revisions and acknowledgements. Test duplicate delivery, missing attachments, interrupted uploads, retries, stale versions and a revoked user.

Verisk documents these exchange capabilities but explicitly says available APIs and workflows depend on the customer’s agreement and entitlements. Therefore, an “Xactimate integration” claim should trigger a deployment-specific test, not an automatic high rating. (developerportal.verisk.com)

Can the carrier control changes and actions?

Introduce a changed carrier guideline and an ambiguous input. Require the producer to demonstrate versioned application, a safe escalation, an authorised override and rollback. Include tests in which source documents contain misleading instructions; the expected result is treatment as evidence, not permission to change authority or disclose data.

For non-AI workflows, test the equivalent risks: spreadsheet changes, rule edits, unauthorised macros and altered review procedures. Do not remove the governance criterion merely because a supplier says it does not use AI.

Can security evidence be tied to the actual service?

Ask security reviewers to inspect the scope and period of assurance reports, relevant exceptions, customer responsibilities and subprocessor coverage. Do not award technical-accuracy points for a security badge.

New York DFS’s third-party guidance discusses risk-based assurance, access, downstream providers, data location, AI training use and exit obligations. It distinguishes regulatory requirements from recommended contractual practices; preserve that distinction in the tender. (dfs.ny.gov)