AI Safety Reports Could Become Your Company's Deployment Rules
Dr. Aditya Vikram Kashyap, AI governance and innovation leader exploring how autonomous systems reshape institutions, markets and power.
gettySuppose a mid-sized U.S. insurer has commissioned an independent assessment of the AI system it wants to use for claims triage. The assessor ran adversarial tests, measured error rates on historical files and reviewed the vendor’s controls. The report is favorable. It covers a defined scope, a named model version under stated conditions and does not say what the company should permit. Management must decide whether the system may close low-value claims without a human signature, who can suspend it and what happens when the vendor ships an upgrade. The board must satisfy itself that those decisions were made deliberately. Mishandled claims and regulatory findings land on the company, not on the report.
The market for such evidence is not new, but its reach is growing. Munich Re says it issued its first AI policy, for an anti-fraud model, in 2018, and now describes generative AI coverage too. Its underwriting evaluates model development pipelines and monitoring processes and sets premiums by model quality and performance stability: a direct link between technical assessment and financial terms. Public institutions and investors are adding capacity: FAR.AI leads a consortium supplying chemical, biological, radiological and nuclear risk modeling and evaluation to the European Commission’s AI Office, and Temasek established Resaro for independent third-party testing of mission-critical AI systems. Generative and agentic applications are the newer frontier, with less settled methods and short loss histories.
Influence arrives through contracts. A large customer that requires an independent assessment before onboarding a vendor, or an insurer that conditions coverage on verified controls, turns assessment criteria into deployment requirements for firms bound by them. Those criteria may originate with the assessor, the customer, the insurer, a technical standard or a regulator. The institutional question, in my view, is whether executives understand the assumptions and limitations in criteria they adopt into their own approval decisions. An assessment can determine whether a counterparty finds a system commercially acceptable without acquiring the institution’s authority to approve deployment.
That voluntary market differs from what Lawrence Lessig and Yoshua Bengio proposed this week in The Economist: that Congress affirm technology companies’ liability for harm their systems cause and make insurance compulsory, so the market polices safety through pricing. They acknowledge that private insurers could not carry these risks alone and suggest government could enable the market, as it did for nuclear power. A compulsory regime would also have to define the risks covered and the capacity available. Existing specialist policies and the authors’ broader proposal need not cover identical exposures.
Either way, underwriting gives assessment findings commercial consequences. A carrier can price for verified controls such as logged actions, hard execution limits and a tested suspension procedure, condition coverage on those controls staying in place and require fresh evidence at renewal after a material model change. Correlation complicates pricing: when many deployers build on the same foundation models, a defect in one can produce claims across many policyholders at once, straining diversification. Carriers can respond with exclusions and sub-limits, and private capacity should not be assumed sufficient for catastrophic AI exposures.
Consider a bank. The treasury team wants faster vendor payments because invoices are backing up behind a manual approval queue. The bank proposes deploying an AI agent to initiate payments with an autonomous-payment limit of $25,000, two human approvers above it, read access to one accounts payable system and a named model version. An assessor tests the agent under those conditions. The executive who owns payment operations, not the assessor, approves those operating conditions, names an officer who can suspend the agent and defines what triggers reassessment. That executive needs the assessment’s scope, findings, exclusions and a plain statement of what the evidence does not cover.
Six months later the pressure persists. Treasury raises the autonomous-payment limit to $100,000 to clear the queue, operations connects the agent to a procurement system and the bank accepts a vendor upgrade that promises better accuracy. Each change has a plausible business rationale. Together they leave the assessment describing a system that no longer exists. The responses should differ. The higher limit expands the agent’s authority beyond approved conditions, so payments above $25,000 should stay blocked until that authority is reapproved on new evidence. The connection and the upgrade require review: if either enables actions beyond the approved conditions, the affected functions should remain blocked pending reapproval; otherwise the bank should document whether existing evidence remains sufficient and commission reassessment where it does not.
Approval can also become unreliable without any configuration change. Performance deteriorates, inputs shift or an agent behaves harmfully in a way no one anticipated. The NIST AI Risk Management Framework counts production monitoring, tracking of emergent risks and the ability to disengage or deactivate a system behaving inconsistently with its intended use among its expected outcomes. Reassessment needs performance triggers as well as change triggers.
How much reliance a finding can support depends on how it was produced. Buyers should examine the assessor’s domain expertise, independence safeguards, commercial relationships (including whether it sells remediation) and access to evidence. Every report should state the version tested, the operating conditions assumed, what was excluded and the limits of its method. Hands-on testing matters, but it does not substitute for inspecting documentation and governance; a model that performs well under test can still be deployed badly.
Before relying on an assessment, a board or executive should have four things: a clear scope, explicit deployment conditions derived from the findings, a named owner for each condition and a process for handling changes and deterioration. With those in place, criteria set by a customer, insurer or assessor become inputs to the institution’s judgment rather than substitutes for it. Public oversight can set the floor through minimum expectations for what assessments cover, transparency about scope and conflicts and accountability when evidence proves wrong. External assessment criteria will keep gaining influence. The institutions that deploy AI well will be those that decide, deliberately, what those criteria are allowed to govern.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
