What Happened
A critique from the AI Now Institute challenges claims that today’s AI systems provide a sound basis for estimating human-extinction risk. It also argues that safety rules designed and assessed primarily by model providers could strengthen those providers’ position while limiting independent scrutiny. These are arguments about evidence and accountability, not proof that severe future risks are impossible [1].
The policy landscape offers different tools for different purposes. The EU AI Act sets legal requirements according to an AI system’s use and risk category. The NIST AI Risk Management Framework is voluntary guidance for managing risk across an AI system’s lifecycle. OECD principles address trustworthy AI at a policy level. The UK AI Security Institute studies and evaluates advanced-model risks; it is not a substitute for a business’s own controls. Research and civil-society groups, including Stanford HAI, Partnership on AI, the Center for AI Safety and AI Now, bring differing evidence and priorities to the debate.
Why It Matters to Businesses
Businesses must govern the application they deploy, not just the model they buy. A provider’s safety report may inform procurement, but it does not establish that a customer-facing agent handles personal data lawfully, resists prompt injection or produces reliable decisions in a particular workflow. Conversely, disagreement over speculative risks does not remove the need to test measurable harms today [1].
Kimbodo Engineering Perspective
We would treat vendor assurances as inputs to assurance, not as the assurance itself. Controls should scale with the use case: a drafting assistant warrants different review and release gates from an agent that can change records, spend money or affect access to services. Governance is most useful when a named owner can show what was tested, what failed and who accepted the residual risk.
How We Would Implement It
- Classify the use case: Record its purpose, users, data, model providers and possible effects on people. Obtain legal review where EU AI Act or sector-specific obligations may apply.
- Set measurable gates: Define task-quality, privacy, security and misuse tests before launch. Include representative cases, adversarial prompts and failure thresholds relevant to the workflow.
- Constrain agents: Use least-privilege tool access, approval for consequential actions and separation between retrieved content and executable instructions.
- Monitor and reassess: Log versions, actions, approvals and incidents with appropriate data minimization. Re-run evaluations after model, prompt, tool or workflow changes.
Risks, Costs and Security
Independent testing, human review and audit trails add cost and latency; omitting them can leave failures undiscovered until they affect customers. Testing also has limits: a passing evaluation is evidence for the scenarios tested, not a guarantee of safety. Procurement should address access to evaluation results, incident notification, data retention and exit options, so governance does not depend entirely on a provider’s own account of its systems [1].
Where Kimbodo Comes In
Kimbodo builds and operates this in production for businesses — see our AI Cost & Governance practice, or Analyze My AI Costs.
