What Happened
AI Now Institute’s Kate Brennan argues that pressure to recover AI infrastructure spending could encourage deployment in public institutions even when a system is unsafe or unsuitable. Citing a Bain estimate, she says AI companies may need $6 trillion in annual revenue by 2031 to sustain $1.5 trillion in yearly capital spending. She calls for rules that keep harmful products off the market and suggests considering strict liability for significant harms, including injuries, deaths and cyberattacks. These are policy proposals, not a description of current law [1].
Why It Matters to Businesses
AI governance is moving beyond statements of principle toward evidence that a particular system is fit for a particular use. The EU AI Act creates legal obligations for covered uses; the NIST AI Risk Management Framework and OECD AI Principles provide governance guidance. Safety evaluations can inform decisions, but a model-level result does not establish that an application is safe with a company’s data, users, tools and operating procedures.
Kimbodo Engineering Perspective
The consequential decision is often not which model to buy, but where to permit automation. A drafting assistant, a benefits-eligibility workflow and an agent that can change production systems need different controls. Brennan’s concern about premature deployment reinforces a practical rule: when failure could materially harm people or systems, require deployment evidence and an accountable owner before launch [1].
How We Would Implement It
- Inventory AI uses, data flows, affected people and available human remedies; classify each use by impact and applicable obligations.
- Define acceptance tests for accuracy, misuse, privacy, security and failure recovery using representative production scenarios.
- Limit agent permissions, require approval for consequential actions, and log inputs, outputs, tool calls and overrides with appropriate access controls.
- Release in stages, monitor failures and drift, and assign owners who can pause or roll back the system.
Risks, Costs and Security
Evaluation, human review and audit logging add cost and latency; insufficient controls can create larger operational and legal exposure. Logs themselves may contain sensitive data, so retention limits and access restrictions matter. No evaluation eliminates all harm, and neither voluntary standards nor a vendor’s safety claims substitute for testing the deployed application. Strict liability would change incentives if adopted, but its scope and effect depend on legislation [1].
Where Kimbodo Comes In
Kimbodo builds and operates this in production for businesses — see our AI Cost & Governance practice, or Analyze My AI Costs.
