What Happened
AI governance now spans three different activities: regulation, including the EU AI Act; voluntary standards and principles, such as the NIST AI Risk Management Framework and OECD AI Principles; and safety research, including model evaluations by research institutes and AI developers. These activities inform one another, but none substitutes for the others.
Governance also has an economic dimension. AI Now Institute co-executive director Amba Kak argues that market concentration and financing risks deserve attention alongside model behavior. A reported estimate puts AI companies’ planned spending at roughly $5 trillion over five years; Bain estimates that justifying it would require nearly $2 trillion in annual revenue. These are projections, not evidence of an imminent financial loss, but they raise questions about supplier dependence and who bears the risk if growth falls short [1].
Why It Matters to Businesses
Businesses need to know which requirements are binding in each market, which standards help demonstrate sound practice, and which research findings reveal risks their own tests may miss. A vendor’s safety evaluation does not establish that a customer’s application is compliant or secure. Likewise, a policy document does not show how an agent behaves with access to production data and tools.
Supplier concentration matters operationally: pricing changes, service disruption, or a change in model behavior can affect applications built around one provider. The financing concerns raised by Kak are a reason to assess that dependency, not a basis for assuming any particular provider will fail [1].
Kimbodo Engineering Perspective
The useful unit of governance is the deployed use case, not the model in isolation. Risk depends on the task, data, users, connected systems, and consequences of an error. NIST-style risk management provides a structure for assigning owners and testing controls; EU legal obligations must be assessed separately for the application and jurisdiction. Safety work from organizations such as UK AISI, Stanford HAI, Partnership on AI, Center for AI Safety, AI Now, Anthropic, and OpenAI can inform threat models, but it should not be treated as certification of a specific deployment.
We would prioritize controls that can be tested and operated: restricted tool permissions, human approval for consequential actions, evaluation against realistic failure cases, and an incident process. Broad principles are valuable only when they change those engineering decisions.
How We Would Implement It
- Inventory and classify: Record each AI use case, its purpose, users, data, models, vendors, jurisdictions, and possible harms. Obtain legal review where regulatory classification is uncertain.
- Set requirements: Map applicable legal duties separately from voluntary NIST and OECD-aligned controls. Assign an owner and evidence artifact to each requirement.
- Build bounded architecture: Put model access behind a service layer; use least-privilege credentials, data minimization, tool allowlists, and approval gates for consequential actions.
- Evaluate before release: Test task performance, harmful outputs, prompt injection, data leakage, and misuse of connected tools using scenarios drawn from the actual workflow.
- Operate and revisit: Log decisions and tool calls with appropriate access controls, monitor failures, define rollback triggers, and reassess when models, prompts, data, or regulations change.
Risks, Costs and Security
The main costs are ongoing: testing, review, logging, incident response, and revalidation after changes. Excessive review can make low-risk workflows impractical, while weak oversight can leave high-impact actions unchecked; controls should scale with the potential harm.
Security testing must cover the whole application, especially retrieved content and connected tools. Procurement should also address data-use terms, portability, exit plans, and provider concentration. Macroeconomic forecasts warrant scrutiny, but an enterprise’s immediate exposure is usually more concrete: an application it cannot audit, secure, or move when conditions change [1].
Where Kimbodo Comes In
Kimbodo builds and operates this in production for businesses — see our AI Cost & Governance practice, or Analyze My AI Costs.
