What Happened
Enterprise AI infrastructure is moving from simple chatbot deployment to agentic systems that use tools, execute code, test applications, interact with browsers and potentially contact people. Recent examples show this shift clearly: Datasette Agent added mechanisms for tools to run JavaScript in the user’s browser and for agents to invisibly open and test generated apps using JavaScript [2][6].
At the same time, open-weight models are becoming a serious architectural option. Industry discussion around models such as Kimi K3 focused on the point that open-weight systems can now compete with proprietary frontier models in some areas, changing the default assumption that enterprise AI must be routed only through closed APIs [4].
Agentic workloads are also becoming more expensive and more security-sensitive. One reported example described Anthropic using Claude to discover cryptographic weaknesses, with roughly $100,000 spent on tokens for the task [3]. That is not a normal chatbot cost profile; it is closer to a cloud batch-compute workload with open-ended spend risk.
A separate lesson is social and operational: when employees connect AI assistants to collaboration tools like Slack, coworkers may react negatively if the assistant contacts them directly, even if they would have accepted the same request from a human colleague [1]. Production AI platforms therefore need not only model orchestration, but also interaction policy, permissions and human-facing etiquette.
Why It Matters to Businesses
The practical implication is that AI platforms can no longer be designed as thin wrappers around a single LLM API. Businesses need infrastructure that supports model choice, workload routing, cost controls, tool governance, auditability and safe user interaction.
- Model strategy is now a portfolio decision. Open-weight models may reduce unit cost, improve data control and enable private deployment, while proprietary frontier models may still win on quality, latency-to-capability and managed operations [4].
- Agent tools increase business value and risk together. Browser execution, app testing and code-running tools make agents more useful, but also expand the attack surface and failure modes [2][6].
- Inference cost can become cloud-scale spend. Long-running research, security analysis or autonomous agent loops can consume material budget quickly, as shown by high-token cryptographic analysis workloads [3].
- Human workflow integration is a product requirement. AI systems that message coworkers, update tickets or trigger approvals need explicit social and permission boundaries, not just API credentials [1].
- MLOps and DevOps are converging. Evaluation, deployment, observability, incident response, secrets management and cost allocation must be designed as one operating model.
Kimbodo Engineering Perspective
The central architecture decision is whether to treat LLMs as an application feature or as shared enterprise infrastructure. For most businesses building multiple AI-powered workflows, the infrastructure approach is safer and cheaper over time.
Do not hard-code the model layer
We would avoid building business logic directly against one provider’s chat API. Instead, use a model gateway that supports proprietary APIs, open-weight models, local inference endpoints and fallback routing. This creates leverage when pricing, latency, accuracy or compliance requirements change.
The trade-off is added platform complexity. A gateway introduces abstraction, testing overhead and operational responsibility. But for organizations with multiple AI use cases, the benefits outweigh the cost because teams can reuse authentication, logging, caching, policy and evaluation infrastructure.
Treat agents as distributed systems
Agents that browse, run scripts, call internal systems or test applications are not just prompts. They are distributed systems with uncertain execution paths. Features like browser-side JavaScript execution and automated app testing are useful, but they need sandboxing, trace capture and deterministic replay where possible [2][6].
The most common production mistake is allowing an agent to gain tool access before the team has defined tool contracts. Every tool should specify input schema, permission scope, timeout, retry behavior, allowed data classes, audit logging and expected failure modes.
Separate capability from permission
A model may be capable of writing a Slack message, creating a pull request or querying customer records. That does not mean it should be allowed to do so autonomously. The Slack example shows that technically valid behavior can still violate workplace expectations [1].
We prefer progressive autonomy: draft first, then suggest, then execute with approval, then execute automatically only for low-risk, reversible actions. This is especially important for systems that interact with employees, customers, production data or financial workflows.
Cost controls must be part of orchestration
Token spend is not a billing detail; it is an architectural constraint. A task that spends six figures on inference may be justified for high-value research or security work, but the platform must make that decision visible and controllable [3].
Budgets should be enforced at the tenant, application, user, workflow and agent-run level. Without that, autonomous loops can turn small product defects into large cloud bills.
How We Would Implement It
Reference architecture
For an enterprise AI platform, we would implement a layered architecture:
- AI control plane: central service for model routing, policy enforcement, budgets, identity, audit logs and evaluation configuration.
- Model gateway: unified interface for proprietary LLM APIs, open-weight inference endpoints, embedding models, rerankers and specialist models.
- Inference data plane: Kubernetes or managed container infrastructure for open-weight model serving, autoscaling, GPU scheduling and batch workloads.
- Tool execution layer: isolated workers for browser automation, code execution, document processing, database access and SaaS integrations.
- Retrieval layer: governed access to vector indexes, relational data, object storage and enterprise search.
- Evaluation and observability layer: prompt/version tracking, test sets, trace capture, cost metrics, latency metrics, hallucination checks and human feedback.
- Approval and workflow layer: human-in-the-loop controls for sensitive actions such as messaging users, modifying records or deploying code.
Model deployment choices
We would classify workloads into three routing categories:
- Commodity workloads: summarization, classification, extraction and rewriting. These are candidates for smaller open-weight models, batch inference and aggressive caching.
- High-reasoning workloads: planning, code analysis, complex troubleshooting and research. These may justify frontier proprietary models or larger open-weight deployments.
- Regulated or sensitive workloads: data subject to contractual, legal or residency controls. These may require private cloud, VPC-hosted inference or self-hosted open weights.
This prevents expensive models from being used by default while preserving access to stronger models when the task requires them.
Cloud orchestration
For production deployment, we would use Kubernetes or a managed equivalent for service orchestration, with separate node pools for CPU services, GPU inference, batch jobs and sandboxed tool execution. GPU workloads should use queueing and autoscaling rather than always-on capacity unless utilization is predictable.
For open-weight models, we would evaluate vLLM, TensorRT-LLM or cloud-native managed inference depending on throughput, latency and operations capacity. The decision should be made through load testing, not vendor preference.
Agent safety implementation
Agent actions should pass through a policy engine before execution. The policy engine should evaluate user identity, data sensitivity, tool risk, environment, budget remaining and whether human approval is required.
- Read-only tools: lower risk, but still logged and scoped.
- Write tools: require stronger permissions, dry-run support and rollback strategy.
- Communication tools: require explicit user approval unless the message type is pre-approved and low risk.
- Browser and code tools: run in isolated sandboxes with network restrictions, timeouts and artifact capture.
This is especially important as agents gain browser execution and app-testing abilities similar to the capabilities described in Datasette Agent [2][6].
Evaluation pipeline
Before production release, each AI workflow should have:
- Golden test cases for expected answers and tool usage.
- Adversarial tests for prompt injection, data leakage and unsafe tool calls.
- Regression tests for model upgrades and prompt changes.
- Cost benchmarks per task type.
- Latency service-level objectives by user journey.
- Human review queues for uncertain or high-impact outputs.
The key is to test the whole system, not just the model response. A model that answers well in isolation may still fail when connected to tools, browsers, databases or human workflows.
Risks, Costs and Security
Primary risks
- Uncontrolled inference spend: autonomous loops, long contexts and repeated retries can create large bills quickly [3].
- Tool misuse: agents may call the right tool at the wrong time, with the wrong scope or without sufficient user approval.
- Data exposure: prompts, retrieved documents, traces and tool outputs may contain regulated or confidential information.
- Supply-chain risk: open-weight models, plugins, browser tools and generated code must be scanned and governed.
- Human trust failure: AI systems that act too independently in workplace channels can reduce adoption even when technically functional [1].
Cost controls
We would implement hard and soft limits across the platform: per-request token caps, per-run budgets, per-user daily budgets, tenant-level monthly budgets, queue limits for expensive models and alerts for abnormal spend. Expensive workflows should require explicit approval or run in batch with budget preallocation.
Caching should be used carefully. It works well for deterministic extraction, embedding reuse and repeated document analysis. It is less reliable for personalized reasoning or workflows where freshness is critical.
Security controls
Security should include least-privilege service accounts, secrets isolation, encrypted prompt and trace storage, private networking for sensitive data paths, sandboxed code execution, egress controls and full audit logging. For regulated environments, traces should support retention policies, redaction and legal hold.
For open-weight deployments, security review should cover model provenance, license terms, dependency scanning, container hardening and access to model artifacts. For proprietary APIs, review should cover data retention, training use, regional processing, contractual controls and incident response obligations.
Bottom line
The winning enterprise AI architecture is not the one with the most capable model. It is the one that can safely route work across models, control tool execution, contain cost, respect human workflows and produce auditable outcomes. As agents gain more autonomy and open-weight models become more competitive, those platform capabilities become the difference between a useful AI system and an operational liability.
Where Kimbodo Comes In
Kimbodo builds and operates this in production for businesses — see our AI Infrastructure & MLOps practice. Wondering what it would cost for your organization? Get a preliminary range, timeline and architecture in about a minute.
