How to Build Production AI Agents Without Losing Control of Cost, Security, and Operations

What Happened

Recent enterprise AI platform patterns show a clear shift: production teams are moving beyond standalone chatbots toward orchestrated agent systems with managed runtime isolation, governed identity, task-specific models, deterministic business rules, and cloud-native observability.

Amazon Bedrock AgentCore is being used as managed infrastructure for agent execution, including serverless code-interpreter sandboxes that run in ephemeral MicroVMs with OS-level session isolation, configurable VPC or internet access, preloaded Python and Node runtimes, file handling, and CloudWatch and CloudTrail integration [1]. Abnormal AI uses this pattern for inline email threat detection at very high scale, reserving expensive agentic analysis for the hardest cases after cheaper heuristic and ML tiers have filtered the majority of traffic [1].

Agent identity and consent are also becoming managed platform capabilities. AgentCore Identity now provides a hosted OAuth consent portal for agents that need access to tools such as GitHub and Slack, binding user sessions, storing tokens in a managed vault, refreshing access tokens where possible, and logging consent operations through CloudTrail [2].

Industry-specific AI platforms are adopting multi-agent architectures. Ninth Wave built an open-finance onboarding assistant on Bedrock AgentCore with a primary routing agent and specialist agents for search, document Q&A, classification, field mapping, analysis, interactive workflows, and readiness scoring. The platform uses tenant-scoped OpenSearch indices and S3 prefixes, OAuth/OIDC, MFA, Secrets Manager, customer-managed KMS keys, least-privilege IAM, CloudWatch, Grafana, and ECS Fargate deployment pipelines [3].

At the model layer, the practical guidance is to start with foundation models and prompt engineering, then escalate to RAG, context optimization, caching, distillation, fine-tuning, reinforcement or preference tuning, continued pre-training, or custom model training only when specific failure signals justify the added cost and operational burden [4].

Agent orchestration is also expanding into business-process automation. A replenishment workflow combines Databricks forecasting, a Genie Agent via Model Context Protocol, Amazon Quick orchestration, S3 Tables supplier availability, and an OpenAPI order service. It forecasts demand, detects surges, selects suppliers using explicit business rules, places routine orders, and sends exceptions to human review [5].

Why It Matters to Businesses

The core business lesson is that AI capability is no longer just a model-selection decision. It is an infrastructure, security, workflow, data, and operating-model decision.

Cost control depends on tiering. Abnormal AI’s three-tier detection pipeline uses cheap heuristics and small models for billions of events, deeper ML for millions, and agentic Code Interpreter sessions only for tens of thousands of difficult cases [1]. This is the right economic pattern for most enterprise AI systems.
Security boundaries must follow the agent. Agents that can call tools, run code, read files, or use user OAuth tokens need sandboxing, least privilege, token isolation, audit trails, and explicit egress controls [1][2].
Multi-agent design is useful when tasks are genuinely different. Ninth Wave’s architecture separates search, classification, mapping, Q&A, workflow, and readiness scoring because each task has different latency, grounding, reasoning, and governance requirements [3].
Deterministic rules still matter. Ninth Wave computes readiness scores deterministically from field-mapping coverage, while the replenishment workflow auto-orders only when one supplier can cover demand and escalates exceptions to humans [3][5]. This reduces uncontrolled autonomy.
Customization should be incremental. Many workloads should stop at prompting, structured outputs, RAG, caching, or distillation. Fine-tuning and continued pre-training require stronger evidence, more data, and a higher operating budget [4].
The bottleneck is shifting from coding to product judgment and operations. As Laurie Voss argues, the cost of writing code is collapsing, while the remaining hard work is discovering what users need, defining it precisely, and operating the resulting software well [6].

Kimbodo Engineering Perspective

For production AI applications, we would treat the agent runtime as critical infrastructure, not as an application feature. The main architectural question is not “which model is best?” but “where should probabilistic reasoning be allowed, under what controls, with what fallback, and at what unit cost?”

Use agents selectively, not everywhere

Agents are valuable when a workflow requires planning, tool use, file inspection, code execution, or multi-step reasoning. They are usually the wrong first choice for high-volume, simple, repeatable classification. Abnormal AI’s architecture is a strong pattern: use deterministic rules and smaller models to reduce the workload, then use agents only where deeper analysis has business value [1].

Prefer managed runtimes when isolation and auditability matter

Managed code-interpreter sandboxes reduce the burden of building session isolation, runtime provisioning, log integration, file handling, and teardown logic [1]. The trade-off is cloud-provider coupling and less low-level control. For regulated or high-volume environments, that trade-off is often acceptable if the managed service provides isolation, VPC controls, no-egress options, audit logs, and predictable lifecycle limits.

Separate orchestration from reasoning

The orchestrator should enforce state transitions, permissions, budgets, and business rules. The model or agent should perform bounded reasoning inside those constraints. The replenishment example demonstrates this well: forecasting and detection are model-assisted, but supplier selection and human escalation are governed by explicit rules [5].

Do not fine-tune to solve architecture problems

Fine-tuning is often overused. If answers are stale, use RAG. If latency is poor, optimize retrieval, chunking, caching, or model size. If the large model is too expensive, consider distillation. Escalate to PEFT, reinforcement tuning, continued pre-training, or custom models only when the failure mode is model behavior that cannot be solved by grounding or orchestration [4].

Design for tenant isolation from the start

Per-tenant OpenSearch indices, S3 prefixes, scoped IAM roles, encryption boundaries, and audit records are easier to build early than retrofit later. Ninth Wave’s design is a useful reference for regulated multi-tenant AI platforms because it combines tenant-scoped grounding with conventional cloud security controls [3].

How We Would Implement It

1. Define the autonomy boundary

We would classify each workflow step into one of four modes:

Deterministic: rules, validation, scoring, routing, policy checks, and financial calculations.
Model-assisted: summarization, extraction, classification, mapping, semantic search, and recommendations.
Agentic: multi-step tool use, code execution, investigation, or file-based analysis.
Human approval: irreversible actions, high-value transactions, policy exceptions, or low-confidence outputs.

2. Build a tiered execution path

For high-volume systems, we would implement a pipeline similar to Abnormal AI’s pattern [1]:

Tier 1: heuristics, rules, cached lookups, small classifiers, and cheap embeddings.
Tier 2: stronger ML models, RAG, reranking, structured extraction, and domain-specific validation.
Tier 3: agentic runtime with tool access, code execution, scratch storage, and stricter budget controls.
Feedback loop: analyst or user corrections feed evaluation sets, regression tests, prompts, retrieval improvements, and model updates.

3. Use a controlled agent runtime

For code-capable agents, we would use isolated ephemeral sandboxes with short default TTLs, no-egress mode where possible, controlled VPC access where necessary, file-size limits, checkpointing for long-running jobs, and CloudWatch and CloudTrail integration [1]. For self-hosted alternatives, we would require equivalent controls: container or MicroVM isolation, per-session credentials, network policies, filesystem quotas, execution timeouts, artifact retention policies, and full audit logging.

4. Implement identity and tool consent as a platform service

Agents that act on behalf of users need explicit consent, scoped tokens, refresh handling, and revocation paths. A managed consent portal can reduce custom OAuth callback and session-binding code, while centralizing token storage and audit events [2]. We would still wrap this with enterprise controls: corporate IdP enforcement, MFA, conditional access, least-privilege scopes, short-lived workload tokens, and alerting on unusual tool access.

5. Choose the model customization level empirically

We would start with a baseline model and structured prompting, then add RAG and retrieval evaluation. Only after measuring failure modes would we escalate [4]:

Prompting: first choice for formatting, instruction following, and workflow-specific behavior.
RAG: first choice for changing factual knowledge and enterprise documents.
Caching and context optimization: first choice for repeated prompts, latency reduction, and cost control.
Distillation: strong option when a large model works but is too slow or expensive.
Fine-tuning: appropriate when enough labeled examples exist and behavior must be changed consistently.
Continued pre-training or custom models: reserved for very large data volumes, specialized domains, and organizations that can operate a model lifecycle.

6. Design the data plane for multi-tenancy

For enterprise platforms, we would use tenant-scoped storage, retrieval, and authorization boundaries: separate S3 prefixes or buckets, tenant-filtered metadata, isolated or logically partitioned vector indices, customer-managed KMS keys where required, and strict service-role separation. Ninth Wave’s use of per-tenant OpenSearch indices and S3 prefixes is a practical pattern for regulated onboarding workflows [3].

7. Add verifiers and deterministic gates

Every agent output that triggers action should pass through programmatic verification. Examples include schema validation, policy checks, retrieval citation checks, price and inventory rules, readiness-score formulas, confidence thresholds, and duplicate-action prevention. The replenishment workflow’s decision rule, where orders are automated only if a single supplier can cover demand, is the type of guardrail we would prefer for business automation [5].

8. Operate it like production software

We would deploy through CI/CD with automated tests, prompt and retrieval regression suites, model-version pinning, canary releases, rollback, telemetry, and cost dashboards. Observability should include model latency, token consumption, tool-call count, sandbox duration, retrieval hit rate, refusal rate, human-escalation rate, task success rate, and business KPIs.

Risks, Costs and Security

Cost risks

Agent sprawl: Multi-agent systems can multiply model calls, tool calls, and latency. Use routing, budgets, and task-specific models.
Overuse of large models: Serving a large teacher model directly can be unnecessarily expensive; distillation may reduce cost and latency with acceptable accuracy loss [4].
Unbounded code execution: Long-running interpreter sessions need TTLs, quotas, filesystem limits, and job cancellation.
Managed platform subscriptions: Orchestration and BI-style automation platforms can introduce recurring enterprise-license costs, as seen with Amazon Quick in the replenishment example [5].

Security risks

Data exfiltration: Code-capable agents should use no-egress sandboxes unless external access is explicitly required [1].
Token misuse: OAuth tokens for tools such as GitHub or Slack should be stored in a managed vault or equivalent secure store, scoped narrowly, refreshed safely, and audited [2].
Tenant leakage: Retrieval systems must enforce tenant isolation at both the application and data layers [3].
Prompt injection and tool abuse: Agents should not be allowed to convert untrusted text into privileged actions without policy checks and explicit tool constraints.
Audit gaps: Consent, tool use, data access, sandbox execution, and administrative changes should produce searchable logs in systems such as CloudTrail and CloudWatch [1][2].

Operational risks

Non-determinism: Keep scoring, billing, eligibility, and irreversible business actions deterministic where possible.
Evaluation drift: Maintain regression suites using real failures, analyst feedback, and edge cases.
Vendor lock-in: Managed services accelerate delivery but increase dependency on provider-specific identity, orchestration, runtime, and observability APIs.
Human workflow failure: Human-in-the-loop steps need queues, SLAs, escalation paths, and auditability, not just a “review required” flag.

The practical path is to combine managed AI infrastructure with conservative system design: tier the workload, isolate execution, govern identity, ground responses, verify outputs, and reserve full autonomy for cases where the business value justifies the risk and cost.

Where Kimbodo Comes In

Kimbodo builds and operates this in production for businesses — see our AI Infrastructure & MLOps practice, or Estimate My Infrastructure.