How to Build Production AI Platforms With Reliable Agents and Predictable Costs

What Happened

Recent production examples point to a consistent architecture lesson: separate agent reasoning from execution, and govern both with measurable quality and resource limits. A working model endpoint is not sufficient evidence that an AI application is reliable.

AWS SMGS Operations reported that pre-launch testing of a multi-agent analytics system caught confident errors that uptime monitoring would miss—including presenting $410.08 as “$410.08 Million.” Its continuous evaluation process raised the accuracy pass rate from 93% to a stable 96%, reduced pure hallucinations from 138 to fewer than 52 per 10,000 responses, and brought issue detection below four hours. These are results from that deployment, not general guarantees. [2]

A biotech architecture illustrates a complementary infrastructure pattern: separate cognitive orchestration, long-running GPU execution, and enterprise governance. Agents coordinate work while specialized models execute on GKE; large scientific artifacts remain in storage rather than passing through conversational context. [3]

Cost controls are becoming more explicit, too. BigQuery introduced generally available daily input and output token quotas for its generative AI SQL functions, excluding embedding functions. These controls can constrain consumption, but require workload-specific configuration to become meaningful budget protections. [5]

Why It Matters to Businesses

AI platforms have several independent failure modes: an available service can produce incorrect financial answers; a plausible plan can omit required approvals; and a valid workflow can consume an unacceptable amount of compute. Each needs a different control. [2][4]

• Reliability requires business-level checks. Test numerical units, evidence, permissions, and task completion—not just response availability.
• Compute placement affects economics and data exposure. Private execution can keep proprietary workloads within an enterprise-controlled environment, but adds capacity planning and operational responsibility. [3]
• Budgets must cover the entire workflow. Token consumption, GPU jobs, tool calls, retries, and evaluation all contribute to operating cost.

The management implication is straightforward: fund evaluation, observability, and execution controls as platform capabilities, rather than treating them as post-launch improvements.

Kimbodo Engineering Perspective

Use agents to select and coordinate work; use explicit services to execute consequential operations. Long-running GPU tasks should not depend on an open conversational request. Submit them as durable jobs with bounded inputs, status reporting, cancellation, and independently enforced permissions.

For compute-heavy pipelines, backend-aware concurrency, fewer network hops, and milestone updates can matter more than faster token generation. Warm GPU capacity and preloaded weights improve responsiveness, but create idle-capacity costs; they are justified only when latency targets and utilization support them. [3]

Evaluation must also inspect what the agent actually did. In an 11-prompt plugin evaluation, an LLM judge classified three prompts as failures even though tool-call logs showed they had passed. Final-answer scoring alone can therefore misrepresent execution quality. Preserve tool trajectories and turn structure, and combine automated scoring with deterministic checks and targeted expert review. [4]

The same principle applies to AI-assisted development: ask which failure could occur and which check would catch it, rather than treating human review as a universal quality guarantee. Continuous automated review can handle routine issues while people examine architecture, authorization, and business consequences. [1]

How We Would Implement It

Separate orchestration, execution, and governance

• Orchestration: Use an agent runtime for planning and routing, backed by durable workflow state. Set limits on steps, retries, elapsed time, and concurrent jobs.
• Execution: Expose typed tool interfaces over asynchronous workers. Use managed endpoints where they meet data and performance requirements; use private GPU infrastructure when proprietary workloads or specialized computation justify it.
• Artifacts: Store large outputs in object storage and pass scoped, short-lived references plus summaries. The biotech pattern uses signed Cloud Storage URIs instead of moving large scientific files through the agent context. [3]
• Governance: Assign workload identities, enforce authorization at tool boundaries, and require approval before expensive or irreversible actions.

Build release gates around observed behavior

Create a version-controlled golden set covering normal tasks, numerical correctness, denied permissions, tool failures, and destructive-action attempts. Run a small sentinel suite on changes and a broader suite before release. AWS’s deployment uses a 15-case CI sentinel alongside a full golden-set gate; suite size should reflect the application’s risk and coverage needs. [2]

Run live and plan-only evaluations together. Live tests expose API and permission behavior; plan-only tests can reveal unsafe intentions hidden by blocked execution. Capture parent-child relationships, parallel calls, tool arguments, timing, and outcomes without flattening turn structure. [4]

Enforce consumption limits before scaling

For BigQuery AI workloads, begin with per-user quotas, measure a representative sample, and constrain output tokens where supported. AI.COUNT_TOKENS provides only a lower-bound estimate: it excludes output and thinking tokens, AI-function instructions, and non-text inputs. Output quotas include thinking tokens. [5]

Connect application-level budgets to admission control, and monitor quality, latency, and cost together. Feed confirmed production failures back into regression tests. [2]

Risks, Costs and Security

• Quota exhaustion affects availability. BigQuery rejects queries when a required quota is exhausted and stops new LLM calls if exhaustion occurs mid-query. Applications need explicit failure handling rather than interpreting incomplete work as success. [5]
• Evaluations have overhead. Broad suites and online judges consume tokens and time. Use risk-based sampling and deterministic validation where possible; do not treat a judge model as ground truth. [4]
• Private infrastructure is not automatic security. Network isolation still needs scoped identities, protected artifact access, audited tool calls, and careful handling of sensitive traces.
• Autonomy needs financial and operational boundaries. Permit reversible work within defined limits, but require approval for costly computation and consequential downstream actions. The biotech design explicitly reserves expensive docking runs and advancement to wet-lab synthesis for human approval. [3]

The production objective is controlled delegation: agents can coordinate work, while independently enforced controls determine what they may execute, spend, and release.

Sources

1. [1] Quoting Addy Osmani, Morgan Westlee Lunt, and Qing Wang
2. [2] Building trust in AI agents: A production-ready evaluation framework with Amazon Bedrock AgentCore
3. [3] Architecting autonomy: A design pattern for high-compute, governed agents in biotech
4. [4] Four evaluation engineering lessons learned from building agent plugins
5. [5] Predictable AI spend in BigQuery: Introducing cost management for generative AI functions