How to Build Production AI Infrastructure for Agents, RAG and LLM Inference Without Losing Cost Control

What Happened

Enterprise AI infrastructure is moving from model hosting toward governed orchestration of agents, tools, retrieval systems, identity, telemetry and specialized compute. Recent platform updates show a clear pattern: production AI systems now need a control plane for discovery and governance, a secure runtime for agent execution, managed retrieval for enterprise data, and workload-specific compute choices.

AWS is expanding its agentic stack around AgentCore Runtime, AgentCore Gateway, Amazon Bedrock Managed Knowledge Bases, Amazon Quick integrations and AWS Agent Registry. AgentCore Runtime can host MCP servers with session isolation, extended execution, persistent storage, authentication, observability, enhanced payloads, bidirectional streaming and evaluations. Amazon Quick can connect to those MCP servers through AgentCore Gateway, which acts as a secure TLS bridge and requires both inbound user authentication and outbound machine-to-machine OAuth credentials [1].

AWS Agent Registry addresses agent sprawl by creating a searchable, governed catalog for agents, MCP servers, skills and custom resources. It separates a Governance Plane, which stores compliance signals, discovery policies and metadata schemas, from a Discovery Plane, which exposes approved records through semantic and lexical search. It also supports lifecycle states, human review, CloudTrail audit trails, IAM, OAuth and PrivateLink [3].

Amazon Bedrock Managed Knowledge Bases are being positioned as managed infrastructure for enterprise RAG and agentic retrieval. New reference architectures use AgentCore Gateway to expose managed knowledge bases as MCP tools, with AgentCore Runtime routing queries across multiple knowledge bases and Managed Knowledge Bases performing decomposition, iterative retrieval and cited synthesis [4]. Another pattern supports multi-tenant chat applications where identity is derived from a verified token and used to enforce per-user or per-tenant retrieval filters on every call [5].

At the infrastructure layer, AWS continues to emphasize workload-specific architecture, including Inferentia, Trainium and NVIDIA GPU instances, while Forrester highlighted AWS for strategy, elastic inference and SageMaker integration [2]. Google Cloud is moving similarly on infrastructure breadth: GKE Agent Sandbox is generally available, Agent Substrate is open source, Managed Lustre supports up to 8 PB, GKE Dataplane V2 scales to 15,000 nodes, C4N VMs provide up to 400 Gbps networking, and confidential accelerator instances are expanding security options for GPU workloads [6].

Why It Matters to Businesses

AI applications are becoming distributed systems

The first generation of enterprise AI projects often wrapped a single model call with a prompt. Current systems coordinate models, tools, document stores, access controls, queues, workflow state, telemetry and cost policies. MCP makes tool access more standard, but it also creates a new integration and security surface. A model can now call APIs, update records, retrieve documents and invoke long-running workflows, so the infrastructure must enforce identity, authorization, rate limits, audit trails and runtime isolation [1].

Agent sprawl is becoming a governance problem

As different teams build agents, tools and skills, businesses risk duplicate functionality, unapproved data access and no central inventory. A registry pattern is emerging as a practical requirement: teams need to know which agents exist, who owns them, which tools they can call, what data they access, and whether they are approved for production use. AWS Agent Registry’s record lifecycle and approval workflow reflect a broader enterprise need, not just a product feature [3].

Managed retrieval reduces platform burden, but not design responsibility

Managed Knowledge Bases can remove vector database operations, embedding pipeline maintenance, chunk indexing, ranking infrastructure and scaling work. However, application teams still own identity, tenancy, document lifecycle, retrieval filtering, conversation persistence, user experience and evaluation. Bedrock does not store chat turns in the multi-tenant pattern, so conversation history must be persisted separately [5].

Compute selection is now a financial architecture decision

Model serving cost depends on workload shape: latency target, batchability, context length, token throughput, model size, accelerator availability and utilization. AWS’s support for Inferentia, Trainium and NVIDIA instances reflects the need to match compute to workload economics [2]. Google’s emphasis on high-bandwidth networking, managed parallel file systems, TPU telemetry and GKE inference optimization shows the same trend: AI cost control depends on infrastructure fit, not just model choice [6].

Kimbodo Engineering Perspective

Do not start with agents; start with control boundaries

In production, the main question is not whether an agent can complete a task in a demo. The main question is what the agent is allowed to know, call, change and remember. For enterprise systems, we would define:

Identity boundary: user, tenant, service account and workload identity must be explicit and verifiable.
Tool boundary: every MCP tool needs an owner, schema, permission model, timeout, retry policy and audit trail.
Data boundary: retrieval must enforce access control at query time, not only at ingestion time.
Runtime boundary: long-running or stateful agent sessions need isolation, quotas and cleanup policies.
Cost boundary: token, retrieval, embedding and accelerator spend must be attributable by tenant, feature and workflow.

Managed platforms accelerate delivery, but increase coupling

Using AgentCore Runtime, AgentCore Gateway and Managed Knowledge Bases can reduce engineering time significantly. Teams avoid building custom MCP hosting, vector ingestion, reranking, streaming retrieval, observability plumbing and autoscaling. The trade-off is architectural coupling to service-specific APIs, quotas, authentication flows and runtime behavior. For some businesses, that is the right trade: faster production readiness and fewer platform engineers. For others, especially those with strict portability requirements, a hybrid design with open protocols, containerized runtimes and external metadata catalogs may be safer.

MCP is useful, but should be treated like an API gateway pattern

MCP standardizes how models discover and call tools, but it does not remove the need for API governance. Each MCP server should be treated as a production integration surface. The Amazon Quick pattern illustrates this clearly: inbound user JWT validation and outbound OAuth service credentials are both required, with IAM permissions for runtime invocation [1]. That dual-auth model is a good production baseline because the system must authenticate both the human request and the machine action.

Retrieval quality needs operational telemetry, not occasional testing

Agentic retrieval introduces routing, decomposition, iterative search, citation synthesis and tool selection. Failures may come from the query planner, retriever, ranker, access filter, context builder or generator. The Bedrock reference architecture emits metrics for knowledge bases, ingestion, retrieval quality, Gateway/MCP behavior, OpenTelemetry spans, token usage and evaluation scores into CloudWatch and X-Ray [4]. This is the right direction: retrieval systems should be measured continuously, not validated once during launch.

How We Would Implement It

1. Define the enterprise AI control plane

We would implement a central control plane before scaling agent development. Its responsibilities would include:

Cataloging agents, MCP servers, skills, tools, models, prompts and knowledge bases.
Capturing owners, environments, data classifications, dependencies, versions and approval state.
Integrating with CI/CD so promotion to production requires metadata, tests, security review and observability configuration.
Exposing a discovery API for developers, IDEs and approved runtime agents.

On AWS, Agent Registry is a strong fit for this role because it already supports governed publishing, curation, approved discovery, semantic search, CloudTrail auditability and MCP exposure for native agent discovery [3]. For multi-cloud estates, we would wrap it with an internal platform API or synchronize records into an enterprise service catalog to avoid losing visibility outside one cloud.

2. Use a secure agent runtime and gateway layer

We would run production agents in isolated containers or managed agent runtimes, with a gateway mediating all tool calls. On AWS, AgentCore Runtime and AgentCore Gateway provide a managed path for MCP hosting, TLS bridging, authentication, streaming and observability [1].

The baseline runtime design would include:

Short-lived credentials for tool access.
Inbound user authentication through an identity provider such as Cognito, Okta, Entra ID or another OIDC provider.
Outbound OAuth or workload identity for machine-to-machine access.
Per-tool authorization checks based on user, tenant, role, data sensitivity and action type.
Network isolation through VPC placement, private endpoints and egress controls where supported.
Structured logs, traces and metrics for every tool call and model invocation.

3. Build retrieval as a governed subsystem

For enterprise RAG, we would not let each product team build its own vector pipeline unless there is a clear reason. A managed knowledge base is usually preferable when the business needs faster delivery, consistent ingestion, integrated ranking and lower operational burden.

For Bedrock Managed Knowledge Bases, we would use two patterns depending on tenancy:

High-isolation tenants: provision separate knowledge bases per tenant or regulated business unit.
Shared infrastructure tenants: use a shared knowledge base with server-enforced metadata filters derived from a verified identity token.

The multi-tenant pattern should never trust client-supplied tenant IDs. The server should derive user_id or tenant_id from the verified JWT, apply that value as a retrieval filter on every call, and optionally discard returned chunks if metadata does not match [5]. For asynchronous ingestion, we would follow the staged upload pattern: API Gateway and Lambda validate requests, S3 stores uploads, SQS decouples ingestion, workers batch documents into ingestion calls, and DynamoDB tracks document status [5].

4. Instrument agent workflows end to end

We would instrument every AI workflow with OpenTelemetry-compatible tracing and business-level metrics. Required signals include:

Model name, version, latency, input tokens, output tokens and cost estimate.
Tool call name, arguments hash, authorization result, latency, status and retry count.
Retrieval query, filters applied, documents returned, citation coverage and reranking score where available.
Agent loop count, stop reason, timeout reason and escalation path.
Evaluation scores for faithfulness, correctness, relevance and tool selection.

The Bedrock agentic retrieval reference architecture’s use of CloudWatch, X-Ray, OpenTelemetry spans, token metrics and continuous sampled evaluation is a practical model [4]. We would add cost allocation tags and tenant-level dashboards so finance, product and engineering teams can see which features are driving spend.

5. Select compute by workload class

We would separate workloads into at least four classes:

Interactive inference: latency-sensitive chat, copilots and agents. Optimize for time to first token, autoscaling speed and cache efficiency.
Batch inference: document processing, enrichment and evaluation. Optimize for throughput and accelerator utilization.
Fine-tuning or training: optimize for distributed networking, storage bandwidth and checkpoint performance.
Retrieval and ingestion: optimize for document parsing, embedding throughput, index freshness and storage cost.

On AWS, this means benchmarking NVIDIA GPU instances against Inferentia or Trainium where supported, rather than assuming one accelerator family for all workloads [2]. On Google Cloud, high-throughput GKE networking, Managed Lustre, Filestore options, TPU telemetry and C4N networking may matter for large-scale training or high-volume serving [6]. The correct architecture may be single-cloud for operational simplicity or multi-cloud for capacity, commercial or specialization reasons.

Risks, Costs and Security

Key risks

Tool misuse: Agents can call external systems incorrectly or with excessive authority. Use least privilege, allowlists, approval gates and action-level policies.
Tenant data leakage: Retrieval filters must be server-generated from verified identity, not accepted from the browser or client application [5].
Agent sprawl: Without a registry, teams duplicate agents, lose ownership visibility and bypass governance. A governed catalog reduces this risk [3].
Opaque retrieval failures: Bad answers may come from routing, missing documents, stale indexes, ranker behavior or generation. Add traces and evaluation metrics [4].
Quota and concurrency limits: Managed services simplify operations but introduce service limits. Bedrock Managed Knowledge Base retrieval bursts and sustained QPS should be tested under expected load [5].
Vendor lock-in: Managed runtimes, knowledge bases and registries improve speed but can make migration harder. Use open protocols, portable metadata and containerized components where feasible.

Cost drivers

Generation tokens: Agent loops, long contexts and multi-step reasoning can multiply token spend.
Retrieval calls: Agentic retrieval may perform multiple searches per user query, especially across multiple knowledge bases.
Embedding and indexing: Ingestion may not carry an extra managed knowledge base charge in some patterns, but embedding model usage, storage and supporting services still cost money [5].
Accelerator underutilization: GPU or TPU capacity is expensive when idle. Batch scheduling, autoscaling and right-sized instance selection matter.
Observability volume: Traces, logs and evaluation data are essential, but high-cardinality telemetry can become a material cost if unmanaged.

Security baseline

For production AI platforms, we would require verified identity, workload-level credentials, private networking where practical, encrypted storage, audit logging, data classification, prompt and tool input validation, output filtering for sensitive data, and incident response procedures specific to AI workflows.

The strongest production pattern is layered control: identity provider for users, IAM or workload identity for services, gateway enforcement for tools, registry governance for discovery, metadata or ACL enforcement for retrieval, and continuous telemetry for operations. The goal is not only to make agents work, but to make their actions understandable, reversible, attributable and cost-controlled.

Where Kimbodo Comes In

Kimbodo builds and operates this in production for businesses — see our AI Infrastructure & MLOps practice, or Estimate My Infrastructure.