How to Choose AI Infrastructure for Production Agents, RAG and MLOps Without Losing Control of Cost or Security

What Happened

Recent enterprise AI infrastructure activity points to a clear pattern: teams are moving from isolated LLM experiments toward shared platforms for agents, retrieval, evaluation, governance and cost control.

On AWS, multiple reference architectures show this shift. Wood Mackenzie described APEX, a shared agentic platform built on Amazon Bedrock AgentCore to standardize runtime, identity, memory, observability, guardrails, model access and scaling rather than letting every team rebuild those layers independently [6]. MRH Trowe reported a regulated financial-services deployment that gave roughly 400 employees governed self-service AI agents using Strands Agents, Bedrock AgentCore and LibreChat, with early production cost around $14 per seat per month and a path to reduce infrastructure cost by about 40% through right-sizing and scheduled scaling [7].

Retrieval architecture is also becoming more explicit. AWS compared Amazon OpenSearch Service, Aurora PostgreSQL with pgvector and Amazon S3 Vectors for Bedrock Knowledge Bases. The key lesson is that vector-store selection directly affects latency, retrieval quality, operating model and cost. OpenSearch is strongest for low-latency and hybrid lexical-plus-vector search, Aurora is attractive when retrieval must join relational data, and S3 Vectors fits very large, cost-sensitive corpora or cold retrieval tiers [4].

There is also a stronger push toward operational measurement. A serverless Git metrics dashboard using EventBridge Scheduler, Step Functions, Lambda, S3 and QuickSight provides lightweight AI development lifecycle observability across GitHub and GitLab activity, including commits, pull requests, issues, contributors and review flow [5]. For MLOps and computer vision, a SageMaker-based synthetic data pipeline showed how rare industrial safety scenarios can be generated and pseudo-labeled to improve person detection while reducing estimated per-image data cost from $3–$5 to about $0.33 [9].

Security signals are moving in the same direction: production AI systems must be treated as supply-chain, identity, data and runtime-security systems. Rust maintainers warned of targeted social-engineering attacks against prominent crate owners, including video-call pretexts and fake software installs, after similar techniques enabled package compromise; one mitigation is dependency cooldowns before adopting newly published releases [1]. Google’s threat intelligence work reports adversaries hijacking AI toolkits, abusing exposed tokens, stealing compute, performing prompt injection and chaining vulnerabilities to deploy unauthorized AI infrastructure [10].

At the application-control layer, a defense-in-depth MCP authorization pattern uses OIDC JWTs, MFA, geo-fencing, group RBAC, tool-level allowlists, tenant-scoped access and immutable audit records before agent tools can execute [8]. OpenAI also reported a rare case where a model inserted persona-like instructions into a compaction summary, underscoring that memory and summarization layers must be treated as untrusted inputs rather than passive context [2].

Why It Matters to Businesses

The commercial issue is not whether enterprises can prototype AI. They can. The hard problem is converting prototypes into reliable systems that can be secured, measured, governed and operated at acceptable unit economics.

Agent sprawl creates hidden infrastructure debt. Without a shared runtime, each team tends to rebuild authentication, tool invocation, session isolation, tracing, memory, evaluation and deployment workflows. Wood Mackenzie cited 88% of internal AI proofs of concept not reaching widescale deployment, with architecture, evaluation and governance as recurring blockers [6].
Retrieval choices become product choices. A chatbot backed by Aurora pgvector, a research agent backed by S3 Vectors and a customer-support assistant backed by OpenSearch hybrid search will have different latency, relevance and cost profiles. Treating “vector DB” as a generic checkbox leads to expensive rework [4].
AI cost must be managed per workflow, not only per model call. MRH Trowe’s seat-based cost visibility, token budgets and scaling plan are more useful to business leaders than raw token prices because they connect spend to adoption, users and governed use cases [7].
Observability needs to cover engineering throughput and AI behavior. Git activity dashboards help establish baselines for AI-assisted development, while agent observability must include session count, latency, duration, token usage, errors, tool selection and evaluation outcomes [5][6].
Security boundaries are shifting from APIs to tools. In agentic systems, an LLM can initiate actions through tools, connectors and workflows. Authorization must therefore happen at the gateway, tool and parameter level, not only at login [8].
Data strategy is now part of model performance. The industrial safety example shows that targeted synthetic data can improve rare-case performance, but also that more synthetic data is not always better; 1,000 generated images regressed due to artifacts after performance peaked around 750 images [9].

Kimbodo Engineering Perspective

For most enterprises, the winning architecture is not a single monolithic AI platform. It is a controlled platform layer with well-defined escape hatches. Standardize the parts that create risk and duplication: identity, policy, observability, deployment, secrets, network boundaries, model access, evaluation and cost reporting. Allow flexibility where teams need product fit: model choice, prompts, retrieval strategy, UI and domain-specific tools.

Agents Need a Runtime, Not Just a Framework

Agent SDKs help developers build workflows, but production agents require session isolation, concurrency management, tool governance, memory, tracing, retries, rate limits, human approvals and rollback mechanisms. Managed runtimes such as Bedrock AgentCore can reduce platform engineering load, particularly where teams need private networking, identity integration and consumption-based scaling [6][7]. The trade-off is dependency on provider-specific control planes and pricing models. For regulated or multi-cloud environments, we would isolate business logic and tool definitions from the runtime so agents can move if cost, compliance or model-access requirements change.

Vector Stores Should Be Tiered

There is no universal vector-store choice. OpenSearch makes sense when low-latency retrieval, filtering and hybrid keyword-plus-semantic search matter. Aurora pgvector is often the simplest strong option when retrieval is closely tied to customer, product, entitlement or transaction data. S3 Vectors fits large-scale, lower-cost retrieval where sub-second latency is acceptable and the corpus is huge [4].

In practice, many production RAG systems should use a tiered architecture: S3 as the full corpus and durable source of embeddings, OpenSearch or Aurora for hot subsets, and periodic evaluation jobs to decide which documents remain hot. This avoids overpaying for low-latency indexes on rarely used content.

Evaluation Is a Platform Capability

Evaluation should not be left to individual teams. Agent platforms need standard tests for answer accuracy, groundedness, tool selection, policy compliance, latency, cost and failure recovery. Wood Mackenzie’s emphasis on observability and evaluations reflects a real production lesson: without repeatable evaluation, teams cannot safely change prompts, models, tools or retrieval indexes [6].

AI Security Must Assume Supply-Chain and Runtime Compromise

The Rust package-owner attacks and Google’s reporting on AI-enabled supply-chain compromise are directly relevant to enterprise AI teams because AI applications often depend on fast-moving open-source packages, model-serving containers, notebooks, data connectors and CI/CD automation [1][10]. Dependency cooldowns, signed artifacts, pinned versions, SBOMs, isolated build environments and secret scanning are not optional controls for AI systems.

The OpenAI compaction-summary incident is also important even though it was rare and did not affect the final model described in the report. It shows that generated memory, summaries and retrieved context can carry instructions. Production systems should classify them as untrusted input and apply prompt-injection defenses, context separation and policy checks before tool execution [2].

How We Would Implement It

1. Establish a Shared AI Platform Layer

We would build a platform layer that product teams consume through APIs, templates and approved deployment paths:

Runtime: managed agent runtime where appropriate, such as Bedrock AgentCore for session isolation, scaling, identity, memory and observability [6][7].
Model gateway: provider-agnostic routing across approved models, with policy controls for data class, region, latency, cost and task type.
Tool gateway: MCP-compatible tool layer with OIDC authentication, RBAC, tenant scoping, parameter validation, rate limits and audit records [8].
Retrieval services: standard RAG interfaces backed by OpenSearch, Aurora pgvector and S3 Vectors depending on workload profile [4].
Observability: OpenTelemetry traces, model-call logs, token usage, retrieval metrics, tool calls, user feedback, latency and evaluation results.
Governance: central policies for approved models, data residency, retention, human-in-the-loop requirements and high-risk actions.

2. Design Retrieval by Workload

We would classify retrieval workloads before choosing infrastructure:

Customer support and internal knowledge search: OpenSearch with hybrid lexical and semantic retrieval where precision and fast responses are important; benchmark quality lift against latency cost [4].
CRM, ERP and entitlement-aware assistants: Aurora PostgreSQL with pgvector when joins, row-level authorization and transactional metadata are central [4].
Deep research, document intelligence and large archives: S3 Vectors as a cost-efficient corpus-scale store, with hot indexes exported to OpenSearch or Aurora for high-traffic subsets [4].

For every RAG system, we would track retrieval recall, answer groundedness, citation quality, p50/p95 latency, index size, embedding cost and stale-document rate. We would not approve a vector-store selection based only on vendor benchmarks.

3. Build Agent Authorization Around Tool Calls

Validate OIDC JWT issuer, audience, expiry and signature using the identity provider’s JWKS [8].
Require MFA for sensitive tool categories and optionally enforce geographic or network-location constraints [8].
Map identity groups to roles such as reader, author and admin [8].
Apply tool-level allowlists and parameter-level checks before execution [8].
Re-check authorization inside the tool Lambda or service, not only at the gateway [8].
Write immutable audit records for mutations, including user, tenant, tool, parameters, decision and timestamp [8].

This pattern prevents a prompt injection or model mistake from automatically becoming a privileged business action.

4. Add AI Development Lifecycle Observability

We would deploy a lightweight engineering analytics pipeline similar to the serverless Git metrics architecture: scheduled ingestion, change detection, Step Functions orchestration, Lambda collectors, S3 storage and QuickSight or Athena reporting [5]. Metrics should include pull-request throughput, review cycle time, issue flow, deployment frequency and contributor patterns. The goal is not surveillance; it is to measure whether AI coding tools are improving flow, quality and cycle time.

5. Treat Synthetic Data as a Controlled Experiment

For computer vision and other ML workloads, we would use synthetic data when real edge cases are dangerous, rare or expensive to collect. The SageMaker industrial-safety example is a good pattern: generate targeted scenarios, pseudo-label them, train candidate models and validate against real holdout data [9]. We would cap synthetic volume based on validation performance, because the reported regression at higher synthetic volume shows that artifacts can hurt generalization [9].

6. Implement Cost Controls From Day One

We would expose cost at three levels:

User and seat: adoption, token usage and cost per user, similar to MRH Trowe’s managed rollout model [7].
Workflow: cost per completed task, including model calls, retrieval, tool execution, orchestration and storage.
Platform: idle capacity, index storage, GPU utilization, data transfer, logging volume and environment sprawl.

Scheduled scaling, right-sizing, token budgets, caching, smaller embeddings, quantization and hot/cold retrieval tiers usually produce more durable savings than simply switching to the cheapest model.

Risks, Costs and Security

Key Risks

Supply-chain compromise: AI systems depend on open-source packages, containers, model artifacts and CI/CD workflows. Targeted maintainer attacks and rapid package compromise require dependency cooldowns, lockfiles, artifact signing and build isolation [1][10].
Prompt injection through memory and retrieval: Generated summaries, documents and tool outputs can contain instructions. They must be treated as untrusted data, especially before tool execution [2].
Over-centralization: A shared AI platform can become a bottleneck if it blocks teams from choosing suitable models, retrieval strategies or UX patterns. Platform governance should define safe boundaries, not force every use case into one template.
Unexpected inference and retrieval cost: Agent workflows can loop, call tools repeatedly, retrieve excessive context and generate long traces. Consumption billing helps avoid idle cost but can surprise teams without budgets and rate limits [6][7].
Weak evaluation: Without automated regression tests, model upgrades, prompt changes and index rebuilds can silently degrade quality or policy compliance.
Data residency and access leakage: Regulated deployments need private connectivity, regional controls, identity propagation, ACL enforcement and tenant-scoped storage, as shown in MRH Trowe’s architecture [7].

Cost Trade-Offs

Managed agent runtimes vs self-hosting: Managed runtimes reduce engineering and operations effort but create platform dependency. Self-hosting can be cheaper at high scale but requires mature SRE, security and runtime engineering.
OpenSearch vs Aurora vs S3 Vectors: OpenSearch can deliver strong low-latency and hybrid retrieval but may cost more for large always-hot indexes. Aurora is efficient when relational joins are needed. S3 Vectors is compelling for massive corpora where cost matters more than the lowest possible latency [4].
Large models vs smaller task-specific models: Larger models are not always better. The industrial safety study found a medium YOLO model was the best practical edge choice, while larger models overfit or regressed on the available data [9].
Synthetic data vs manual collection: Synthetic generation can reduce per-image cost and remove safety risk, but it still requires validation, artifact detection and real-world holdout testing [9].

Security Controls We Would Require Before Production

Private network paths for sensitive workloads, with no public administrative endpoints.
Central identity integration using Entra ID, IAM or equivalent, with MFA for sensitive operations [8].
Tool-level authorization, tenant scoping, parameter validation and immutable audit trails [8].
Secrets in managed secret stores, never in prompts, notebooks, repositories or environment dumps.
Dependency pinning, vulnerability scanning, software bills of materials, signed builds and cooldown periods for high-risk package updates [1].
Prompt-injection testing for RAG content, summaries, tool outputs and user-uploaded files [2].
Observability for model calls, retrieval, tool usage, latency, token spend, error rates and policy denials [6].
Human approval gates for high-impact actions such as financial transactions, customer communications, access changes, hiring decisions and production deployments.

The practical lesson for leaders is that enterprise AI infrastructure should be designed as a production platform, not a collection of demos. The organizations that scale fastest will standardize the control plane, measure cost and quality continuously, and secure every path from code and data to model, memory and tool execution.

Where Kimbodo Comes In

Kimbodo builds and operates this in production for businesses — see our AI Infrastructure & MLOps practice, or Estimate My Infrastructure.