How to Build Production AI Agent Platforms Without Losing Control of Cost, Security or Grounding

What Happened

Enterprise AI infrastructure is moving from isolated LLM calls toward governed agent platforms: model routing, tool interfaces, identity, memory, evaluation, runtime orchestration and security controls are becoming first-class architecture concerns.

Several developments point in the same direction. DeepSeek V4 Flash 0731 shows that open-weight and lower-cost models are becoming credible options for agentic workloads: it is a 304 billion parameter model with a 167 GB Hugging Face model file, ranked strongly on intelligence-versus-cost benchmarks, with listed pricing of $0.14 per million input tokens and $0.27 per million output tokens [1]. But the same example also shows that deployment defaults matter: raising OpenRouter reasoning effort to high materially improved output quality in a simple test [1].

The Model Context Protocol is also becoming simpler to operate. Stateless MCP removes session state and the prior two-request flow; clients can now make a single POST with protocol headers and client metadata, which reduces implementation complexity for clients and servers [2]. Example implementations include a Python CLI for listing and calling tools, plus a Datasette MCP endpoint exposing read-only database tools such as schema inspection and SQL execution [2].

Agent platform tooling is maturing around lifecycle management. Google’s agent development flow describes setup, build, deploy, govern, evaluate and publish stages using Agents CLI, ADK, Agent Runtime, Agent Identity, Agent Registry, Agent Gateway, Model Armor and CI evaluations [7]. Gemini Enterprise Agent Platform is making related components generally available, including Agent Runtime, Agent Identity, Agent Memory Bank, Agent Gateway, Agent Registry, Agent Evaluation and Agent Observability [8].

Evaluation is becoming a production dependency rather than an experiment. The smevals project formalizes small evaluation suites as YAML tasks, model configs, run records, graders and reports, giving teams a lightweight way to compare models, prompts and harnesses [5]. In the agent lifecycle example, every cited SEC accession number and 8-K item code must appear verbatim in tool output, and these checks are integrated into CI to prevent grounding regressions [7].

Why It Matters to Businesses

The core business issue is no longer whether an LLM can answer a prompt. It is whether an AI system can perform useful work with controlled cost, auditable data access, predictable failure modes and measurable quality.

Model choice is now a portfolio decision. Low-cost frontier-adjacent models can reduce inference spend, but teams must benchmark them against their own tasks rather than relying on aggregate rankings [1][5].
Agent tooling increases both capability and blast radius. MCP and browser-executed tools make it easier for agents to act on real systems, but every new tool becomes a permission boundary that must be designed, logged and tested [2][6].
Governance is becoming architectural. Agent identity, registries, gateways and memory systems are not compliance add-ons; they determine who the agent is, what it can access, what it remembers and how behavior is audited [7][8].
Grounding is a product requirement. The strongest production designs constrain models to narrate deterministic tool output, especially in regulated or decision-support workflows [7].
Open-weight models change negotiating leverage. The growing quality of open-weight models gives enterprises more deployment options, from hosted APIs to private inference, but shifts responsibility for serving, optimization and security onto the engineering organization [1][4].

Kimbodo Engineering Perspective

For production systems, we would treat the LLM as one component inside a controlled execution platform, not as the platform itself. The important design decisions are around orchestration, permissions, tool contracts, evaluation, observability and cost routing.

Use the cheapest model that passes task-specific evaluations

DeepSeek V4 Flash 0731 is attractive on price and benchmark position, but a 167 GB model artifact changes the economics of self-hosting [1]. For many teams, hosted inference will be cheaper and faster to operationalize until utilization is high and stable. Private deployment can make sense when data residency, latency, customization or volume justify GPU capacity planning, quantization work, autoscaling and model-serving operations.

The reasoning-effort example is also important: higher reasoning settings can improve outcomes but increase latency and token usage [1]. In production, we would not set high reasoning globally. We would route simple extraction and classification tasks to cheaper settings, escalate only on ambiguity, and measure the incremental value in evals.

Prefer explicit tools over unrestricted execution

Stateless MCP is directionally good for enterprise architecture because it simplifies client-server interaction and makes agent capabilities easier to reason about [2]. A typed tool such as list databases, get schema or execute read-only SQL is safer than giving an agent arbitrary shell or network access [2].

That does not make tools inherently safe. Tool outputs can contain prompt injection, database queries can leak sensitive data, and browser-executed JavaScript tools expand the trust boundary into the user’s environment [6]. The right pattern is explicit tools, narrow scopes, deterministic validation and full audit logging.

Separate deterministic work from model judgment

The strongest agent architecture in the examples is the Industry Watch workflow: fetch from known sources, reconcile claims and filings deterministically, then ask the model to narrate grounded, cited results [7]. That separation reduces hallucination risk and makes errors easier to diagnose. We would apply the same pattern to financial analysis, compliance review, support automation, procurement workflows and clinical administration.

Treat evals as release gates

Small, fast evaluation suites are often more valuable than large but rarely run benchmarks. Tools like smevals are useful because they make model, prompt and harness changes comparable across repeatable tasks [5]. For production, every change to prompts, tools, model versions, routing rules and memory schemas should run through CI evaluations before release.

How We Would Implement It

1. Define the agent control plane

Start with a control plane that owns agent registration, versioning, identity, deployment status, model routing policy, tool permissions and observability. Managed platforms such as Gemini Enterprise Agent Platform provide components for runtime, identity, gateway, registry, evaluation and observability [8]. If building independently, replicate those primitives rather than wiring agents directly to models and tools.

Maintain an Agent Registry with owner, version, purpose, approved tools, data classification and rollback target.
Assign each agent a non-human identity with least-privilege access and auditable credentials, matching the Agent Identity pattern [7][8].
Route all model and tool calls through an Agent Gateway for policy enforcement, logging, egress control and safety filters [7][8].

2. Use a model routing layer

Implement a model gateway that can route requests across hosted proprietary models, hosted open-weight models and, where justified, private inference. DeepSeek V4 Flash 0731-style models should be evaluated as candidates for cost-sensitive agentic workloads, but only after task-specific testing [1][5].

Use small or low-cost models for classification, routing, summarization and extraction.
Use stronger models or higher reasoning effort only for complex synthesis, planning or disputed cases.
Capture model name, version, parameters, reasoning settings, latency, token usage and cost per task.
Run canary comparisons before switching default models.

3. Standardize tool interfaces with stateless MCP where appropriate

For internal tools, define explicit contracts: input schema, output schema, allowed resources, timeout, rate limit, data classification and idempotency. Stateless MCP can reduce integration complexity by using a single request pattern rather than session-dependent flows [2].

Expose read-only tools before write tools.
Return structured evidence, not prose blobs, wherever possible.
Mark untrusted content from the web, email, documents and databases before it reaches the model.
Require policy approval before tools can access browsers, production databases, SaaS APIs or customer data.

4. Build deterministic pipelines around the agent

Use the model for language and judgment; use code for reconciliation, joining, validation, deduplication and scoring. The Industry Watch pattern is a good template: fetch from trusted systems, join deterministically, score using explicit rules, then ask the model to explain only the grounded results [7].

Keep source fetchers independent from the model.
Perform joins, deduplication and materiality scoring in code.
Require every generated citation to map back to tool output.
Store intermediate artifacts for audit and replay.

5. Add memory only after governance is clear

Memory can improve personalization and long-running workflows, as described by Agent Memory Bank and Agent Runtime [8]. But memory also creates retention, privacy, access-control and poisoning risks. Start with narrow schemas for preferences, decisions and account history; avoid free-form memory until you have deletion, review and provenance controls.

6. Make evaluation part of CI and operations

Create evaluation suites for each agent capability. Follow the smevals-style structure of tasks, model configs, runs and graders [5]. Include deterministic checks, model-graded checks where necessary, and regression tests for known failures.

Block releases when grounding, citation or permission tests fail.
Track quality by model version and prompt version.
Cluster failures into prompt issues, tool issues, retrieval issues, policy issues and model limitations.
Optimize only after identifying which failure class dominates.

Risks, Costs and Security

Cost risks

Reasoning settings can silently raise spend. Higher reasoning effort may improve quality, but it should be selectively enabled and measured per workflow [1].
Self-hosting open-weight models is not automatically cheaper. A 167 GB model artifact implies serious GPU memory, serving, scaling and operations requirements [1].
Long-running agents need budget controls. Agent runtimes that can operate for days are useful for asynchronous work, but require task limits, timeout policies and cost ceilings [8].
Evaluation has an upfront cost but reduces production waste. Without evals, teams tend to overpay for larger models, ship regressions and debug failures manually [5][7].

Security risks

Prompt injection through tool output. News, web pages, documents and database rows must be treated as untrusted inputs, as the agent lifecycle guidance explicitly recommends for news text [7].
Over-permissioned agents. Per-agent identity, least privilege and non-repudiable audit trails are essential controls [7][8].
Uncontrolled egress. Agents should operate behind allow-lists, as in the example that restricts access to SEC, GDELT and investor-relations feeds [7].
Browser execution risk. Tools that run JavaScript in a user’s browser can be powerful, but they require strong consent, origin controls, sandboxing and logging [6].
Memory leakage or poisoning. Persistent memory should use structured schemas, access controls, provenance and deletion workflows [8].

Operational controls we would require

Central gateway for all model and tool calls.
Per-agent IAM identity and scoped credentials.
Environment-specific registries for development, staging and production agents.
Egress allow-lists and network segmentation.
Prompt-injection filtering and policy checks such as Model Armor-style controls [7][8].
Immutable logs for prompts, tool calls, model responses, costs and user-visible outputs.
Replayable traces for incident response and regression testing.
CI evaluation gates before prompt, model, tool or memory changes reach production.

The practical lesson is clear: production AI platforms are becoming distributed systems with probabilistic components. The winning architecture is not the one with the most capable model in isolation. It is the one that combines model flexibility, explicit tools, deterministic validation, governed identity, cost-aware routing and continuous evaluation.

Where Kimbodo Comes In

Kimbodo builds and operates this in production for businesses — see our AI Infrastructure & MLOps practice. Wondering what it would cost for your organization? Get a preliminary range, timeline and architecture in about a minute.

Estimate My Infrastructure