How to Choose Between Self-Hosted LLM Infrastructure and Managed Agent Platforms for Enterprise AI

What Happened

Two recent enterprise AI implementation patterns show where production infrastructure is heading: very large open-weight model deployment on dedicated GPU clusters, and embedded agentic applications built on managed AI services.

On the infrastructure-heavy side, Qwen3.8-2.4T-A95B is a 2.4 trillion-parameter open-weight model designed for agentic and reasoning workloads, including multi-step coding, long-horizon planning, tool use, and large-context reasoning. Although the model activates about 95 billion parameters per token, quantized NVFP4/MXFP4 weights reduce storage from roughly 4.8 TB in BF16 to about 1.2 TB, making single-node deployment possible on an Amazon SageMaker HyperPod ml.p6-b300.48xlarge with 8 NVIDIA B300 GPUs and 2.1 TB aggregate GPU memory [1].

The recommended deployment pattern uses SageMaker HyperPod on EKS, vLLM, the HyperPod InferenceOperator CRD, local NVMe caching, KEDA autoscaling, and Prometheus metrics. Key runtime choices include tensor parallelism across 8 GPUs, NVFP4 quantization, prefix caching, expert parallelism, and Multi-Token Prediction speculative decoding. Benchmarks on one p6-b300 node show that MTP materially improves time-to-first-token and modestly improves throughput, especially when combined with expert parallelism [1].

On the application side, HPE Zerto built an embedded troubleshooting assistant for data-protection and disaster-recovery teams using Amazon Bedrock. The system runs as an on-premises Strands Agents pod inside the Zerto Virtual Manager appliance, while model inference and knowledge-base queries go over HTTPS to AWS. Its architecture uses an orchestrator agent, specialized sub-agents, MCP servers exposing Zerto APIs, Amazon Bedrock Knowledge Bases for RAG, Bedrock Guardrails, tenant-aware IAM, CloudWatch quotas, DynamoDB, Lambda, and streaming UI responses over Server-Sent Events [2].

The Zerto implementation is notable because it is not a generic chatbot. It is an embedded operational workflow tool: it reads product state, reasons over live system data, retrieves approved knowledge, and returns compact, redacted, actionable guidance. Zerto reports more than 20% customer adoption and about a 10% reduction in support cases for supported workflows [2].

Why It Matters to Businesses

Enterprise AI buyers now face a practical infrastructure split. Some workloads justify dedicated model infrastructure because they require control, scale, context length, latency tuning, or model customization. Other workloads are better served by managed model platforms because the business value is in workflow integration, governance, and time-to-market rather than owning the inference stack.

• Self-hosted LLM infrastructure gives control but creates operational burden. Deploying a trillion-scale MoE model requires GPU capacity planning, quantization validation, model caching, autoscaling, observability, safety testing, and ongoing vLLM tuning [1].
• Managed agent platforms reduce infrastructure ownership but shift focus to architecture discipline. Zerto still had to design agent decomposition, MCP access patterns, tenant isolation, guardrails, telemetry, quotas, evaluation, and redaction [2].
• Cost optimization is no longer only about tokens. Teams must compare reserved GPU capacity, idle GPU risk, model download and cold-start behavior, managed inference pricing, knowledge-base query costs, support-case reduction, and engineering maintenance.
• Production AI is becoming an orchestration problem. The differentiator is often not the model alone, but how models are connected to tools, data, evaluation, monitoring, identity, and deployment automation.

The business implication is straightforward: enterprises should not standardize on one AI infrastructure pattern for all use cases. A troubleshooting assistant embedded in a SaaS or appliance product has different requirements from a high-volume coding agent, research agent, or internal platform serving many downstream teams.

Kimbodo Engineering Perspective

Use dedicated GPU inference when model control is the product requirement

Running a model like Qwen3.8-2.4T-A95B can make sense when an organization needs a large-context, open-weight, high-capability model with control over serving behavior, quantization, batching, speculative decoding, caching, and tool-calling semantics. The HyperPod and vLLM pattern is appropriate for platform teams that already operate Kubernetes, GPU workloads, and observability pipelines [1].

The trade-off is that deployment success depends on operational maturity. A single node may fit the quantized weights, KV state, activations, and overhead, but that does not make it simple. Teams still need cold-start planning, local cache lifecycle management, capacity buffers, request admission controls, failure handling, and benchmarking against real workloads. First deployment can require downloading roughly 1.2 TB of model artifacts, with subsequent restarts improved by NVMe cache [1].

Use managed model platforms when workflow delivery matters more than model ownership

The Zerto architecture reflects a more common enterprise pattern: embed AI into an existing operational workflow, keep sensitive system execution local where appropriate, and use managed model inference and RAG services for the language and retrieval layer [2]. This approach reduces GPU infrastructure ownership and can speed delivery, especially when the assistant must be governed, tenant-aware, and integrated into an existing product UI.

However, managed does not mean simple. The hard engineering work moves to agent boundaries, data contracts, identity propagation, retrieval quality, evaluation, redaction, and observability. Zerto’s use of specialized sub-agents is an important production lesson: broad autonomous agents often become unreliable, recursive, or hard to test. Smaller agents with scoped tools and responsibilities are easier to evaluate and govern [2].

The right decision is usually hybrid

For many enterprises, the target architecture should support both modes: managed model APIs for fast product integration and dedicated inference clusters for workloads where scale, privacy, customization, or unit economics justify it. The platform should abstract model access without hiding operational details that matter, such as context limits, latency, token cost, safety policy, tool permissions, and logging behavior.

How We Would Implement It

1. Classify workloads before choosing infrastructure

We would start by segmenting AI workloads into clear classes:

• Embedded workflow agents: customer support, troubleshooting, operational copilots, internal helpdesk, compliance assistants.
• High-throughput inference services: coding assistants, document processing, analytics copilots, batch reasoning pipelines.
• Long-context reasoning systems: legal review, incident reconstruction, engineering analysis, large technical documentation workflows.
• Regulated or sensitive workloads: systems requiring strict data residency, tenant isolation, private networking, or local execution.

For each class, we would define latency targets, concurrency, context size, data sensitivity, expected token volume, availability requirements, and acceptable model providers.

2. Build a model gateway instead of hard-coding providers

We would implement a model gateway that standardizes authentication, request routing, tracing, retries, policy enforcement, and cost attribution across managed APIs and self-hosted endpoints. This prevents application teams from directly coupling business logic to a single model or serving stack.

• Route low-risk or bursty workloads to managed platforms such as Bedrock where appropriate.
• Route high-volume or specialized workloads to self-hosted vLLM endpoints on Kubernetes.
• Apply per-tenant quotas, rate limits, data-loss-prevention checks, and audit logging at the gateway.
• Capture latency, token usage, cache hit rate, tool calls, safety interventions, and error modes.

3. For self-hosted LLMs, standardize on Kubernetes-native inference operations

For a Qwen-class deployment, we would use a Kubernetes-native pattern similar to the HyperPod InferenceOperator model: declarative endpoint configuration, image versioning, node selectors, GPU topology awareness, health checks, autoscaling, and Prometheus scraping [1].

• Use vLLM with tensor parallelism across the 8-GPU node for the large quantized model [1].
• Enable NVFP4 quantization only after validating quality against business-specific evaluation sets.
• Enable prefix caching for repeated system prompts, agent instructions, and retrieval templates [1].
• Enable MTP speculative decoding with conservative settings first, then increase speculative tokens only if acceptance rates remain high [1].
• Scrape metrics such as speculative decoding acceptance, GPU cache usage, running requests, and end-to-end latency for autoscaling and tuning [1].
• Benchmark with realistic prompt lengths, tool traces, streaming responses, and concurrency rather than synthetic single-turn tests.

4. For agentic applications, separate orchestration from tools

For embedded assistants like the Zerto troubleshooting system, we would use a hub-and-spoke agent design: one orchestrator for intent routing and state management, and specialized agents for bounded domains such as infrastructure health, replication status, incident diagnosis, documentation retrieval, and remediation planning [2].

• Expose enterprise system functions through MCP servers or equivalent typed tool interfaces.
• Keep tool permissions narrow and tenant-scoped.
• Use RAG only against curated, versioned, access-controlled knowledge sources.
• Stream intermediate progress to the UI using SSE or WebSockets so users can see what the agent is doing [2].
• Return compact, redacted outputs rather than raw logs, credentials, or full diagnostic dumps [2].

5. Treat evaluation as a release gate

We would make AI evaluation part of CI/CD. Zerto’s use of PyTest and Strands Agents Evals SDK across basic, multi-turn, and dynamic tests is the right pattern for production agents [2]. The same principle applies to self-hosted models: every model, quantization mode, prompt template, retrieval change, or tool update should be tested before rollout.

• Measure task success, factuality, tool-call correctness, latency, token usage, refusal behavior, and unsafe output rate.
• Use golden-path tests for known workflows and adversarial tests for boundary conditions.
• Store full traces for internal evaluation, while redacting sensitive content from analytics and support views.
• Canary model upgrades by tenant, region, or internal user group before broad deployment.

Risks, Costs and Security

Cost risks

The largest cost risk in self-hosted inference is underutilized GPU capacity. A p6-b300-class node is appropriate only if the workload can justify high fixed infrastructure cost through sustained demand, strict latency needs, or model-control requirements. Otherwise, managed inference may be cheaper despite higher per-token pricing.

For managed agent platforms, the cost risk is uncontrolled usage: long contexts, recursive agent loops, repeated retrieval calls, excessive tool retries, and unbounded multi-turn sessions. Quotas, token budgets, request timeouts, and per-tenant cost attribution are mandatory.

Operational risks

• Cold starts and model loading: Large model artifacts can add significant deployment time; local NVMe caching reduces repeat startup time but must be managed [1].
• Autoscaling mismatch: GPU workloads do not scale like stateless web services. Queue depth, KV cache pressure, token throughput, and batch size are better signals than CPU utilization.
• Quality regressions from quantization: NVFP4/MXFP4 can make deployment feasible, but teams must validate reasoning quality, tool-call accuracy, and long-context behavior after quantization [1].
• Agent instability: Overly broad agents can recurse, call tools unnecessarily, or produce inconsistent plans. Specialized sub-agents and bounded tool access reduce this risk [2].

Security risks

Production AI systems expand the attack surface. The main risks are prompt injection, data exfiltration through tool calls, cross-tenant leakage, unsafe remediation advice, retrieval poisoning, over-privileged service roles, and leakage through logs or traces.

Controls should include tenant-aware IAM, least-privilege tool permissions, guardrails, network egress controls, encrypted transport, private connectivity where possible, secrets redaction, prompt-injection testing, audit trails, and policy enforcement at both the model gateway and tool layer. Zerto’s use of tenant role tagging, guardrails, quotas, compact redacted outputs, and CloudWatch telemetry is a strong reference pattern for enterprise agent governance [2].

Decision guidance

• Choose managed model services when speed, governance, elastic usage, and product workflow integration are more important than controlling the serving stack.
• Choose self-hosted inference when latency, volume, model customization, data-control requirements, or unit economics justify GPU operations.
• Choose a hybrid platform when multiple business units will build AI applications with different risk, cost, and performance profiles.

The production lesson is clear: enterprise AI infrastructure should be designed as a portfolio, not a single bet. The winning architecture gives teams safe access to managed models where they accelerate delivery, dedicated inference where control and economics justify it, and a common operating layer for identity, observability, evaluation, cost management, and security.

Sources

1. [1] Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM
2. [2] How HPE Zerto built an agentic troubleshooting system with Amazon Bedrock