How to Choose AI Inference Infrastructure Without Overpaying or Losing Control

What Happened

Recent deployment guidance illustrates three approaches to enterprise inference: managed model APIs with global routing, dedicated endpoints for compact models, and distributed GPU serving with separate prompt-processing and token-generation workers. They solve different problems; treating them as interchangeable creates avoidable cost and operational risk. [2][4][5]

• Managed inference: Amazon Bedrock offers GPT-6 Sol and Luna through global cross-Region inference from Singapore, Jakarta, Malaysia, Thailand, and Taipei, with Astra available from Singapore. Applications invoke inference profiles through Bedrock Runtime, and Bedrock selects a supported processing Region. [2]
• Dedicated inference: AWS provides llama.cpp containers for serving GGUF models on SageMaker AI CPU or GPU endpoints. An optional SemIf extension scores caller-declared choices rather than generating an answer token. [4]
• Distributed inference: Ray LLM containers combine vLLM, Ray Serve LLM, NIXL, and EFA support to separate prefill from decode across GPU nodes on EKS. The example serves Qwen3.5-9B through one OpenAI-compatible endpoint. [5]

Supporting infrastructure also matters. Python 3.15 adds explicit lazy imports and a low-overhead sampling profiler, while Python 3.10 is no longer supported. These changes affect runtime maintenance and diagnostics, but do not establish a broad inference-performance improvement. [1]

Why It Matters to Businesses

The deployment decision should follow the workload, not the model’s popularity. Open-ended assistance, bounded business decisions, and high-volume long-context generation have different quality, latency, governance, and utilization requirements.

For constrained decisions, a smaller dedicated model may be sufficient. On 231 serial JevBench tasks, Qwen3.5-4B Q4_K_M achieved 77.5% accuracy on CPU and 77.9% on GPU. GPU p95 latency was 0.599 seconds versus 19.291 seconds on CPU. The quality difference was only one answer; the more consequential result was tail latency. These measurements are configuration-specific, not service guarantees. [4]

Managed routing reduces serving responsibilities but does not remove governance work. Bedrock billing and monitoring remain in the source Region, while processing can occur elsewhere. CloudTrail records the processing location in additionalEventData.inferenceRegion. A local endpoint therefore does not establish local processing. [2]

Distributed serving addresses another bottleneck: long prompts competing with token generation. Separating prefill and decode allows independent scaling, but adds cluster, network, and capacity-management responsibilities. It should be justified by measured demand rather than assumed to be cheaper. [5]

Kimbodo Engineering Perspective

Choose the least complex architecture that meets the requirements

Our default would be to evaluate managed inference first for general-purpose applications, dedicated compact-model endpoints for bounded workloads, and distributed serving only when load tests demonstrate a bottleneck that simpler deployments cannot resolve.

• Managed Bedrock: Prefer it when avoiding model-serving operations is valuable and cross-Region processing is acceptable. Validate quotas, retention terms, and authorization before rollout. [2]
• SageMaker with llama.cpp: Consider it when a selected GGUF model passes task-specific evaluation and dedicated compute is justified. CPU can suit latency-tolerant work; GPU can be necessary for interactive response times. [4]
• EKS with Ray LLM: Consider it when independent prefill/decode scaling addresses an observed bottleneck and the team can operate Kubernetes, GPU capacity, and inter-node transport. [5]

An OpenAI-compatible interface is not a complete portability layer. Bedrock and SageMaker still require service-specific invocation, authentication, quotas, and observability. We would keep those concerns in provider adapters rather than spread them through business logic. [2][4]

For decision workflows, models should recommend or score—not authorize. SemIf’s ordered probabilities and logits require validation before being treated as confidence. Authorization, eligibility rules, and irreversible actions should remain deterministic. [4]

How We Would Implement It

1. Establish acceptance criteria

Build a representative evaluation set covering ordinary cases, ambiguous inputs, long prompts, and failure scenarios. Define minimum task quality, p95 latency, concurrency, processing-location restrictions, and allowable retention. Compare cost per accepted task, including retries, rather than token or instance prices alone.

2. Introduce a policy-aware inference gateway

Place a gateway between applications and serving backends. It should enforce identity, approved model profiles, input-size limits, output budgets, timeouts, and permitted processing locations. Route by workload and data classification; do not silently fall back to a backend with different residency or retention properties.

3. Configure each backend explicitly

• Bedrock: Use approved global inference profile IDs and least-privilege IAM. Where global routing requires an exception for aws:RequestedRegion being unspecified, scope the SCP exception with bedrock:InferenceProfileArn. Grant streaming or bearer-token permissions only where needed. [2]
• SageMaker: Package the GGUF artifact and use the appropriate CPU or CUDA container. If choice scoring is required, deploy SemIf separately using its child image pinned to an immutable ECR digest. Benchmark both latency and task quality before selecting hardware. [4]
• Ray on EKS: Separate system workloads from GPU workers. The documented EFA configuration places both GPU nodes in the same Availability Zone. Confirm in worker logs that NIXL selected LIBFABRIC; successful responses alone do not prove the intended transport is active. [5]

4. Measure application outcomes and infrastructure behavior

Track task success, end-to-end latency, time to first token, errors, retries, token usage, and compute utilization where available. For Bedrock, record the actual processing Region and alert around 70–80% of quota. Its source-Region TPM accounting weights output tokens at 10×, versus 1× for input and cache-write tokens; keep max_tokens near expected output. This is quota accounting, not a pricing multiplier. [2]

5. Make deployment and teardown reproducible

Version model artifacts, containers, serving configuration, evaluations, and infrastructure together. Test rollback and removal procedures. SageMaker cleanup scripts remove only named endpoint resources; build resources, images, S3 objects, logs, and IAM resources need separate attention. Ray teardown must also remove GPU capacity and the EKS cluster to stop the associated charges. [4][5]

Risks, Costs and Security

• Data retention: AWS may retain classifier-flagged Bedrock traffic for up to 30 days in the destination Region. Eligible customers can request full zero data retention; confirm eligibility and configuration rather than assuming it. [2]
• Idle capacity: Dedicated endpoints and distributed clusters incur provisioned-resource costs. Compare realistic utilization, operational effort, and latency requirements before replacing managed inference.
• Availability trade-offs: The illustrated Ray deployment uses a same-AZ GPU pair for EFA. It is not, by itself, a multi-AZ availability design; recovery capacity and failover require additional planning. [5]
• Secrets exposure: Keep credentials out of prompts and AI debugging workflows. The web-edit example was designed to edit files directly over Tailscale without sending credentials through an LLM. Private connectivity still requires authentication, restricted file access, and auditing. [3]
• Runtime risk: Retire unsupported Python versions and test framework compatibility before upgrading. Python 3.15 performed broadly similarly to 3.14 in the cited benchmarks; one experimental-JIT result is not a basis for production capacity planning. [1]

The practical goal is controlled inference, not maximum infrastructure sophistication. Choose the smallest operational footprint that meets measured quality, latency, security, and governance requirements—and increase complexity only when evidence supports it.

Sources

1. [1] Hello Python 3.15, goodbye 3.10, and now I can use 3.11
2. [2] Accessing OpenAI GPT models on Amazon Bedrock with global cross-Region inference in Thailand, Malaysia, Singapore, Indonesia, and Taiwan
3. [3] web-edit 0.1a0
4. [4] Deploy decision models with llama.cpp on Amazon SageMaker AI
5. [5] Run multi-node inference workloads using Ray LLM Deep Learning Containers