What Happened
Google Cloud introduced a broad set of AI infrastructure updates focused on running large-scale agentic systems, LLM inference workloads and enterprise automation on Kubernetes and managed cloud services.
The most significant infrastructure shift is the move toward high-density, fast-resuming agent execution environments. The new open-source GKE Agent Substrate is designed to run millions of sandboxes at roughly 10x density, resume in under 500 ms, and support more than 500 suspend/resume activations per second, with native zero-trust kernel and network isolation. Google also made GKE Agent Sandbox generally available for safer agent execution on Kubernetes [1].
For inference workloads, Google expanded GKE capabilities around scale-to-zero, custom PromQL-based HPA scaling, pod snapshots that can reduce inference startup time by up to 89%, and multi-cluster inference routing through GKE Inference Gateway and the llm-d router. The routing layer is designed to pool GPUs and TPUs globally across clusters with memory-aware scheduling and less than 1% routing overhead [1].
On the storage and compute side, Google announced high-performance options including Filestore agent volumes, Cloud Storage Rapid, Rapid Cache, M4N VMs with Titanium offload and Hyperdisk Extreme, Z4D bare-metal instances, Cloud Managed Lustre, and C4N network/storage-optimized VMs. These are aimed at AI workloads that need fast model loading, high-throughput checkpointing, distributed training, low-latency retrieval, or large-scale data preprocessing [1].
Google also launched the Google Cloud CLI Remote MCP Server in public preview. It exposes gcloud and bq commands as MCP tools, allowing agents to run operational and data-management tasks without local CLI installation. It supports keyless Agent Identity on hosted Google Cloud platforms or OAuth 2.0 for external runtimes, enforces caller IAM and organization policy constraints, integrates with Model Armor, and can log invocations to Cloud Audit Logs [2].
Why It Matters to Businesses
Enterprise AI systems are moving from simple chat interfaces to distributed applications that call tools, execute workflows, retrieve business data, run code, inspect infrastructure and coordinate multiple models. That changes the infrastructure problem. Teams now need to operate not just models, but fleets of agents, sandboxes, vector stores, queues, GPUs, CPUs, data pipelines, observability systems and security controls.
The business impact is most visible in four areas:
- Lower latency for interactive AI: Pod snapshots, faster sandbox resume and high-performance storage reduce cold-start penalties for agents and inference services [1].
- Better accelerator utilization: Memory-aware routing, cooperative time-slicing and multi-cluster pooling help teams avoid stranded GPU and TPU capacity [1].
- Safer automation: Remote MCP access to cloud CLI operations can eliminate unmanaged local binaries and ambient credentials, while preserving IAM and audit controls [2].
- More practical scale-to-zero economics: Agent and inference workloads are often bursty. Scale-to-zero and fast restore make it easier to reduce idle spend without making users wait too long [1].
For business leaders, the key takeaway is that AI infrastructure cost is no longer just model hosting cost. It includes orchestration overhead, cold starts, storage access patterns, identity design, audit logging, security scanning, network isolation, retry behavior, and the cost of idle accelerators. A platform that looks cheaper in a benchmark can become expensive if it wastes GPU memory, reloads large models too often, or forces every team to build its own agent execution layer.
Kimbodo Engineering Perspective
These updates reflect a production reality we see often: enterprise AI systems fail less because of model quality alone and more because of weak platform engineering around the model. The core architectural question is not “Which LLM should we use?” but “How do we safely route work to models, tools, data and infrastructure under cost, latency and governance constraints?”
Agent sandboxes are becoming a first-class infrastructure primitive
Agents that browse, run code, call APIs or manipulate cloud resources need isolation. Running them as ordinary application pods without strong boundaries creates unacceptable risk. A sandbox layer with fast suspend/resume and zero-trust kernel and network isolation is a strong fit for high-volume agent execution [1].
The trade-off is operational complexity. High-density sandboxes require careful quota management, image lifecycle discipline, egress controls, per-task identity and observability. For many enterprises, the right first step is not to run millions of sandboxes, but to standardize a secure sandbox profile for a few high-value workflows: data analysis, document processing, customer-support actions or infrastructure diagnostics.
Inference routing should be treated as a control plane
Multi-cluster inference routing is valuable because model serving is constrained by memory, accelerator type, model version, context length, batching and locality. A simple load balancer is not enough. Runtime-aware routing can improve utilization and reduce queueing, especially when multiple models and hardware profiles are involved [1].
The trade-off is that routing becomes business-critical. If the router does not understand model compatibility, tenant boundaries, data residency, cost tiers and degradation rules, it can create reliability or compliance failures. Enterprises should design routing policies explicitly: premium users may get low-latency GPU pools; batch workloads may use lower-cost queues; regulated data may stay in specific regions.
MCP is useful, but tool permissions matter more than tool access
The Remote MCP Server makes it easier for agents to use Google Cloud operations and BigQuery workflows through standardized tools [2]. This is powerful, but it should not be deployed as a broad “agent can run gcloud” capability. The secure pattern is least-privilege tool access, scoped service identities, approval gates for destructive operations, and full logging.
The strongest feature is not convenience; it is governance. Zero ambient credentials, IAM enforcement, organization policy constraints, Model Armor integration and Cloud Audit Logs make MCP-based automation more practical for enterprises [2]. Still, teams should assume an LLM may generate incorrect commands and design guardrails accordingly.
How We Would Implement It
1. Define workload classes before choosing infrastructure
We would separate workloads into distinct classes because each has different scaling and security needs:
- Interactive inference: Low-latency model calls for applications and agents.
- Batch inference: High-throughput, delay-tolerant document, media or analytics processing.
- Agent execution: Sandboxed tasks that may call tools, run code or use browsers.
- RAG and data access: Retrieval, embedding, search, permissions filtering and source grounding.
- Training and fine-tuning: Accelerator-heavy workloads with different storage and checkpointing needs.
- Operations automation: MCP-enabled cloud, data and platform tasks.
2. Build a Kubernetes-based AI runtime with separate node pools
For enterprises already standardized on Google Cloud, we would use GKE as the core orchestration layer. The platform would include separate node pools for CPU agents, GPU inference, TPU inference or training, memory-heavy retrieval services, and system workloads. This avoids mixing latency-sensitive inference with bursty agent tasks.
We would enable GKE Dataplane V2, Network Policies, workload identity, centralized logging, metrics and trace collection. For agent workloads, we would evaluate GKE Agent Sandbox and the Agent Substrate pattern where task volume, isolation requirements and cold-start latency justify it [1].
3. Use model-serving gateways instead of direct service calls
Applications should not call individual model pods directly. We would place an inference gateway in front of model-serving backends and route based on model, tenant, latency target, context size, accelerator availability and region. For larger deployments, multi-cluster GKE Inference Gateway with llm-d-style routing is the right architectural direction because it can pool accelerators across clusters and make memory-aware decisions [1].
The gateway should enforce request limits, authentication, tenant attribution, prompt logging policy, safety filters, budget tags and fallback rules. For example, if a premium model is saturated, the gateway may route non-critical traffic to a smaller model, queue batch jobs, or return a controlled retry response.
4. Design for fast cold starts, but do not rely on them alone
Scale-to-zero and pod snapshots can materially reduce idle spend and startup latency [1]. We would use them for bursty models, development environments, internal tools and intermittent agent pools. For customer-facing real-time workloads, we would still keep a warm minimum replica count for critical models.
The practical rule is simple: use scale-to-zero where user tolerance allows it, and keep warm capacity where latency directly affects revenue or customer experience.
5. Match storage to model and data access patterns
AI systems often underperform because storage is treated as an afterthought. We would choose storage based on access pattern:
- Model weights and hot artifacts: Use low-latency storage or caching to reduce model load time.
- Training checkpoints: Use high-throughput parallel file systems such as managed Lustre-style storage when needed [1].
- RAG source data: Use object storage, metadata stores and vector/search indexes with clear refresh pipelines.
- Agent scratch space: Use ephemeral isolated storage unless persistence is explicitly required.
Filestore agent volumes, Cloud Storage Rapid and Rapid Cache are relevant where repeated model or data access becomes a latency bottleneck [1]. The cost decision should be based on measured startup time, cache hit rates and accelerator idle time, not storage price alone.
6. Introduce MCP with a controlled automation boundary
We would not give agents broad cloud administration access on day one. For the Google Cloud CLI Remote MCP Server, we would start with narrow, read-heavy use cases such as BigQuery job inspection, table metadata review, cost diagnostics or deployment status checks [2].
A production MCP rollout should include:
- Dedicated agent identities with least-privilege IAM roles.
- Tool allowlists for approved gcloud and bq operations.
- Human approval gates for destructive or high-cost actions.
- Audit logging for every tool invocation and authorization decision [2].
- Prompt and response screening for sensitive or unsafe content using controls such as Model Armor [2].
- Policy tests that simulate prompt injection and mistaken command generation.
Risks, Costs and Security
Cost risks
The largest cost risk is idle accelerator capacity. GPU and TPU utilization can drop when models are overprovisioned, routing is naive, batch sizes are too small, or workloads are fragmented across clusters. Multi-cluster pooling and runtime-aware routing can help, but only if paired with accurate metrics, admission control and workload prioritization [1].
Fast storage and premium compute can also become expensive if applied everywhere. High-throughput file systems, rapid object storage and network-optimized VMs should be reserved for workloads where they reduce total cost by cutting model load time, training duration or accelerator idle time.
Security risks
Agentic systems expand the attack surface because they combine natural-language inputs, tool execution, data access and sometimes code execution. The main risks are prompt injection, excessive tool permissions, data exfiltration, unsafe shell or CLI commands, tenant boundary failures and unreviewed autonomous actions.
Controls should include sandbox isolation, network egress restrictions, per-task identity, short-lived credentials, organization policies, audit logs, command allowlists and human approval for high-risk actions. The Remote MCP Server’s use of caller IAM, zero ambient credentials and Cloud Audit Logs is aligned with this model, but enterprises still need to design least-privilege roles and operational guardrails [2].
Reliability risks
As inference gateways, MCP servers and agent substrates become shared platform components, they become critical dependencies. Outages or misconfiguration can affect many applications at once. We would deploy these components with multi-zone redundancy, conservative rollout policies, synthetic tests, quota monitoring and clear fallback paths.
For model serving, the most important reliability decision is graceful degradation. Not every request should fail when the largest model or fastest accelerator pool is unavailable. Applications should define acceptable fallbacks by workflow: smaller model, delayed processing, cached answer, human handoff or retry queue.
Implementation cost
The engineering cost is meaningful. A production AI platform needs platform engineering, MLOps, cloud security, data engineering and application teams working from a shared architecture. The payoff is reuse: once identity, routing, observability, sandboxing, deployment and cost controls are standardized, new AI applications can be launched faster and operated more safely.
The practical recommendation is to build incrementally. Start with one or two high-value AI workflows, create the shared runtime components they need, and harden those components before expanding. Avoid building a large internal AI platform before there are real workloads to validate latency, cost and governance assumptions.
Where Kimbodo Comes In
Kimbodo builds and operates this in production for businesses — see our AI Infrastructure & MLOps practice, or Estimate My Infrastructure.
