How to Cut LLM Costs in Enterprise AI Workflows with CPU Pre-Filtering and Agentic Remediation

What Happened

Google described an architecture for cost-effective, high-throughput generative AI workflows using Apache Beam and Google Dataflow. The pattern combines lightweight CPU inference upstream with selective downstream LLM agent execution [1].

The example pipeline uses a DistilBERT sentiment model, distilbert-base-uncased-finetuned-sst-2-english, through Beam’s RunInference transform and HuggingFacePipelineModelHandler. This stage classifies incoming messages and filters out routine POSITIVE or NEUTRAL events. Only NEGATIVE messages are routed to a heavier agentic workflow [1].

The downstream workflow uses an ADK agent running gemini-3.5-flash. The agent is wrapped with ADKAgentModelHandler and can invoke tools for multi-step remediation. In the example, these tools include BigQuery lookups for user email addresses, order and inventory checks, and Gmail-based notification sending using the gmail.send scope [1].

The key design choice is that the pipeline does not invoke the LLM for every event. In the example, around 95% of events are handled by the CPU-based classifier and dropped before they reach the agent. The LLM is reserved for the small subset of events that require reasoning, tool use, and dynamic remediation [1].

Why It Matters to Businesses

This is a practical production AI pattern: use cheaper deterministic or small-model stages to protect expensive agentic stages. For business systems processing large volumes of customer messages, tickets, transactions, alerts, or operational events, calling an LLM for every input is often unnecessary and economically fragile.

• Lower inference cost: CPU inference handles high-volume classification, while LLM calls are limited to cases with business value or operational urgency [1].
• Lower latency for routine events: lightweight models can classify messages in milliseconds, avoiding unnecessary round trips to external model APIs [1].
• Reduced quota pressure: sparse LLM invocation lowers the risk of hitting rate limits during traffic spikes [1].
• More maintainable orchestration: the agent handles dynamic tool selection and multi-step remediation instead of encoding many brittle conditional branches in the data pipeline [1].
• Better separation of concerns: streaming infrastructure manages throughput and parallelism, while the agent layer manages reasoning and action selection.

For executives, the lesson is that enterprise AI cost control is primarily an architecture problem, not only a model-pricing problem. The highest-leverage decision is often where the LLM sits in the workflow and what is allowed to reach it.

Kimbodo Engineering Perspective

This architecture is directionally correct for production AI systems: do not put an LLM on the hot path unless the task requires language reasoning, planning, tool selection, or synthesis. Many enterprise workflows contain a mix of simple classification, policy checks, entity extraction, retrieval, and exception handling. Treating all of those as LLM tasks is expensive and difficult to operate.

The Beam/Dataflow pattern is especially useful when the input stream is large, continuous, and operationally important. Dataflow provides distributed execution, scaling, batching, and streaming semantics; Beam’s inference integration lets teams place ML transforms directly in the pipeline rather than bolting on a separate service for every classification step [1].

The main engineering trade-off is between simplicity and control. A single LLM agent can simplify remediation logic, but production systems still need strict boundaries around what the agent can see, decide, and execute. Tool use should be explicit, logged, permissioned, and reversible where possible. The agent should not become an opaque replacement for workflow governance.

Another important trade-off is false negatives. If the upstream classifier incorrectly labels a serious event as POSITIVE or NEUTRAL, the remediation agent never sees it. That may be acceptable for low-risk customer feedback triage, but not for fraud, safety, regulated complaints, outages, or financial exceptions. In high-risk domains, the pre-filter should use confidence thresholds, escalation rules, sampling, and audit queues rather than hard drops.

We would also be careful with email-sending agents. Gmail or similar notification tools are useful, but any system that sends external communications needs approval policies, templates, rate limits, suppression lists, and human review for sensitive cases. The LLM can draft or select the response path, but the enterprise application should enforce communication rules.

How We Would Implement It

1. Classify Events Before Agent Invocation

We would place a low-cost classifier early in the pipeline. Depending on the use case, this could be a small transformer model, gradient-boosted model, rules engine, embedding similarity check, or a combination. The goal is not to make the final business decision; it is to route traffic intelligently.

• Run high-volume classification on CPU where possible.
• Use batch inference in the streaming framework to improve throughput.
• Emit classification label, confidence score, model version, and routing decision.
• Route only qualified events to the agentic path.

2. Use Confidence-Based Routing, Not Only Labels

A production version should avoid simple label-only routing. For example:

• NEGATIVE with high confidence: send to the remediation agent.
• NEGATIVE with low confidence: send to review queue or secondary model.
• POSITIVE or NEUTRAL with high confidence: suppress or archive.
• Any event containing regulated terms, VIP accounts, legal language, threats, outage signals, or refund requests: override and escalate.

This reduces cost while avoiding excessive dependence on one lightweight classifier.

3. Isolate the Agent Layer

The LLM agent should run as a bounded decisioning component with explicit tools. In the pattern described by Google, the agent can look up users, look up orders, check inventory and prices, and send email through configured tools [1]. We would implement those tools as typed service interfaces rather than giving the model broad database or API access.

• Expose narrow tool functions with strict input schemas.
• Validate every tool argument server-side.
• Apply authorization at the tool layer, not only in the prompt.
• Return minimal data needed for the remediation decision.
• Log every tool call, response summary, latency, and decision.

4. Keep Workflow State Outside the Prompt

The pipeline should store durable state in operational systems: event logs, BigQuery tables, case management systems, or workflow databases. The LLM prompt should receive only the current task context and tool results needed for that step.

This makes the system easier to audit, replay, and debug. It also reduces token cost and limits exposure of sensitive data.

5. Add Observability Across the Full Path

Teams should monitor the entire AI workflow, not only model latency. Key metrics include:

• Input event volume.
• Classifier label distribution.
• Classifier confidence distribution.
• Percentage of events routed to the LLM.
• LLM token usage and cost per event type.
• Tool-call success and failure rates.
• Remediation completion rate.
• Human override rate.
• False-positive and false-negative samples from audits.

6. Use Progressive Rollout

We would not deploy automated remediation at full autonomy on day one. A safer rollout path is:

• Shadow mode: run the classifier and agent without taking action.
• Recommendation mode: show proposed remediation to human operators.
• Guarded automation: allow automated action for low-risk cases only.
• Expanded automation: increase scope after measuring accuracy, cost, and incident rates.

Risks, Costs and Security

The architecture reduces unnecessary LLM calls, but it introduces its own operational and governance requirements.

• Misrouting risk: a lightweight classifier can suppress events that should have reached the agent. Mitigation requires confidence thresholds, override rules, sampling, and periodic evaluation.
• Agent overreach: an LLM with tool access can take incorrect or excessive actions. Tools should be narrow, permissioned, logged, and protected by business rules.
• Data leakage: customer messages, order history, and email addresses may contain sensitive information. Minimize prompt context, redact unnecessary fields, and enforce data access controls.
• Email and notification abuse: automated communication requires rate limits, approved templates, suppression controls, and review workflows for sensitive cases.
• Cost drift: if classifier thresholds change or input distribution shifts, more events may reach the LLM. Track routing percentage and cost per thousand events continuously.
• Model drift: sentiment, complaint language, and customer behavior change over time. Retrain or recalibrate the pre-filter based on production samples.
• Dependency risk: the workflow depends on model APIs, BigQuery, Gmail, and orchestration infrastructure. Use retries, dead-letter queues, idempotency keys, and graceful degradation.

The core production lesson is simple: LLMs should be used where they create differentiated value, not where cheaper infrastructure can make an adequate routing decision. A hybrid design using CPU inference for high-volume filtering and agents for sparse, high-value remediation can reduce cost, latency, and operational load while preserving flexibility for complex cases [1].

Sources

1. [1] Building cost-effective, high-throughput gen AI workflows in Google Dataflow