How to Build Enterprise AI Agents That Preserve Provenance, Control Cloud Costs, and Stay Secure

What Happened

A recent AI-assisted workflow used ChatGPT with GPT-6 Astra to generate looped 5K and 10K running routes from a home location. The agent geocoded the start point with Nominatim, fetched roads and trails from OpenStreetMap via Overpass, calculated local loops, and produced an embedded visualization plus downloadable GPX and GeoJSON artifacts. One job ran for about 27 minutes and produced a 5.1 km route with a shareable HTML map [1].

The important production lesson was not the route itself. After the conversation was compacted, the assistant could explain the general workflow but the exact executed code and intermediate steps were no longer accessible in the UI [1]. For enterprise AI systems, that is a serious gap: users need reproducibility, auditability, and artifact-level provenance, especially when agents run tools, call external APIs, or generate files used in business decisions.

The visualization also showed a practical security pattern. The output embedded full route geometry in a local JSON script block and rendered the map with D3 from an allow-listed CDN. The content security policy permitted only specific external origins, with other origins blocked [1]. That is the right direction for enterprise AI applications: generated artifacts should be useful, but constrained by default.

Separately, Hugging Face’s security guidance includes a message directed at AI agents, advising researchers to use the public CyberGym benchmark rather than attempting to hack Hugging Face [3]. This reflects a new operational reality: autonomous and semi-autonomous AI systems can cause security incidents if their tool access, objectives, and network permissions are not tightly governed.

Paul Ford’s observation remains relevant: LLMs can produce good code, but skilled software still requires human judgment, collaboration, and craft. AI also makes it easier to produce poorly executed systems at scale [2].

Why It Matters to Businesses

Enterprise AI platforms are moving from chat interfaces to tool-using systems that create artifacts, query data, call APIs, run code, and trigger workflows. That changes the architecture requirements. A model response is no longer enough; the platform must preserve what happened, why it happened, which tools were called, which inputs were used, and which outputs were produced.

For business and technology leaders, the core issue is accountability. If an AI agent generates a financial forecast, customer segmentation, legal summary, infrastructure change plan, or route optimization file, the organization needs a durable record of the execution. Conversation compaction can reduce token costs and improve usability, but it must not destroy operational evidence.

The main implications are:

• Auditability: Every tool call, prompt version, model version, input file, external API request, and generated artifact should be recoverable.
• Reproducibility: Teams need enough execution detail to rerun or validate important outputs, even after chat context is summarized or compacted.
• Cost control: Long-running AI jobs can consume model tokens, compute, external API quota, storage, and engineer time. A 27-minute agent run is not just a chat interaction; it is a distributed workload [1].
• Security: Tool-using agents need egress controls, identity boundaries, content policies, secrets management, and abuse prevention.
• Reliability: AI-generated code and workflows can be useful, but they require production engineering practices: testing, monitoring, rollback, and human review for high-impact actions [2].

Kimbodo Engineering Perspective

The most important design principle is to separate conversation memory from execution provenance. Conversation memory can be summarized, compacted, or personalized. Execution provenance should be immutable, queryable, and retained according to business policy.

In production systems, we would not rely on a chat transcript as the system of record. The transcript is a user experience layer. The system of record should be an append-only execution log with references to prompts, tool calls, environment metadata, input datasets, generated code, intermediate outputs, final artifacts, and policy decisions.

Compaction Is Useful, but It Cannot Be Destructive

Context compaction is necessary because long-running agent sessions become expensive and exceed model context limits. However, compaction should produce a summary for model continuity while preserving the original pre-compacted content in durable storage. Users and authorized tools should be able to retrieve exact prior text, code, tool inputs, and outputs through controlled APIs.

Artifacts Need First-Class Lifecycle Management

Generated files such as HTML visualizations, GPX files, GeoJSON, CSV exports, notebooks, SQL queries, and infrastructure plans should be treated as managed artifacts. They need object storage, metadata, checksums, retention rules, access controls, and links back to the execution that created them.

Agent Tooling Should Be Narrow and Observable

Giving an LLM broad network and code execution access is not an architecture; it is an incident waiting to happen. Tool access should be brokered through a gateway that enforces allow-lists, rate limits, data policies, authentication scopes, and logging. The Hugging Face security message aimed at AI agents is a useful reminder that autonomous systems can accidentally cross boundaries if their goals and permissions are underspecified [3].

Human Craft Still Determines Production Quality

LLMs can accelerate development, generate useful code, and automate repetitive tasks. They do not remove the need for architecture, threat modeling, test design, data governance, or operational ownership. As Ford argues, AI can help produce software, but it can also amplify low-quality execution [2]. The differentiator is not whether a company uses AI; it is whether it engineers AI systems with discipline.

How We Would Implement It

For an enterprise AI platform that runs tool-using agents, Kimbodo would design the system around durable execution, governed tools, asynchronous orchestration, and artifact provenance.

1. Use an Agent Orchestrator with Durable State

The orchestrator should manage plans, tool calls, retries, model invocations, user approvals, and job state. Long-running jobs should not depend on an active chat session. Use a workflow engine or queue-backed service for multi-step tasks, with resumability and idempotency built in.

• Store every execution as a job with a unique ID.
• Persist each model call, tool call, input, output, error, and approval event.
• Support pause, resume, cancel, retry, and replay.
• Separate user-visible summaries from the full execution record.

2. Preserve Provenance in an Append-Only Event Log

Every meaningful action should produce an event. For example: prompt submitted, model selected, tool authorized, geocoder called, Overpass query executed, route generated, visualization rendered, artifact stored, CSP applied, and user download requested. The event log becomes the audit trail and the backbone for debugging.

• Store prompt templates and prompt variables separately.
• Record model name, model parameters, system instructions, and policy context.
• Capture tool inputs and outputs with redaction where needed.
• Link events to artifact IDs and object storage locations.
• Expose provenance through an internal API and user-facing “view execution details” interface.

3. Treat Generated Code as an Artifact, Not a Hidden Side Effect

If an agent writes or executes code, the platform should save the exact code, dependencies, runtime image, environment variables used, and execution logs. If the code is modified during retries, each version should be retained. This directly addresses the failure mode where a user can see the final answer but cannot retrieve the code that created it [1].

4. Run Tools in Sandboxed Execution Environments

Code execution should run in short-lived sandboxes with constrained CPU, memory, filesystem, network, and runtime permissions. For cloud deployments, this can be implemented with isolated containers, Kubernetes jobs, Firecracker-style microVMs, or managed batch workloads depending on the sensitivity and performance requirements.

• Default to no outbound network access.
• Allow egress only through a policy-controlled proxy.
• Inject secrets just-in-time and scope them to the job.
• Destroy the runtime after execution.
• Persist only approved artifacts and logs.

5. Use a Tool Gateway for External APIs

External services such as geocoders, data warehouses, CRMs, ticketing systems, and cloud APIs should be accessed through a tool gateway rather than directly from the model or sandbox. The gateway should enforce authentication, authorization, schema validation, rate limits, egress policy, and logging.

In the route-generation example, calls to Nominatim and Overpass would be mediated by this gateway, with query payloads, response metadata, cache hits, and quota usage recorded [1].

6. Store Artifacts in Object Storage with Metadata

Generated GPX, GeoJSON, HTML, charts, reports, and logs should be stored in object storage with immutable version IDs where appropriate. Metadata should include creator job, user, tenant, model, tool versions, data sources, checksum, classification, expiration, and access policy.

For visualizations, the platform should apply a restrictive content security policy. The example of allowing only specific CDN origins and blocking other external origins is a practical pattern, but many enterprises should go further by self-hosting approved JavaScript assets to reduce supply-chain exposure [1].

7. Add Evaluation, Testing, and Release Controls

Production AI workflows need MLOps and software delivery controls. That includes prompt versioning, regression tests, tool contract tests, golden datasets, offline evaluations, human review queues, and staged rollouts. For high-impact use cases, model upgrades should be treated like application releases, not silent substitutions.

• Maintain a model and prompt registry.
• Run evaluations before changing models or system prompts.
• Track quality, latency, cost, and safety metrics by workflow.
• Use canary deployments for agent behavior changes.
• Provide rollback paths for prompts, tools, and models.

8. Build Cost Controls into the Architecture

Long-running agent workloads can become expensive quickly. Cost controls should be implemented at the platform level rather than left to individual application teams.

• Route simple tasks to smaller or cheaper models.
• Use larger models only for planning, reasoning, or high-value steps.
• Cache deterministic tool results where licensing and freshness requirements allow.
• Set per-tenant budgets and per-job spending limits.
• Emit cost telemetry for tokens, compute, storage, API calls, and human review.

Risks, Costs and Security

The main risk in enterprise AI infrastructure is treating agent execution as an extension of chat. It is not. It is distributed computing with probabilistic planning, external tools, generated code, and business data. That requires conventional engineering controls plus AI-specific safeguards.

Key Risks

• Lost provenance: Compaction, summarization, or UI limitations can make it impossible to reconstruct what happened unless execution data is stored separately [1].
• Unbounded tool use: Agents may call APIs repeatedly, trigger rate limits, create excessive cloud cost, or interact with systems outside their intended scope.
• Prompt injection: Data fetched from external sources can contain instructions that attempt to redirect the agent or exfiltrate information.
• Generated-code defects: LLM-written code may appear correct while hiding edge-case failures, security flaws, or licensing issues [2].
• Accidental security probing: Agents given open-ended cyber or research goals may interact with real targets instead of approved benchmarks, a concern reflected in Hugging Face’s AI-agent-facing security guidance [3].
• Supply-chain exposure: Generated HTML that loads external scripts must be constrained by CSP, dependency allow-lists, and preferably self-hosted assets for sensitive environments.

Cost Trade-Offs

• Full provenance increases storage cost but reduces audit, debugging, compliance, and incident-response cost.
• Sandboxed execution adds infrastructure overhead but materially reduces blast radius.
• Self-hosting models can improve data control but may cost more operationally than managed APIs unless utilization is high and predictable.
• Managed LLM APIs reduce platform burden but require careful vendor, data retention, latency, and cost governance.
• Human review slows automation but is appropriate for irreversible, regulated, or high-value actions.

Security Controls We Would Require

• Tenant isolation across data, prompts, artifacts, logs, and vector stores.
• Role-based and attribute-based access control for users, agents, tools, and artifacts.
• Egress allow-lists enforced outside the model runtime.
• Secrets managed through a vault with short-lived credentials.
• Immutable audit logs for model calls, tool calls, approvals, and artifact access.
• Content security policies for generated web artifacts.
• Network isolation for sandboxes and batch jobs.
• Prompt-injection detection and tool-output sanitization.
• Security testing using approved environments and benchmarks rather than live third-party targets [3].

The practical lesson is straightforward: enterprise AI platforms should be designed as production systems from the start. Models generate value, but the durable advantage comes from orchestration, provenance, security, cost control, and disciplined engineering.

Sources

1. [1] Generating running routes with GPT-6 Astra and ChatGPT Work
2. [2] Quoting Paul Ford
3. [3] Quoting huggingface.co/security.txt