Cut LLM Token Costs in Large-Scale Code Migrations with Sandboxed Search-API Scripts

What Happened

Sourcegraph published an approach that runs scripted audits inside a sandbox which calls Sourcegraph search APIs, and emits a single compact CSV checklist summarizing migration findings instead of returning thousands of files into an LLM context [1]. The sandboxed scripts do analysis near the search layer; the audit artifact is token-efficient and designed to minimize LLM context size while preserving actionable results [1].

Why It Matters to Businesses

This pattern directly addresses three commercial pain points for teams doing large-scale code changes or compliance audits:

• Token and compute cost reduction: Producing a single CSV checklist shrinks the data sent to LLMs, lowering inference cost and latency for downstream AI assistants and automation.
• Operational scalability: Moving analysis into sandboxed scripts that call search APIs avoids materializing large result sets, enabling audits on massive repositories without blowing up memory or token budgets.
• Actionable, auditable outputs: A compact, machine-readable artifact (CSV) integrates readily with CI/CD, ticketing, and governance workflows for traceability and remediation.

Kimbodo Engineering Perspective

Why this pattern is practical

We view the sandboxed-script + compact-artifact pattern as an effective middle ground between full LLM-heavy analysis and brittle rule-only tooling. It keeps expensive LLM usage focused on summarization, decisioning, and triage, while using deterministic script logic for bulk search and filtering.

Trade-offs

• Pros: Lower LLM cost, repeatability, easier CI integration, smaller provenance surface for LLM inputs.
• Cons: Scripts must be maintained and tested; compact artifacts (CSV) can omit nuance (false positives/negatives) requiring downstream verification; adds operational complexity (sandbox orchestration, search API rate limits).

When to use it

• Large repositories or monorepos where raw search outputs are huge.
• Frequent automated audits or migrations where repeated LLM consumption would be expensive.
• Regulated contexts requiring auditable artifacts and traceable remediation steps.

How We Would Implement It

Concrete architecture and steps to implement a sandboxed-search + CSV audit pipeline suitable for production.

Architecture components

• Search API layer: Sourcegraph (or equivalent) providing fast code search and repo graph queries.
• Script sandbox: Lightweight isolation for user-authored scripts (Wasmtime, Firecracker microVMs, or gVisor containers) with network controls limiting calls to the search API and monitored I/O.
• Script runtime: Standardized language (Python/TypeScript) with a small SDK that wraps search responses and enforces resource/time limits.
• Artifact generator: Deterministic CSV (or optionally Parquet/NDJSON) creator with schema versioning and checksums.
• LLM orchestrator: Component that consumes the CSV for summarization, triage, or human-facing recommendations; keeps prompts minimal by referencing the CSV artifact rather than embedding raw search results.
• Storage and provenance: S3-compatible storage with signed artifacts, metadata, and audit logs (use Sigstore/cosign for signing where required).
• CI/CD integration: Jobs that run sandboxed audits on pull requests, nightly runs, or migration plan stages; emit tickets or automated patch proposals.

Implementation steps

1. Define CSV schema up front (finding id, repo, path, line range, rationale tags, severity, remediation suggestion, script version, search query hash).
2. Build a minimal script SDK that enforces timeouts, result-size caps, and telemetry hooks; provide templates for common migration patterns.
3. Deploy sandbox runtime with strict syscall/network policies and per-job quotas; limit sandbox outbound calls to the search API and internal telemetry endpoints.
4. Run scripts so they call Sourcegraph search APIs, perform filtering/heuristics, and write a single CSV artifact to storage.
5. Have an LLM worker that ingests the CSV and generates summaries, prioritization, or patch suggestions—send the CSV reference in prompts instead of embedding file contents to reduce token usage.
6. Integrate artifact signing and retention policies; record script versions and environment snapshots for reproducibility.
7. Include automated verification steps: dry-run, unit tests for scripts, and canary runs on subset repos before full migration.

Risks, Costs and Security

Adopting this approach reduces LLM expense but introduces operational and security considerations:

• Sandbox escape risk: Script execution must be strongly isolated. Use hardened runtimes (gVisor, Firecracker, or Wasmtime) and limit syscalls, filesystem and network access.
• Search API limits and consistency: Heavy use of search APIs can hit rate limits; results depend on index freshness—build throttling and index-health checks into pipelines.
• Loss of nuance: Summaries in a CSV can miss context. Always include file location and snippet references and provide easy drill-down links back to source for human review.
• Provenance and auditability: You must record script version, SDK version, search query hashes, and environment metadata; sign artifacts to defend against tampering.
• Maintenance cost: Scripts and heuristics require ongoing upkeep as code patterns and repo structures change; budget for tests and periodic reviews.
• Access control: Restrict who can run scripts and view CSV artifacts; integrate with org RBAC and secret management to avoid data leakage.

Bottom line: Running deterministic, sandboxed scripts that call code-search APIs and emit compact, signed artifacts is an efficient pattern to reduce LLM token costs and scale code migrations. It trades some upfront engineering and operational work for substantial recurring savings, better CI integration, and improved auditability—if implemented with strong sandboxing, provenance, and verification controls [1].

Sources

1. [1] A smarter way to run code migrations with less LLM context