Cut RAG Costs and Hallucinations by Precomputing Structured Context and Using Hybrid Search

What Happened

Production teams running retrieval-augmented generation (RAG) and LLM agents found that starting agents without domain-specific context drives exploratory tool calls, higher latency, token cost and hallucinations. The practical fix is to assemble a structured, precomputed context layer from catalog, policy and session signals and provide it to the model before the agent’s first action. Assembled in parallel, this replaces multiple sequential tool calls and produces deterministic, governed behavior with much lower token usage and fewer errors. A documented experiment reported up to 75% input‑token reduction and accuracy increasing from 60% to 92% on a hard retail benchmark when this pattern was applied [1].

Key components of the approach include a semantic metadata/vocabulary index to map customer language to canonical fields, a human‑authored policy layer (age checks, routing, seasonal weightings), live facet state to avoid recommending unavailable items, and profile/session signals for personalization and deduplication [1]. The recommended rollout is phased: Phase 1 — vocabulary/semantic metadata; Phase 2 — facet state and policies; Phase 3 — profiles/session; Phase 4 — feedback governance and continuous learning [1].

Why It Matters to Businesses

• Lower operational cost: Fewer LLM calls and fewer input tokens directly reduce model spend (the reported experiment showed large token reductions) [1].
• Better accuracy and fewer hallucinations: Precomputed facts (inventory, policies, canonical vocab) constrain the model, reducing incorrect recommendations or unavailable product suggestions [1].
• Improved latency and UX: Parallel prefetching of context replaces multiple ~600ms–1s sequential tool calls, producing faster agent responses and fewer refresh cycles [1].
• Governance and compliance: A policy/percolation layer lets teams enforce rules (age, promotions, routing) consistently across automated agents.
• Operational observability: Structured context and deterministic prefetching make it possible to log and audit why an agent made a decision, simplifying troubleshooting and fairness checks.

Kimbodo Engineering Perspective

When building RAG for production we prioritize determinism, observability and operational cost. The following practical judgments reflect trade‑offs we apply on client projects:

• Hybrid search first: Combine dense vector similarity with keyword/aggregation capabilities (hybrid search) to handle exact-match filters, facets and vocabulary mapping. Pure vector similarity alone is brittle for inventory/facet scenarios.
• Precompute canonical metadata: Build and version a semantic metadata/vocabulary index (colors, brands, product types) so the model operates on canonical fields, not free text. This dramatically reduces hallucinated filters and misinterpretations [1].
• Parallel prefetching: Assemble multiple context slices (vocabulary mapping, live facets, user profile) in parallel to avoid sequential tool-call latency. Cache the assembled context per session where freshness permits.
• Chunking and embeddings: Use chunk sizes that preserve semantic coherence for retrieval (not arbitrarily small). Embed chunks with stable, production‑grade encoders and store embeddings alongside rich metadata to enable deterministic filtering.
• Control plane vs data plane: Separate fast, ephemeral context (live facet counts, session) from slower canonical metadata (vocabulary, policies). This reduces write contention and lets you tune freshness per signal.
• Orchestration tool choice: Use LlamaIndex or LangChain for building retrieval + prompt pipelines, but keep the retrieval and policy enforcement outside the LLM process so you can audit and scale them independently.
• Measure the right metrics: Track token cost per user task, retrieval precision@k, hallucination rate (golded examples), latency percentiles, and business KPIs (conversion, returns, support escalations).

How We Would Implement It

Below is a compact, production-ready architecture and stepwise rollout we use at Kimbodo.

Architecture (components)

• Ingest / ETL: Pipeline to normalize product/catalog data, policies, and user events into canonical entities and metadata.
• Embedding service: Hosted encoder (managed or self‑hosted) that produces stable embeddings; run in batch for canonical docs and incrementally for new content.
• Vector store + metadata store: Vector DB (Qdrant, Pinecone, Weaviate, Milvus) for similarity queries with rich per-document metadata. Use a search engine like Elasticsearch or Vespa alongside for keyword, aggregations, percolation and live facet counts.
• Context assembly service: Microservice that concurrently fetches semantic vocabulary mappings, top‑K retrievals, live facet state, and user profile/session history and returns a single structured context blob to the agent.
• Policy/percolation engine: Rule engine (Elastic percolator or a small policy service) to enforce age, routing and promotion rules before results reach the LLM.
• LLM orchestration: LlamaIndex or LangChain to build the prompt template that consumes the precomputed context and retrievals; model calls happen after context injection.
• Cache and CDN: Redis or similar for per-session context caching and TTLs to reduce repeated assembly.
• Observability: Structured logging of context blobs, retrieval IDs, chosen documents, user signals and model responses for auditing and A/B testing.

Concrete steps (phased rollout)

• Phase 0 — Prototype: Build a minimal pipeline using Elastic Cloud or a managed vector DB and implement a context assembler that supplies vocabulary mappings and top retrievals to an LLM via LangChain or LlamaIndex. Validate retrieval precision and token usage.
• Phase 1 — Semantic metadata: Deploy a canonical vocabulary index and ensure the context assembler maps customer language to canonical fields (brands, colors, categories) [1].
• Phase 2 — Facet state & policies: Add live facet counts and a policy/percolator layer to avoid zero-result recommendations and enforce rules [1].
• Phase 3 — Profiles & session signals: Include user history, cart state and personalization signals; introduce deduplication logic.
• Phase 4 — Governance and continuous feedback: Hook logging into labeling and retraining loops, monitor hallucinations, and iterate on vocabulary and policy rules [1].

Risks, Costs and Security

Adopting RAG with vector stores and precomputed context reduces many operational risks but introduces others that must be managed explicitly.

• Data leakage / PII: Embeddings can leak sensitive data. Never embed raw PII without anonymization; apply access controls, encryption at rest and in transit, and minimal retention policies.
• Retrieval poisoning: Malicious or corrupted documents can push harmful content into top-K results. Use provenance checks, signed ingest pipelines and anomaly detection on similarity distributions.
• Freshness vs cost: Live facet counts and session signals require low-latency updates; choose TTLs and cache invalidation carefully to balance cost and relevance.
• Model & embedding cost: Embeddings and LLM calls are recurring costs. Precompute and cache embeddings for static content; batch embeddings for new items and use retrieval filtering to limit LLM context size.
• Vendor lock-in: Vector DBs and managed LLMs differ in APIs and feature sets. Abstract retrieval and embedding layers behind a thin internal interface to allow migration.
• Security posture: Apply RBAC on vector stores and search indices, audit logs for access, rate‑limit public APIs, and sanitize user inputs to prevent prompt-injection via retrieved content.
• Compliance and auditability: Keep signed, versioned context blobs and retrieval traces so you can explain model outputs for regulatory or legal reviews.

In short: implement hybrid search + semantic metadata, precompute and parallelize context assembly, enforce policies before the LLM sees results, and instrument aggressively for cost and safety. The approach reduces tokens, latency and hallucinations while creating an auditable pipeline you can operate at scale [1].

Sources

1. [1] AI shopping agents: Why context comes before the query