Stop Hallucinations and Cut Latency in RAG — Retrieval Architecture, Vector DB Choices, and Reranking Playbook

What Happened

Recent work shows that retrieval — not the generator — is the primary failure point in retrieval-augmented generation (RAG) and agent workflows: missing or low-quality candidates cause hallucination, context rot, excessive tool calls and unpredictable latency. Improving search quality reduces downstream errors and tool usage, but “improve retrieval” is too vague: you must trace and evaluate each retrieval step (query construction, filtering, BM25/semantic matching, reranking, API/SQL calls, context assembly) to know what to fix [1].

Systems that combine hybrid retrieval (term + semantic), multi-stage ranking, and typed rerankers show measurable gains. Example: applying TypeSafe’s Jev reranker on Elasticsearch hybrid candidates closed ~25% of the gap to perfect ranking (nDCG@10 +0.0214; Exact MRR improvements) by returning structured probabilities/signals that the app blends into a final ranking; latency and pricing are practical for production (70–500 ms observed, published low per‑token pricing) [2].

Operationally, at scale you must handle millions–billions of chunks, enforce permissions/freshness, control token budgets via strict candidate bounds and summaries, and capture fine-grained traces for stepwise debugging and labeling. Engines like Vespa illustrate the model: store chunked units with stable fields, support hybrid NN/term queries, phased ranking, document summaries and query tracing while leaving tenant filters and label collection to the app [1].

Why It Matters to Businesses

Search quality directly affects:

• Answer correctness and trust: better retrieval reduces hallucinations and downstream latency by reducing unnecessary tool/model calls.
• Cost control: fewer generator tokens and fewer tool invocations lower per-query cost at scale.
• Latency and SLA compliance: multi-stage retrieval and tight candidate bounds control tail latency for interactive agents and assistants.
• Compliance and safety: accurate retrieval with permissions/freshness reduces regulatory risk from returning stale or unauthorized data.
• Operational visibility: tracing every retrieval step enables focused investments (tune query builders vs. ranker vs. generator) rather than broad, wasted changes.

Kimbodo Engineering Perspective

Practical judgments and trade-offs we apply when building production RAG systems:

• Multi-stage retrieval is essential. Cheap first-pass filters (BM25, metadata, time windows) followed by semantic ANN and a reranker balances recall, precision and cost. Early filtering shrinks candidate sets before expensive scoring or generator use [1].
• Choose the right storage/compute mix. Vector DBs (Pinecone, Qdrant, Milvus, Weaviate) are great for fast ANN at medium scale; Elasticsearch/Vespa provide hybrid term + NN and richer query languages and phased ranking when you need deterministic filtering, custom rank profiles, and query tracing [1].
• Typed rerankers beat free-text scoring for production. Rerankers that emit structured probabilities/signals (e.g., Jev) let you combine business constraints (compatibility, exact match, freshness) explicitly and improve metrics with predictable latency and cost characteristics [2].
• Operationalize tracing and labels first. If you can’t inspect each retrieval step’s inputs, outputs and labels, you’ll be guessing whether to tune queries, ranking or generation. Instrumentation costs are small relative to repeated model experiments [1].
• Keep app-level policy outside the store. Tenant filters, RBAC, time windows and context budgets should be enforced by the application layer so indexes remain stable and embeddable across use cases [1].
• Use orchestration frameworks strategically. LlamaIndex and LangChain are useful for rapid prototyping and agent orchestration; Haystack adds pipeline primitives. For production at scale, wrap these with hardened logic for tracing, retries, timeouts and security.

How We Would Implement It

Architecture Blueprint

• Ingest → chunk → embed → index (vector + metadata) into a hybrid-capable store (Elasticsearch/Vespa for heavy filtering & tracing; Pinecone/Qdrant/Milvus/Weaviate when a managed ANN focus is preferred).
• Query planner: generate structured retrieval requests (term filters, time/freshness, permission masks, semantic query embedding).
• Multi-stage retrieval: cheap filter → ANN/top-k → reranker (typed signals) → summarizer/concat to generation context with strict token budget.
• Tracing & label store: capture each retrieval step (query, candidate list, reranker signals, relevance labels) into a trace database for offline analysis and supervised reranker training [1].

Concrete Implementation Steps

• 1. Chunking and canonical fields: store stable chunk IDs, source ID, timestamp, permissions, metadata and text summary alongside embeddings. Use chunk sizes tuned to your domain (shorter chunks for code/tickets, longer for policies) [1].
• 2. Embeddings: pick a single embedding model family and freeze it; compute embeddings at ingest and store proactively. For mixed workloads, support dual-embeddings (semantic + dense features) but prefer one canonical vector per chunk for simplicity.
• 3. Index choice: if you need deterministic text filters, phased ranking, complex query DSL and tracing, use Vespa or Elasticsearch with ANN capabilities; if you prioritize managed scaling or simple ANN, choose Pinecone/Qdrant/Milvus/Weaviate.
• 4. Retrieval pipeline: implement early metadata/time/permission filters → BM25 or multi_match as first pass (when applicable) → semantic ANN (top-N tuned for recall) → send top candidates as structured state to a typed reranker (e.g., Jev schema: relationship and Noul signals) [2].
• 5. Reranking and signal blending: prefer rerankers that return probabilities/typed outputs. Blend signals (exact probability, expected utility, compatibility flags) into a composite ranker and tune by business metric (nDCG/MRR for search, task success for agents) [2].
• 6. Context assembly: select winners within a strict token budget, optionally replace candidates with model summaries to increase information density; enforce recency and permissions at assembly time [1].
• 7. Instrumentation and labeling: record every retrieval step, model inputs/outputs, reranker signals and user feedback. Use traces to decide whether to tune query builders, embeddings, the ranker or the generator [1].
• 8. Metrics and rollout: track nDCG, MRR, tool-call count, generator token usage, latency percentiles and task success. A/B reranker strategies and conservative canary rollouts reduce regressions [2].

Risks, Costs and Security

Key risks and mitigations:

• Hallucination and incorrect grounding: Occurs when retrieval misses relevant chunks. Mitigate by tighter filters, higher recall ANN top-N, stronger reranking and model summaries; instrument to detect generator contradictions to source documents [1].
• Data leakage & PII exposure: Embeddings can leak information. Avoid storing raw PII in indexable text, apply redaction, encryption-at-rest, and access controls. Consider techniques like vector encryption or limiting embedding persistence for sensitive data.
• Index poisoning and adversarial queries: Monitor query patterns and implement rate limits, anomaly detection, and provenance checks. Use signed ingestion and immutable chunk IDs to prevent unauthorized inserts.
• Latency and cost: ANN search, reranker inference, and summarization add cost/latency. Control cost by early filtering, caching hot queries, precomputing summaries, limiting reranker calls to top-k and observing service latency budgets (target 95–99th percentile within SLA). Jev-like rerankers have practical latencies (70–500 ms) and economical pricing models to consider [2].
• Compliance and auditability: Keep query traces, access logs and relevance labels for audits. Enforce tenant isolation at the app layer and use field-level encryption where necessary [1].

Cost considerations:

• Storage: vector index size scales with chunk count; plan for sharding and retention/TTL strategies.
• Compute: embedding generation at ingest and reranker inference are the dominant CPU/GPU costs; batch embeddings and warm caches to amortize cost.
• Operational: engineering time for trace instrumentation, labeling, and retraining rerankers is often the largest ongoing expense but yields outsized accuracy improvements.

Sources

1. [1] Why retrieval quality is becoming the defining challenge in AI agent architecture
2. [2] Using Jev as a search reranker: benchmarks and how to implement