Build Production-Grade RAG: Practical Choices for Vector Databases, Hybrid Search and Safe ES|QL Behavior

What Happened

Elasticsearch 9.5 introduced an unmapped_fields option for ES|QL that prevents queries from failing when they reference fields missing from index mappings. The option accepts NULLIFY (return NULLs) or LOAD (read values from _source) and resolves partially unmapped non-keyword fields (PUNKs) by injecting an “unmapped” field into the query plan so the planner and optimizer can continue unchanged. To avoid mixed-type read errors, fully unmapped fields default to KEYWORD and partially unmapped legs are implicitly cast; optimizer passes limit unsafe Lucene filter pushdowns when some shards lack mappings, restoring fast paths where shards fully map a field. The rollout included expanded CSV spec tests and generative testing, with AI-assisted test generation used to scale coverage [1].

Why It Matters to Businesses

• Operational stability: Queries referencing newly introduced or inconsistently mapped fields no longer cause runtime errors or force immediate reindexing, reducing incident risk during schema evolution [1].
• Faster ingestion and schema agility: Teams can add fields, ingest heterogeneous sources, or iterate chunking and metadata schemes without costly reindex windows.
• Lower maintenance cost: Avoiding frequent reindexing reduces I/O, compute and downtime for large document bases — material for budget-constrained production RAG systems.
• Deterministic behavior for RAG pipelines: When retrieval queries silently fail or error on unmapped fields, downstream LLM reranking and generation produce worse results. NULLIFY/LOAD stabilizes retrieval and reduces hallucination risk from unpredictable empty recalls [1].

Kimbodo Engineering Perspective

Building RAG systems requires balancing recall, latency, cost and operational resilience. The ES|QL change is a practical improvement for teams that rely on Elasticsearch for metadata, hybrid search or as a combined sparse+dense platform, but it’s one part of a broader engineering surface. Our core judgements:

• Use the right tool for intent: Choose managed vector DBs (Pinecone, Weaviate) for rapid delivery, self-hosted engines (Qdrant, Milvus) for cost and control, and search engines (Elasticsearch, Vespa) when you need rich text analytics, aggregations or global consistency across sparse+dense queries.
• Design for schema evolution: Treat metadata mappings as first-class artifacts. Where possible, default to permissive behavior (using NULLIFY/LOAD in ES when acceptable) while adding monitoring and tests to detect semantic type drift [1].
• Prefer hybrid retrieval: Combine sparse (BM25/Elasticsearch/Vespa) and dense (vector DB) to improve recall for exact-match and semantic intents; rerank with a cross-encoder when precision matters.
• Measure recall vs latency trade-offs: Tune ANN index parameters (HNSW ef/efConstruction, IVF nlist, quantization) to hit SLOs; favor more CPU/TPS for lower latency, more RAM/SSD for higher recall under budget constraints.
• Testing matters: Use synthetic and generative tests to exercise partially-mapped schemas and retrieval failure modes — the same approach used to validate ES|QL changes scaled coverage and reduced regressions [1].

How We Would Implement It

Reference architecture

• Ingestion layer: connectors → dedupe → canonicalization → chunking (adaptive lengths based on document type) → metadata extraction and normalization.
• Embedding service: batched embedding calls to chosen model (self-hosted or API) with deterministic pre- and post-processing (lowercasing, normalization, stopword policies as needed).
• Storage & indexing:
  • Primary document store: object storage (S3) or DB with stable IDs and full _source retention.
  • Metadata & sparse search: Elasticsearch or Vespa when you need full-text, filtering, aggregations, and analytics.
  • Vector index: choose between:
    • Managed (Pinecone, Weaviate) for ops simplicity and auto-scaling.
    • Self-hosted (Qdrant, Milvus) when you need cost control, local data, or custom metric support.
    • Elasticsearch/Vespa when you want tight hybrid integration plus ES|QL behavior improvements (use ES unmapped_fields to avoid reindexing pain) [1].
• Retrieval pipeline: sparse recall → dense recall (vector DB) → union/merge with provenance, filter by metadata → reranker (cross-encoder or cheap intent classifier) → LLM input construction.
• Generation: LLM with soft and hard constraints, citation insertion using provenance tokens, hallucination detectors and fallback to source snippets.
• Observability & control: logs, latency and recall SLOs, drift detection, vector index integrity checks, and end-to-end test suites with synthetic queries.

Concrete steps and configurations

• Start with schema templates for metadata; version them and include migration plans.
• If using Elasticsearch for metadata and hybrid search:
  • Enable unmapped_fields=LOAD for development and staging to reduce reindex churn; use NULLIFY in critical production paths where silent NULLs are safer than on-demand _source reads [1].
  • Monitor query rewrite logs (ResolveUnmapped traces) during rollout to confirm behavior matches expectations [1].
• Choose vector DB index parameters based on benchmarks:
  • For HNSW-based systems (Qdrant, Faiss HNSW): tune efConstruction for build time vs recall, ef for query-time recall/latency; document target ef and run load tests.
  • For IVF/OPQ (Milvus, Faiss): choose nlist by dataset size and retrain quantizers periodically.
• Implement a provenance model: store (document_id, chunk_id, score, distance, retrieval_method) and include raw snippets in generation context to enable citation and auditability.
• Reranking & safety: use a cross-encoder reranker when precision matters; enforce token-level match checks for facts drawn from retrieval before accepting LLM assertions.
• CI/CD & tests: add generative tests that mutate schema and sample documents to exercise partially-mapped fields and retrieval fallbacks — mirror the approach used in ES|QL testing to surface edge cases early [1].

Risks, Costs and Security

• Costs: Embedding model calls and vector storage can dominate costs. Use batching, caching, dimensionality reduction (carefully), and TTL policies for ephemeral content. Managed services trade higher per-query cost for reduced ops.
• Operational risk: Index rebuilds are expensive. Use schema evolution strategies (like ES unmapped_fields) and blue-green index migrations to reduce downtime and operational burden [1].
• Accuracy & drift: Embedding model updates, data drift and inconsistent metadata types cause retrieval regressions. Track recall metrics and run periodic offline reembedding experiments before rollouts.
• Security & compliance:
  • Enforce network isolation, private VPC endpoints and strict RBAC for vector DBs and embedding APIs.
  • Encrypt embeddings and documents at rest; ensure key management (KMS) and audit logs for access.
  • Implement PII detection and redaction before embedding if sensitive data is involved; maintain deletion and data retention workflows that include vector index compaction and tombstones.
• Model safety: Use output filtering, provenance checks and human-in-the-loop escalation for high-stakes domains. Log generation inputs and outputs for incident analysis.
• Vendor lock-in: Avoid proprietary vector formats in core downstream assets; keep canonical document store and re-embed scripts so you can migrate vector indexes if needed.

In short: treat retrieval as an engineering-first subsystem — design for schema evolution, measure recall/latency, and place reranking and provenance close to the LLM. Use Elasticsearch’s unmapped_fields behavior to reduce reindex churn during schema change, and combine managed and self-hosted components according to your ops capability and cost profile [1].

Sources

1. [1] Taming PUNKs: How ES|QL queries Elasticsearch fields it was never told about