What Happened
Two Elasticsearch capabilities address different parts of the retrieval problem. ES|QL Data Federation can query Parquet, CSV, TSV and NDJSON files in Amazon S3 without copying or indexing them, and can query those datasets alongside Elasticsearch indices. It requires an Enterprise subscription and is experimental in Elastic Cloud Hosted 9.5 and in technical preview in 9.6 [1].
A separate RAG design pairs Elasticsearch chunk retrieval with SpiceDB relationship-based authorization. A LangGraph workflow retrieves candidate chunks, checks their document-level view permissions in a bulk call, and passes only permitted chunks to the language model. Sharing changes update relationships rather than requiring an Elasticsearch reindex [2].
Why It Matters to Businesses
S3 federation can avoid a second stored copy when teams need to inspect operational data, but it does not replace indexing for every search workload: federated datasets cannot serve as LOOKUP JOIN targets or support time-series functions [1]. For internal RAG, relevance alone is not sufficient. Department-level filters can miss individual sharing rules and exceptions; permission checks must apply to the exact documents retrieved [2].
Kimbodo Engineering Perspective
Keep retrieval, authorization and generation as separate, testable stages. A vector database or search engine should find candidates; it should not be assumed to be the authority for access decisions when permissions change independently. Retrieve more than the desired answer context, then retain the first permitted chunks up to the target count. This improves answer coverage without relaxing access controls [2].
Use federation for exploratory queries and suitable read workloads, with column selection and filters to reduce S3 reads. Index data when required query features or retrieval performance justify the additional pipeline and stored copy [1].
How We Would Implement It
- Store chunks with a reliable document ID, generate embeddings, and retrieve candidates through Elasticsearch kNN [2].
- Check each candidate’s view permission with SpiceDB CheckBulkPermissions before constructing the model prompt. Drop chunks with missing IDs or errored checks; stop generation if the authorization service is unavailable [2].
- For users with narrow access, consider a permitted-ID pre-filter; use bulk post-filtering when permitted result sets become too large for that approach [2].
- For S3 federation, use a private encrypted bucket, prefix-scoped read permissions, federated identity and deployment-restricted IAM. Apply ES|QL column selection, filters and limits to constrain reads [1].
Risks, Costs and Security
Never include denied chunk content in model input or responses. Keep SpiceDB relationships synchronized with the access-control system of record and use persistent storage; fail closed on authorization outages [2]. Federation adds an Enterprise licensing requirement and potential S3 read charges, while its preview status warrants validation before production dependence. Remove unused datasets, identity resources and S3 objects to avoid continuing costs [1].
Where Kimbodo Comes In
Kimbodo builds and operates this in production for businesses — see our RAG Development Services practice, or Estimate My RAG System.
