Findings
-
[1] 2026-09-28 The best LLM writes correct Elasticsearch ES|QL 59% of the time. Here's what breaks the other 41%.
We gave four models 500 natural-language questions from BIRD's Mini-Dev set, asked each for one Elasticsearch Query Language (ES|QL) query, and graded all 6,000 answers on whether the right rows came back. The best model got 59% correct on a… props[field] = {"type": "keyword", "ignore_above": KEYWORD_IGNORE_ABOVE}All in, that’s roughly 3.9 million documents across 75 indices, running on Elasticsearch 9.5. If you haven’t built a join like this before, this walkthrough of native joins in Elasticsearch covers the index-mode requirements in… Elasticsearch 9.2 added a second form that lifts this restriction, but only when the two keys have different names. That condition is where our run went sideways, and we come back to it at the end of this section.A model… Give it advice it didn’t need, and you can take 10 points off it.Roughly half of the failures throw a precise, actionable Elasticsearch error (Unknown column [gender_id] in right side of join tells you exactly what to fix), so a…
Where Kimbodo Comes In
Kimbodo builds and operates this in production for businesses — see our RAG Development Services practice, or Estimate My RAG System.
