What Happened
A comparison across 25 selected retrieval benchmarks found that reranking more results did not reliably improve search. With Elasticsearch BM25 as the initial retriever, reranking 50 candidates reduced error by 18.75%; reranking 100 reduced it by 20.97% but used twice the reranking tokens. The largest datasets were excluded from this comparison [1].
Initial retrieval quality mattered more. The tested embedding retriever reduced error by 25.7% on average versus BM25. For that retriever, reranking 20 candidates produced a further 8.5% error reduction, while reranking 200 performed worse on average than not reranking [1].
Why It Matters to Businesses
RAG teams should not treat a larger reranking pool as a default quality upgrade. A stronger first-stage retriever may deliver better results while sending fewer documents to an expensive reranker. Existing BM25 systems can also test reranking without reindexing, although they may need a larger candidate pool to achieve competitive quality [1].
Kimbodo Engineering Perspective
The practical choice is not “vector search versus keyword search.” It is which first-stage retrieval strategy finds relevant, authorized material for a particular workload, and whether reranking improves the answers enough to justify its latency and cost. Listwise rerankers rank a pool together; pointwise and pairwise approaches score candidates differently. Their results and costs should be measured rather than inferred from the model category [1].
How We Would Implement It
- Build an evaluation set from real queries, with relevance judgments and checks for whether generated answers are supported by retrieved passages.
- Benchmark BM25 in Elasticsearch against embedding retrieval in the selected vector platform, such as Weaviate, Pinecone, Qdrant, Milvus or Vespa. Use LlamaIndex, LangChain or Haystack for orchestration only where they simplify integration and testing.
- Test reranking pools of 20, 50 and 100 against a no-reranking baseline. Include 200 only if smaller pools leave a measurable recall gap. Track retrieval error, answer quality, reranking tokens and end-to-end latency [1].
- Version chunking, embeddings, indexes and evaluation data so changes can be compared and rolled back.
Risks, Costs and Security
Apply tenant and document permissions before results reach a reranker or model; otherwise, retrieved text can leak across access boundaries. Treat retrieved content as untrusted input, and review whether an external embedding or reranking service may receive sensitive passages. More candidates increase token use and latency without a guaranteed quality gain, so set pool sizes from workload-specific tests, not benchmark averages [1].
Where Kimbodo Comes In
Kimbodo builds and operates this in production for businesses — see our RAG Development Services practice, or Estimate My RAG System.
