Vector similarity grasps meaning but misses the literal (codes, acronyms). Hybrid dense + BM25 search fused with RRF, then cross-encoder reranking: the combo that makes a RAG genuinely precise.
Vector search has a flaw we prefer not to mention: it is excellent at meaning, mediocre at the literal. Ask it to find a document about "early contract termination" and it shines. Ask it for the product code XR-4420-B or the exact acronym DSP2 and it may miss entirely. That is why serious RAG systems almost never use vector similarity alone. They combine hybrid search and reranking.
Here is why, and how to assemble them — with the numbers, trade-offs and settings that make the difference in production.
An embedding captures the meaning of a passage in a continuous space. Two phrasings of the same idea end up close. That is dense retrieval's strength — and its limit.
Some queries are not about meaning but exact matches:
v2.14.0);SIREN, IBAN);On these, an embedding model may judge as "close" passages that share the topic but not the exact term. The mechanism is easy to grasp: a subword tokenizer splits XR-4420-B into fragments (XR, -, 44, 20…) with no stable meaning, and the resulting vector lands next to any other alphanumeric reference. To the embedding, XR-4420-B and XR-4420-C are nearly indistinguishable; to your user, they are two different parts. Result: the right document — the one literally containing the reference — may not surface, or surfaces at rank 30, outside the context passed to the LLM.
Dense search understands what you mean; lexical search finds what you wrote. A good RAG needs both.
Take a technical support corpus. The query: "error E-1042 after update v2.14.0". The only useful document is a release note that mentions E-1042 and v2.14.0 verbatim.
E-1042 and v2.14.0 are rare tokens with very high IDF weight. The document containing both comes out on top immediately.This is exactly the kind of query — common in support, legal, finance, R&D — where you lose a customer if the RAG answers off-target.
BM25 (Best Matching 25) is the reference algorithm for lexical retrieval, inherited from the TF-IDF family. It ranks a document against a query using three ingredients:
k1, typically ~1.2–2.0).E-1042) is worth much more than a common word (the, update). This is what makes BM25 surgical on identifiers and jargon.b, typically ~0.75).BM25's strength lies in IDF: a token appearing in just one document of the corpus becomes a near-deterministic signal. No semantic generalization, no "approximately" — precisely what dense cannot do. In return, BM25 is blind to synonyms ("termination" ≠ "cancellation") and to rephrasing. Hence the complementarity, not the competition.
Dense generalizes, BM25 discriminates. One finds the document about the same topic, the other finds the document containing the right word. Fusing them means you stop choosing.
Hybrid search combines these two complementary engines:
Run both in parallel, then fuse the rankings. The most robust method is RRF (Reciprocal Rank Fusion), which combines ranks (not scores, hard to compare across engines) into a unified ranking.
Query
├──► dense search (pgvector / HNSW) ──► top-k_dense
└──► lexical search (BM25) ──► top-k_lexical
│
RRF fusion
│
unified candidates (large top-N)
RRF ignores raw scores — a cosine distance (between 0 and 1) and a BM25 score (unbounded, corpus-dependent) are not comparable. It keeps only the rank of each document in each list and applies:
score_RRF(d) = Σ 1 / (k + rank_i(d))
lists i
where k is a damping constant (usual value: 60) preventing the very first rank from crushing all others. Take three documents A, B, C and two engines:
| Doc | Dense rank | BM25 rank | Calc (k=60) | RRF score |
|---|---|---|---|---|
| A | 1 | 4 | 1/61 + 1/64 | 0.0320 |
| B | 3 | 1 | 1/63 + 1/61 | 0.0323 |
| C | 2 | 30 | 1/62 + 1/90 | 0.0272 |
Reading the result: B takes the lead even though it is first in only one engine, because it ranks well in both. A, strong in dense and decent in BM25, follows closely. C, brilliant in dense (rank 2) but nearly absent in BM25 (rank 30), comes last: one engine is not enough to carry it. That is the whole point of RRF — cross-engine consensus beats an isolated spike, with no need to calibrate fragile weights between incompatible score scales.
RRF rewards agreement between engines. A document both dense and BM25 deem relevant rises strongly; one carried by a single engine stays a candidate without dominating. That is what makes fusion robust without fine tuning.
Fusion yields a candidate list — say the top 50. But 50 is too much for the LLM context, and the order is not yet optimal. Enter reranking.
The distinction is central. A bi-encoder — that is what your embeddings are — encodes the query and each document separately, into two vectors, then compares by dot product or cosine. Upside: you precompute all document vectors at indexing time, so a query costs only one encoding plus an index lookup. Downside: query and document never "see" each other; the whole interaction is collapsed into a distance between two points.
A cross-encoder — that is the reranker — concatenates query and candidate into a single input ([query] [SEP] [document]) and lets the transformer's attention cross every word of the query with every word of the document. It outputs a relevance score directly. This is far finer: it "understands" that the query's v2.14.0 matches the document's v2.14.0 exactly, and that context links them. The cost: nothing can be precomputed, since the score depends on the pair. You need one model pass per (query, candidate) pair at query time.
Hence the two-stage architecture: the bi-encoder (fast, precomputed) filters millions of documents down to a few dozen; the cross-encoder (slow, precise) reorders those few dozen.
The winning pattern:
Mature 2026 rerankers: Cohere Rerank 3, bge-reranker-v2-m3 (open, multilingual), Voyage rerank, the Jina reranker.
candidates (top-50) ──► cross-encoder reranker ──► top-8 ──► LLM
(recall) (precision)
Retrieve broad then rerank fine: the most cost-effective recall/precision trade-off in RAG. Vector similarity is the coarse filter; the reranker does the sorting that counts.
Reranking is not free, but it is largely affordable if you respect the funnel. Prudent orders of magnitude to expect in 2026:
bge-reranker-v2-m3 on GPU: expect a few milliseconds per pair, i.e. a few tens of ms for 50 batched candidates. No marginal per-request cost, but a GPU instance to provision and maintain.Compared to LLM generation (often hundreds of ms to several seconds), the reranking overhead is almost always a good investment: it improves context precision, hence answer quality, hence cuts iterations.
Because the cross-encoder is expensive: one model pass per (query, candidate) pair. Reranking 10,000 documents per query would be unaffordable in latency and cost — and pointless: beyond a few dozen candidates, you mostly rerank noise. Hence the funnel: hybrid search narrows to a few dozen candidates the reranker can process quickly, concentrating the precise compute where it matters.
ef_search on pgvector/HNSW — this parameter controls how wide the graph search explores at query time. Too low, the HNSW index misses good candidates (degraded recall); too high, latency climbs. Aim for an ef_search that keeps your dense top-N stable, typically a few hundred. It is the index-side counterpart to the application-side top-N.k around 60 — a proven value. Touching it rarely helps: if fusion disappoints, the problem is almost always upstream (embedding quality, chunking, one engine's recall), not in k.Vector similarity alone is a good start and a bad finish. Add RRF-fused lexical search to recover what meaning alone misses — codes, references, jargon; add a cross-encoder reranker to turn a decent candidate list into precise context, finally letting query and documents "see" each other. That combination — broad then fine, meaning then literal — separates a demo RAG from one that answers correctly in production. The good news: every building block (BM25, RRF, reranking) is now mature, cheap, and available both as an API and as self-hostable open source.
RAG impresses in a demo and disappoints in production. Three patterns change everything: hybrid vector + BM25 search with RRF fusion, mandatory cross-encoder reranking, and decomposing complex questions into sub-queries.
Vector RAG retrieves isolated passages; GraphRAG links entities via a knowledge graph. Strengths, weaknesses, ingestion cost and a hybrid approach: when the graph genuinely changes the game.
Single-pass RAG fails on multi-hop and comparison questions. Agentic RAG lets an agent decompose, retrieve iteratively, and stop at the right time — with its guardrails and cost. When to use it, and when not.
Start free. First Knowledge Pulse audit in 60 seconds.
Start free