6. Reranking with cross-encoders

Embedding search is a bi-encoder: query and document are encoded separately, so scoring is fast but coarse. A cross-encoder reranker feeds the query and each candidate together through a model that scores true relevance — far more accurate, far more expensive.

The standard shape: retrieve top 50-100 candidates (hybrid), rerank to the top 5-8, then generate.

python
pairs = [(query, c.text) for c in candidates]
scores = reranker.predict(pairs)                 # cross-encoder
top = [c for _, c in sorted(zip(scores, candidates), reverse=True)[:6]]

Why this pays off: - Precision at small k is what the generator actually consumes. - Fewer, better chunks reduce prompt cost *and* the lost-in-the-middle effect. - Rerankers are also where you can inject business signals (freshness, authority, product version).

Gotcha: reranking latency scales with candidate count. Cap candidates, batch the scoring, and cache reranks for repeated queries.

Read in another language

How to study this topic

Scoring

Check your understanding

Badges