Book a 30-min call
← cd ../blogs
$ cat posts/the-embedding-ceiling-and-the-reranker-budget.mdx

The embedding ceiling and the reranker budget

August 4, 2026 · ImmovableTech Team

  • Retrieval
  • Inference
  • Production AI

The plateau nobody budgets for

Every retrieval project we have worked on hits the same wall in roughly the same week. Someone swaps the embedding model for a newer one, re-embeds the corpus over a weekend, runs the eval set, and recall moves by less than the noise in the measurement. They try a larger dimension. Same result. The conclusion drawn in the retro is usually that retrieval is as good as it gets and the remaining errors are the generation model’s problem.

That conclusion is wrong, and it is wrong in a specific way that changes what you should build next. The plateau is not the limit of embedding models. It is the limit of scoring a query and a document independently and comparing the two numbers. No model that works that way, at any dimension, gets past it. The way through is a second stage that reads the query and the document together, and the interesting engineering question stops being which embedding model and starts being how many candidates you can afford to feed that second stage.

What a single vector actually cannot represent

The careful version of this claim matters, because the loose version is everywhere and it is overstated.

Weller and colleagues at Google DeepMind connected top-k retrieval to the sign rank of the query-document relevance matrix, which places a lower bound on the embedding dimension needed to represent a given set of relevance patterns. They then built LIMIT, a deliberately trivial dataset — documents are people, queries are of the form “who likes X” — sized so that all top-2 combinations over 46 documents give just over 1,000 queries. State-of-the-art embedding models struggle to reach 20% recall@100 on the 50,000-document version, and on the 46-document version the paper reports the best models below 60% recall@2. BM25 comes close to perfect, because a lexical model is a very high-dimensional sparse vector and dimension is exactly the resource in question.

Here is where we have to be honest, because a follow-up matters. A team at Microsoft Research India showed that by a result of Alon and colleagues, embeddings of dimension 2k + 1 suffice to represent a relevance matrix where each query has at most k relevant documents. LIMIT has k = 2, so five dimensions are enough in principle. Their bound applies to a fixed, known set of queries and documents rather than to a model that must generalise to new ones, which is the practically important gap — but it does mean the LIMIT failures are not purely a dimension-counting proof. They attribute most of the observed damage to domain shift and to cosine similarity being misaligned with the task’s notion of relevance. Fine-tuning lifted single-vector recall@10 from about 1% to about 40% on their setup.

So the accurate statement is narrower than the headline: single-vector similarity has a real representational capacity limit, that limit is not the whole explanation for why these models fail, and the failures that remain after you control for dimension are failures of a scoring function that never sees the query and the document at the same time. Both papers agree on the part that matters operationally — the same follow-up found multi-vector models reaching 98% recall@10 after fine-tuning where single-vector models reached 40%, and losing about 1% on MS MARCO afterwards against a drop of more than 40% for the single-vector models.

The thing a second stage recovers

A bi-encoder embeds the query and the document separately, which is the entire reason it is fast: document vectors are computed once at index time and the query only has to be compared against them. That independence is also what throws information away. The document’s representation cannot depend on the query, so nothing about the interaction between the two survives into the score.

A cross-encoder has no such constraint. It reads the concatenated pair and can attend across it. The LIMIT authors ran a long-context model as a reranker over all 46 documents and all 1,000 queries and it solved every one in a single generation, against embedding models that could not solve the task at recall@20. That is not a better model beating a worse one. It is a different architecture not having the constraint.

Multi-vector, late-interaction retrieval sits between the two. ColBERT-style models keep one vector per token and score with a max-similarity operator, which recovers some interaction while remaining precomputable — and the storage bill is per token rather than per document, which is the reason it stays a deliberate decision rather than a default. It performed markedly better than single-vector models on LIMIT while still not solving it.

The budget is a latency dial

If reranking is where the quality comes from, the design parameter is how many candidates the reranker sees, because latency scales with that count. Recent numbers make the shape concrete. Jina’s release of jina-reranker-v3.5 on 3 August 2026 reports a 0.6B listwise reranker reaching 63.20 nDCG@10 on BEIR, reranking the top-100 candidates from a 0.5B first-stage embedding model that scores 56.26 on its own under the same pipeline. Roughly seven points of nDCG@10 come from the second stage, and the reranked 0.6B model sits above the 4B Qwen3-Reranker’s 62.28 on that benchmark under their unified protocol.

The latency table is the part worth pinning to the wall. On a single A100 at batch size 1, reranking 100 candidates takes 305 ms when the documents average 145 tokens, and 10.3 seconds when they average 1,904 tokens. Same candidate count, same model, more than thirty times the latency. Your budget is not a number of documents, it is a number of tokens, and chunk size sets it as much as top-k does.

BEIR, worth saying plainly, is a zero-shot English retrieval suite scored by nDCG@10. It measures ranking quality at the top of the list on public data. It does not measure your recall, your latency or your corpus. Its test sets are also public, so models end up indirectly trained on them — the generalisation gap that led the MTEB maintainers to launch RTEB in October 2025 with private splits, precisely to measure performance on data a model has not seen. Use these numbers to rule models out, not to pick a winner by a point.

Which means you have to measure the curve yourself:

import numpy as np


def recall_at_k_curve(run, qrels, candidate_ks):
    """First-stage recall at each candidate depth — the reranker's ceiling."""
    curve = {}
    for k in candidate_ks:
        per_query = [
            len(set(run[q][:k]) & qrels[q]) / len(qrels[q]) for q in qrels
        ]
        curve[k] = float(np.mean(per_query))
    return curve


ks = [10, 25, 50, 100, 200, 400, 800]
ceiling = recall_at_k_curve(first_stage_run, qrels, ks)
marginal = {b: ceiling[b] - ceiling[a] for a, b in zip(ks, ks[1:])}

The curve flattens at a point specific to your corpus and query mix, and the knee is where the marginal recall stops paying for the added rerank latency. Ours has landed between 50 and 150 on every corpus we have measured, but we have never been able to guess it correctly in advance.

More candidates is not monotonically better

We assumed for a long time that feeding the reranker more candidates could only help, bounded by cost. It is not true, and the finding is well documented. Databricks researchers scaled candidate counts past 5,000 across academic and enterprise datasets and found that pointwise cross-encoders improve initially and then degrade, often below the standalone retriever. Rerankers helped at some candidate count in 85.0% of academic and 88.9% of enterprise experiments, but in 53.3% and 44.4% respectively, scaling to the largest count produced recall@10 worse than retrieval alone. Only 23.3% and 22.2% of experiments never got worse at any depth.

The failure mode they name is “phantom hits” — the reranker scoring a document with no lexical or semantic connection to the query very highly, exactly the documents the retriever had correctly buried. Their one encouraging result is that listwise reranking, where the model sees candidates together rather than scoring each in isolation, kept improving as the count grew. That is the same architectural direction the current generation of listwise rerankers has taken, and it is the reason we now treat “pointwise or listwise” as a more load-bearing choice than “which vendor”.

The reranker is the cheap part

Teams resist the second stage on cost grounds and then discover the arithmetic runs the other way. Look at how reranking is actually billed: Cohere defines a search unit as one query with up to 100 documents. Reranking 10 candidates and reranking 100 cost the same. What is metered is the query, not the depth — with the caveat that documents over 500 tokens are split into chunks and each chunk counts towards the 100, which is the same token-not-document lesson the latency table teaches.

Set that against the generation call the reranker feeds, which is billed per token and produces output tokens at a multiple of input price. The reranker’s job is to make that call shorter and better-grounded by putting the right passages at the top. In our experience the second stage is comfortably the smaller line item, and it reduces the larger one.

A reranker cannot rank what the retriever never returned

This is the limit of everything above. First-stage recall@k is a hard ceiling on the whole pipeline, and if that is where you are broken, no reranker rescues you.

The jina report contains an unusually clean illustration. On Struct-IR, a benchmark over millions of structured records, first-stage in-schema recall@5 is around 0.04 — so end-to-end retrieve-then-rerank is almost entirely recall-bound, and the benchmark cannot separate rerankers at all unless gold documents are injected into the candidate pool. Relevance over records turns on equality, numeric and date bounds and logic across fields, not on lexical or semantic overlap. That is a filter, not a similarity search, and no reranking budget fixes it.

Most retrieval problems we are called into are somewhere on that spectrum. Before we touch either model now, we check whether chunk boundaries are splitting the answer across two passages, and whether the query as issued resembles the language of the corpus at all. Query rewriting and chunking have moved more recall for us than any model swap, and they are cheaper.

What we’d do differently

We spent the better part of a sprint moving a client index from 1,024 to 4,096 dimensions — re-embedding the corpus, resizing the index and re-tuning the pre-filter — for a gain our own A/B framework could not distinguish from noise. The same effort spent on a reranking stage would have bought a capability the embedding cannot have at any dimension.

Worse, when we did add a reranker we set top-k to 200 on the reasoning that more candidates could only help, and quietly shipped a configuration that was past the knee. Aggregate nDCG@10 looked fine because the head of the ranking was unaffected; it was recall on the specific, long-tail queries people complained about that had degraded. It took us weeks to find, and we only found it by plotting recall against candidate depth rather than reading a single number. On the search relevance work documented in our portfolio we now run a deliberately modest candidate budget, chosen from that curve, and the eval set is stratified so tail queries cannot hide inside an average.

The order we would use now: fix chunking, measure the recall-versus-k curve of the retriever you already have, add a reranker sized to the knee of that curve, and only then ask whether a better embedding model is worth a weekend. We have never once reached that last step and found the answer was yes. Retrieval quality is the input to everything downstream, including the hallucination detection pipeline we built on top of one, and a bigger generation model does not repair it later.

References

We design and tune production retrieval pipelines as part of our AI & Machine Learning Engineering practice. Talk to us if your recall has stopped responding to better embedding models.