What Is Reranking? How Rerankers Make Search and RAG Smarter

A customer asks the assistant on a website: "How do I return my order?" The system searches the knowledge base and puts… "Delivery terms" in first place. The return policy ends up fourth or fifth. The assistant leans on the top fragments, starts talking about delivery, and the customer gets a useless answer.

The search didn't make a mistake: all of those documents really are about orders and are close in topic. The order is just imprecise. The step that fixes this is called reranking.

What is reranking?

Reranking is the second stage of search. The first stage quickly picks a few dozen candidates out of thousands or millions of documents. The reranker then takes this short list, carefully compares each document with the question and puts them back in order by how relevant they actually are.

Think of a library. The first stage is the librarian who hears your question and brings twenty books from the right shelf. The reranker is the expert who opens each of those twenty books, flips through them and says: "These two answer exactly what you asked."

Why one stage isn't enough

Modern semantic search is built on embeddings: every document is turned into a list of numbers in advance, the question is turned into numbers the same way, and the system finds the documents closest in meaning. We explain how this works in detail in our article on embeddings.

The strength of this approach is speed. Documents are converted once and stored, and searching through millions of records takes milliseconds. The weakness is that the question and the document are converted separately. When the model reads a document, it doesn't see the question. As a result, documents that are "close in topic" and documents that "actually answer the question" get almost the same scores.

How a reranker works

A reranker is usually a cross-encoder model. It reads the question and the document together, looks at how they relate word by word, and returns a single number: how well this document answers this question. For example, 0.94 means highly relevant and 0.05 means barely related.

This is more accurate but more expensive: the model runs separately for every question and document pair. Checking a million documents this way would take far too long. So the two stages are combined: fast search picks the candidates, and the slower, precise reranker sorts only those.

A real example: before and after

Let's see what happens to the question from the start of the article. Vector search found five documents and gave them almost identical scores, between 0.73 and 0.81. The document that matters most, the return policy, ended up at the bottom.

After reranking, the picture changes completely. "Return policy" rises to first place with 0.96, and "Refunds to your card" to second. "Gift cards" gets 0.03 and practically drops off the list. Notice that the gap between scores becomes sharp as well. That also helps the system recognise when no good answer was found.

Where reranking is used

• RAG and AI assistants. The language model writes its answer from the retrieved fragments. The more precise the fragments, the more precise the answer. This sharply reduces the risk of the assistant making things up, which we cover in our article on AI mistakes.
• Site search. In an online store, a query for "warm kids' winter boots" pushes down products that mention "kids" and "shoes" but are actually summer sandals.
• Corporate knowledge bases. When an employee searches contracts, manuals and policies, the most precise clause comes out on top.
• Recommendation systems. Hundreds of candidates are selected first, then reordered for each specific user.

Large search engines have used multi-stage ranking for years: a broad first pass, then precise sorting with heavier models. Reranking is the version of that idea that is within reach for business projects.

How reranking affects costs

At first glance, an extra model means extra cost. In practice it is often the opposite. Without a reranker, 20–30 fragments are sent to the language model "just in case". With a reranker, the 3–5 most precise ones are enough. That saves thousands of tokens on every request. We show how tokens affect the budget, with numbers, in our article on AI integration costs.

On top of that, language models make worse use of information buried in the middle of a long context. A few precise fragments are both cheaper and better.

Types of rerankers

• Cloud APIs: Cohere Rerank, Voyage AI, Jina Reranker. Easy to connect, with support for many languages.
• Open models: the BAAI bge-reranker family and others. They run on your own server and your data never leaves it, which matters for companies with strict data security requirements.
• A language model as a reranker. You can give a large model the candidates and ask it to pick the best ones. Flexible, but slower and more expensive.

For texts in languages other than English, such as Russian or Azerbaijani, choose a multilingual model and test it on your own data. Rerankers trained only on English can perform noticeably worse in other languages.

Do you need a reranker?

If your search or assistant returns results that are "roughly right, but not quite it", that is the classic sign you need reranking. Especially when:

• your knowledge base has many documents that are close in topic;
• questions are long or specific, for example about a particular contract clause;
• the assistant finds the right document but answers from a different one;
• too much context is sent to the language model and costs keep growing.

Adding a reranker to existing search usually doesn't mean rebuilding the whole system: it plugs in as a separate step between search and the language model, and the result can be measured with test questions.

The bottom line

Reranking is search taking a second look: fast search finds the candidates, and the reranker sorts them by what the question actually means. The user sees the right answer first, the AI assistant answers more accurately, and fewer tokens go to the language model.

In our AI integration projects we build semantic search, reranking and RAG together and measure quality on real questions. If you'd like to check how accurately your search or assistant works, get in touch.