Improve reranking and retrieval quality

Locate the retrieval failure#

Reranking reorders an existing candidate set for a particular query. Cohere’s reranking tutorial places it after keyword or semantic retrieval. It cannot promote a document that the first stage never found.

Before changing the ranker, label a review set with acceptable evidence identifiers. Mark whether each candidate directly answers the question, supplies useful background, or is irrelevant. First inspect candidate recall: does the pool contain the evidence at all? Only then ask whether useful evidence appears early enough. Keep ambiguous labels for review instead of forcing artificial certainty.

Construct an honest candidate pool#

Use identical candidate sets when comparing the original ordering against a reranked ordering. Otherwise, an apparent improvement may come from a changed search query or larger pool. Record each passage’s identifier, revision and retrieval method.

Deduplicate near-identical chunks before they consume the context budget. Include difficult cases such as an obsolete policy with matching keywords and a current policy expressed differently. Apply access restrictions before sending passages to any external service. Where several sources are needed, inspect whether a single document crowds out complementary evidence.

Shape inputs and interpret scores#

Serialize the information needed to judge relevance: a concise title, section label and passage text are often easier to inspect than an unstructured record dump. Keep essential fields near the beginning. Cohere’s best-practice guide explains why chunking and truncation behavior matter for long documents and structured inputs.

Read the limits for the exact model being tested. Check whether relevant text survives preprocessing rather than assuming the whole document was considered. Interpret scores according to that model’s documented semantics. Do not treat a ranking score as a probability, or reuse a threshold across domains, without validating that interpretation on your own labeled examples.

Preserve identifiers through ranking#

The following is provider-neutral Python pseudocode, not an Align API. The named functions stand for components supplied by the application developer.

candidates = search(query, allowed_scope)
candidates = deduplicate(candidates)
scores = score_query_passage_pairs(query, candidates)
ranked = sorted(
    zip(candidates, scores),
    key=lambda pair: pair[1],
    reverse=True,
)
context = select_within_budget(ranked, token_budget)

Keep identifiers and source metadata attached to candidates throughout this process. Return selected passages rather than detached scores. Log why context selection dropped a passage, especially when a token limit excludes otherwise relevant evidence.

Measure gains and limits#

Compare the same questions before and after reranking. Track whether relevant evidence moves upward, whether multi-source questions retain enough coverage, and whether answer quality improves. Mean reciprocal rank can summarize the position of the first useful result; graded relevance metrics can capture improvements beyond that first result.

Review tail latency and processing cost alongside quality. Test rare terminology, multilingual queries and tables separately. Reranking adds work and may reward topical overlap over the exact evidence a task needs. It does not establish factual truth, document freshness or user authorization. Keep those checks in the application, and retain the simpler baseline if the measured gain does not justify the added cost.

Report changes by query type rather than only as an overall average. A gain on common questions can conceal regressions on the cases that matter most.