Relevance Feedback and Query Expansion: Rocchio and RM3

Every retrieval model so far has matched a query against documents as written. But a query and the documents that answer it routinely use different words. An analyst searches for rate guidance; the filing that answers the question discusses the company’s outlook and rate forecast and never once writes “guidance.” BM25 and query likelihood cannot match a term that is not there, so the most on-point document sinks beneath worse ones that happen to share the query’s vocabulary. This is the vocabulary mismatch problem, and relevance feedback is the classical fix: use the documents you have already retrieved to repair the query, then search again.

The catch is in the assumption. We rarely have real relevance judgments at query time, so pseudo-relevance feedback simply assumes the top-k retrieved documents are relevant and harvests expansion terms from them. When that assumption holds, feedback bridges the vocabulary gap and recall jumps. When it fails — when an off-topic document sits near the top — its words pollute the query and the results drift away from the user’s intent. The whole topic lives in the tension between those two outcomes.

feedback documents = 2query “rate guidance” · pseudo-relevance feedback (RM3)

A. recall@4 vs feedback size — rises, then drifts

0.500

1.001

1.002

0.753

0.504

0.755

number of feedback documents

B. RM1 relevance model — bridge vs off-topic terms

forecast0.243

outlook0.243

rate0.137

guidance0.122

charge0.031

C. re-ranked top-6 — relevant documents in accent; recall@4 = 1.00

1.10-K · rate guidance + outlook✓ relevant, in top-4
2.10-K · rate guidance + forecast✓ relevant, in top-4
3.10-K · rate outlook/forecast (synonym)✓ relevant, in top-4
4.News · rate forecast/outlook (synonym)✓ relevant, in top-4
5.10-K · cost guidance / budget
6.10-K · segment guidance

A little feedback (1–2 docs) adds outlook/forecast and surfaces the synonym filings — recall climbs to 1.0. Push the slider higher and off-topic feedback adds budget/costs, the query drifts, and a relevant document drops out.

The laboratory above runs RM3 over a worked corpus for the query rate guidance. Two of the four relevant documents are synonym-only — they say outlook and forecast, never “guidance” — so the no-feedback ranking misses them and recall@4 is only $0.5$ . Slide the feedback size to one or two documents and the relevance model adds outlook and forecast, the synonym filings surface, and recall reaches $1.0$ . Keep sliding: off-topic feedback documents add budget and costs, the query drifts, and a relevant document drops back out.

What we cover

Relevance feedback and the vocabulary-mismatch problem.
Rocchio: moving the query toward the relevant centroid.
The relevance model RM1.
RM3: interpolating the relevance model with the original query.
Query drift: the honest failure mode, and a finance case study.

Rocchio: the centroid update

The vector-space answer, due to Rocchio, treats query repair as geometry: push the query vector toward the documents marked relevant and away from those marked non-relevant.

Definition 1 (Rocchio update).

Given sets of relevant documents $D_r$ and non-relevant documents $D_{nr}$ (as tf-idf vectors), the updated query is

q' = \alpha\, q + \beta\,\frac{1}{|D_r|}\sum_{d \in D_r} d - \gamma\,\frac{1}{|D_{nr}|}\sum_{d \in D_{nr}} d,

with weights $\alpha, \beta, \gamma \ge 0$ . In pseudo-relevance feedback we set $\gamma = 0$ and take $D_r$ to be the top retrieved documents, then re-rank by cosine similarity to $q'$ .

The two averages are the centroids of the relevant and non-relevant sets, so Rocchio is a nearest-centroid construction: it shifts the query toward the middle of what looks relevant. This is not an arbitrary recipe — it is the solution to an optimization.

Proposition 1 (Rocchio maximizes the relevant–non-relevant margin).

Among query vectors of fixed norm, the one that maximizes the difference between the average similarity to the relevant set and the average similarity to the non-relevant set,

J(q') = \frac{1}{|D_r|}\sum_{d \in D_r} q'^\top d \;-\; \frac{1}{|D_{nr}|}\sum_{d \in D_{nr}} q'^\top d,

points in the direction of the centroid difference $\bar{d}_r - \bar{d}_{nr}$ . Rocchio’s $\beta$ and $\gamma$ terms are exactly this direction; the $\alpha\,q$ term anchors the result to the original query.

Proof.

Write the objective as a single inner product. Pulling the sums inside,

J(q') = q'^\top\!\left(\frac{1}{|D_r|}\sum_{d \in D_r} d - \frac{1}{|D_{nr}|}\sum_{d \in D_{nr}} d\right) = q'^\top\big(\bar{d}_r - \bar{d}_{nr}\big),

where $\bar{d}_r$ and $\bar{d}_{nr}$ are the two centroids. Maximizing a linear function $q'^\top v$ over the sphere $\lVert q' \rVert = c$ is the textbook Cauchy–Schwarz problem: $q'^\top v \le \lVert q' \rVert\,\lVert v \rVert$ with equality iff $q'$ is parallel to $v$ . So the optimal direction is $v = \bar{d}_r - \bar{d}_{nr}$ — the centroid difference. Rocchio realizes this direction with separate non-negative weights $\beta$ on $\bar{d}_r$ and $\gamma$ on $\bar{d}_{nr}$ , and adds $\alpha\,q$ so the repaired query does not wander arbitrarily far from what the user actually asked. $\;\blacksquare$

∎

Because $q'$ now points toward the relevant centroid, its similarity to those documents — and to their unseen near-neighbors in the same vocabulary — rises. The companion notebook confirms that the Rocchio update strictly increases the query’s mean cosine to the feedback documents and lifts recall@4 on the worked corpus.

The relevance model RM1

The language-model analogue, due to Lavrenko and Croft, is more probabilistic. Instead of moving a vector, it estimates a distribution: a relevance model $P(w \mid R)$ that we imagine the relevant documents were generated from.

Definition 2 (RM1 relevance model).

Estimate the probability of a word under the relevance model by averaging the document models, weighted by how well each document explains the query:

P(w \mid R) = \sum_{d} P(w \mid d)\, P(d \mid q), \qquad P(d \mid q) \propto P(q \mid d)\,P(d),

where $P(q \mid d)$ is the query likelihood and the document prior $P(d)$ is taken uniform over the feedback set.

The intuition is a sampling story: if we believe the query was drawn from the relevant class, then documents that generate the query well ( $P(q \mid d)$ large) are our best proxies for that class, and the words they contain are the words the relevant class favors. Summing each feedback document’s model weighted by its query likelihood produces a genuine probability distribution over the vocabulary — the notebook verifies $\sum_w P(w \mid R) = 1$ — whose highest-probability terms are the expansion terms. On the worked corpus with clean feedback, those top terms are exactly forecast and outlook, each at weight $0.243$ : the relevance model has discovered the synonyms the literal query lacked. This is a point estimate of the expanded query from a small sample of documents, and — as we will see — its bias when that sample is contaminated is precisely query drift.

RM3: interpolating with the original query

A pure relevance model can wander, so RM3 anchors it to the original query, exactly as Rocchio’s $\alpha\,q$ term does.

Definition 3 (RM3).

Interpolate the relevance model with the original (maximum-likelihood) query model and re-score by cross-entropy:

P_{\text{RM3}}(w) = (1 - \alpha)\,P_{\text{ml}}(w \mid q) + \alpha\,P_{\text{RM1}}(w \mid R), \qquad \text{score}(d) = \sum_w P_{\text{RM3}}(w)\,\log P(w \mid d).

The re-scoring is the KL / cross-entropy view from the query-likelihood topic, with the thin empirical query model swapped for the richer $P_{\text{RM3}}$ . The mixing weight $\alpha$ interpolates between two familiar endpoints.

Proposition 2 (RM3 limits).

At $\alpha = 0$ , $P_{\text{RM3}} = P_{\text{ml}}(\cdot \mid q)$ and the ranking is exactly the original query-likelihood ranking — no expansion. At $\alpha = 1$ , $P_{\text{RM3}} = P_{\text{RM1}}$ , the pure relevance model. Intermediate $\alpha$ blends the two.

The proof is immediate from the definition — substitute $\alpha = 0$ and $\alpha = 1$ — but the content is real: $\alpha$ is the dial between trusting the user’s words and trusting the feedback. The notebook confirms that at $\alpha = 0$ the RM3 ranking equals the unexpanded query-likelihood ranking (recall $0.5$ ), while a moderate $\alpha = 0.5$ with a little feedback bridges the vocabulary gap to recall $1.0$ .

Query drift: the honest failure

Everything above assumed the feedback documents are relevant. Pseudo-relevance feedback assumes it by fiat — it trusts the top-k of the initial ranking — and that is where it can fail.

⚠ Rigor flag

Pseudo-relevance feedback is effective on average but carries three honest qualifications:

The relevance assumption is often false. PRF treats the top-k retrieved documents as relevant. When one is not — a tax rate filing surfacing for an interest-rate query — its terms enter the relevance model and the query drifts toward them. The notebook makes this concrete: recall@4 climbs from $0.5$ to $1.0$ with one or two feedback documents, then falls back toward $0.5$ as the feedback set grows to include off-topic documents whose terms (budget, costs) crowd out the bridging ones.
No per-query guarantee. PRF improves mean effectiveness across queries while hurting some individual queries outright. There is no theorem that it never lowers recall for a given query — and the drift curve shows it plainly does.
The parameters are tuned. Rocchio’s $\alpha, \beta, \gamma$ , RM3’s $\alpha$ , the number of feedback documents, and the number of expansion terms are all set empirically; the smoothing used in the re-ranking must itself be scaled to document length. None of them are derived from first principles.

The drift is not a bug to be patched away — it is the structural cost of fabricating relevance judgments, and naming it is the point.

Finance case study

§ Finance case study text

Query expansion earns its keep in financial retrieval precisely because the domain is full of vocabulary mismatch. An analyst’s rate guidance must match a filing’s outlook for the rate environment; a query for credit deterioration must reach a disclosure of rising provisions for loan losses. Pseudo-relevance feedback over 10-K / 10-Q filings and earnings-call transcripts bridges these gaps by harvesting the domain’s own synonyms from the top retrieved documents.

But the finance domain is also where drift bites hardest, because of polysemy: rate means an interest rate, a tax rate, a dividend rate, or a growth rate depending on context. A pseudo-relevant document about the effective tax rate injects tax and charge into the relevance model and pulls an interest-rate query toward accounting disclosures. The practical discipline this topic teaches — use few feedback documents, interpolate with the original query (RM3’s $\alpha$ ), and cap the expansion — is exactly the guardrail a production retriever needs.

In the production system, query expansion runs before fusion: the expanded lexical query feeds reciprocal rank fusion alongside dense retrieval, so the dense leg’s paraphrase-matching and the expanded lexical leg’s synonym-bridging reinforce each other, while RM3’s interpolation with the original query keeps the drift bounded.

Implementation

The companion notebook implements both relevance-feedback families from scratch over the worked finance corpus — Rocchio on tf-idf vectors and RM1/RM3 on language models, with the query likelihood $P(q \mid d)$ supplying the feedback weights — and turns each claim into an assertion. It verifies that the RM1 relevance model is a proper distribution, that the Rocchio update equals $\alpha\,q + \beta$ times the feedback centroid and raises the query’s mean similarity to it, and that RM3 recovers the unexpanded query-likelihood ranking at $\alpha = 0$ .

The two headline numbers are the gain and the drift, measured on the same curve. For the query rate guidance, whose four relevant documents include two synonym-only filings, recall@4 starts at $0.50$ with no feedback, rises to $1.00$ once one or two feedback documents inject forecast and outlook (each weighted $0.243$ in the relevance model), and falls back to $0.50$ as the feedback set grows to four and off-topic terms enter — the query-drift failure, reproduced exactly. The full recall-versus-feedback curve and the expansion terms at each feedback size are printed by the harness so the page, the notebook, and the laboratory read one set of numbers.