Learning to Rank: Pointwise, Pairwise, and RankNet

Overview & motivation

The evaluation layer taught us to measure a ranking: precision and recall at a cutoff, Average Precision as the area under the precision–recall curve, and NDCG with its graded gains and rank discounts. Every one of those numbers grades a ranking that some retriever already produced. This topic asks the inverse question. Given labeled examples of what a good ranking looks like, can we learn the scoring function that produces one — and in particular, can we learn to combine several retrievers better than a fixed rule?

Learning to rank is the reduction of that question to supervised learning, and there are three reductions, distinguished by what the model is asked to predict. The pointwise approach predicts each document’s grade and regresses for it. The pairwise approach predicts, for each pair of documents, which one should rank higher. The listwise approach scores a whole permutation at once. We develop the first two in full — the pairwise method, RankNet, is the center of the topic — and preview the third.

query 1 — ranked by pointwise score

Ag3

Bg0

NDCG = 1.000

query 2 — ranked by pointwise score

Dg0

Eg0

Fg0

Cg1

NDCG = 0.431 — the relevant doc C is mis-ranked

calibrated MSE (lower = better fit)

0.800

mean NDCG (higher = better order)

0.715

Ranking is not regression. Pointwise least squares is the global MSE minimizer (0.80 vs 1.21) and even spends a query-level feature on calibration — yet it mis-orders query 2 and scores NDCG 0.715. Pairwise RankNet ignores that feature, orders both queries better, and scores NDCG 0.816 at worse MSE. Order beats calibration.

120 synthetic finance documents, 40 queries (24 train / 16 test); features are the three legs' per-document scores, grades are the top-10 exact-MaxSim oracle tertiles. Numbers mirror learning_to_rank_pairwise.py; the lab recomputes the witness orders, the linear scores and σ, the bars, and the gaps in closed form.

The lab above is the topic in miniature. Panel A contrasts the three reductions on a worked query; Panel B is the theorem that makes the whole field necessary; Panel C dissects RankNet’s gradient; and Panel D is the finance payoff — a learned combiner of three retrieval legs against reciprocal-rank fusion. We score over the capstone’s complementary-view finance corpus: $N = 120$ documents, $Q = 40$ queries, and three retrieval legs — a lexical (BM25) leg, a dense dual-encoder leg, and a late-interaction leg — each reading a different window of every document’s tokens, so each recalls neighbors the others miss. Each document’s feature vector is the triple of leg scores $\mathbf{x}_d = [\,s_{\text{lex}}, s_{\text{dense}}, s_{\text{late}}\,]$ , standardized within its query; the relevance grades come from the same exact-MaxSim oracle the evaluation layer used.

Three reductions: pointwise, pairwise, listwise

Write the linear scorer as $s_i = \mathbf{w}^\top \mathbf{x}_i$ , with feature vector $\mathbf{x}_i$ and weight vector $\mathbf{w}$ to be learned. Each document carries a graded relevance label $g_i \in \{0,1,2,3\}$ . The three reductions differ in the loss they put on $\mathbf{w}$ .

Definition 1 (The three learning-to-rank reductions).

Given training documents with features $\mathbf{x}_i$ and grades $g_i$ , grouped by query:

Pointwise. Regress the score onto the grade: $\displaystyle \min_{\mathbf{w},b} \sum_i (\mathbf{w}^\top\mathbf{x}_i + b - g_i)^2$ . A standard least-squares problem; the prediction is a calibrated estimate of the grade itself.
Pairwise. For every within-query ordered pair $(i, j)$ with $g_i > g_j$ , predict that $i$ outranks $j$ , and penalize getting the comparison wrong. RankNet is the canonical instance.
Listwise. Define a loss on the whole ranked list at once — a probability distribution over permutations, or a direct smoothing of a rank metric. Previewed at the end.

A geometric reading separates pointwise from the rest. Pointwise fits a function through the $(\text{feature}, \text{grade})$ points and is judged by how close its predictions land to the grades — it cares about absolute values. The pairwise and listwise losses care only about order: which document is scored above which. That distinction has a precise consequence.

Remark (Order is invariant to a score shift).

A ranking is the argsort of the scores, so it is unchanged by adding a constant $c$ to every score: $s_i - s_j = (s_i + c) - (s_j + c)$ . The pairwise model depends on the scores only through their differences $s_i - s_j$ , so it too is invariant to a constant shift — it fixes order, not absolute calibration. Pointwise regression has no such freedom: it must pin each score to a grade, an extra burden that, as we will see, can cost it the ranking.

This is the same order-not-absolute invariance that appeared as the modality gap in cross-modal retrieval: a degree of freedom that the metric we care about is blind to, but that a calibration loss spends effort fitting anyway.

RankNet: preferences as Bernoulli trials

RankNet models each preference probabilistically. For a pair with $g_i > g_j$ , the model’s belief that $i$ should outrank $j$ is the logistic of the score difference, and the loss is the cross-entropy against the certain label.

Definition 2 (The RankNet pairwise model and loss).

For an ordered preference pair $(i, j)$ — meaning $i$ should outrank $j$ — RankNet sets

P(i \succ j) \;=\; \sigma(s_i - s_j) \;=\; \frac{1}{1 + e^{-(s_i - s_j)}}.

Because the label is certain ( $i$ truly should outrank $j$ ), the per-pair cross-entropy is $\ell_{ij} = -\log \sigma(s_i - s_j)$ , and the total loss over the labeled preference set $\mathcal{P}$ is

L(\mathbf{w}) \;=\; \sum_{(i,j)\in\mathcal{P}} -\log \sigma(s_i - s_j) \;=\; \sum_{(i,j)\in\mathcal{P}} \log\!\big(1 + e^{-\mathbf{w}^\top(\mathbf{x}_i - \mathbf{x}_j)}\big).

This is exactly the negative log-likelihood of the observed preferences under a Bernoulli model — so fitting RankNet is the maximum-likelihood estimation of a logistic regression, applied to preference pairs rather than to single labels. Term by term the cross-entropy is a KL divergence from the target preference distribution to the model’s, so minimizing the loss pulls the model’s beliefs toward the labels. And because the loss is a function of $\mathbf{w}$ through the linear map $\mathbf{w} \mapsto \mathbf{w}^\top(\mathbf{x}_i - \mathbf{x}_j)$ , it has the property that makes it trainable without any of the usual deep-learning machinery.

Theorem 1 (The pairwise logistic loss is convex in the weights).

For a linear scorer $s = \mathbf{w}^\top\mathbf{x}$ , the RankNet loss $L(\mathbf{w})$ is convex in $\mathbf{w}$ , so it has a single global minimum reachable by Newton’s method.

Proof. Write $z_{ij} = \mathbf{w}^\top(\mathbf{x}_i - \mathbf{x}_j)$ and the softplus $\zeta(u) = \log(1 + e^{u})$ . Each term of the loss is $\zeta(-z_{ij})$ . The softplus is convex (its second derivative $\sigma(u)\big(1-\sigma(u)\big)$ is nonnegative), and $-z_{ij}$ is a linear function of $\mathbf{w}$ ; the composition of a convex function with an affine map is convex, and a sum of convex functions is convex. Hence $L$ is convex. Concretely its Hessian is

\nabla^2 L(\mathbf{w}) = \sum_{(i,j)} \sigma(z_{ij})\big(1-\sigma(z_{ij})\big)\, (\mathbf{x}_i - \mathbf{x}_j)(\mathbf{x}_i - \mathbf{x}_j)^\top,

a nonnegative-weighted sum of rank-one outer products, so it is positive semidefinite — the algebraic witness of convexity. $\square$

Convexity is the whole reason the linear RankNet needs no stochastic gradient descent: there is one optimum, and a second-order convex optimization step finds it deterministically. The companion notebook fits it with L-BFGS-B and the closed-form gradient, checks the Hessian is positive semidefinite, and confirms that two different starting points land on the same weights — a global optimum, fully reproducible. (The convexity is conditional on the linear scorer; a deep RankNet, where $s$ is a neural network of $\mathbf{x}$ , is non-convex and is trained by SGD, as the original paper did.)

The gradient has a structure that is the bridge to everything downstream.

Proposition 1 (The gradient factorizes into per-document lambda forces).

Define the pairwise force $\lambda_{ij} = -\sigma\!\big(-(s_i - s_j)\big)$ for each preference pair $(i,j)$ , and the per-document force $\lambda_i = \sum_{j} \lambda_{ij}$ , summing over the pairs that involve document $i$ (with the sign that pushes the higher-graded document up and the lower one down). Then

\nabla_{\mathbf{w}} L \;=\; \sum_{(i,j)} \lambda_{ij}\,(\mathbf{x}_i - \mathbf{x}_j) \;=\; \sum_i \lambda_i\,\mathbf{x}_i.

The gradient is a weighted sum of feature vectors, one weight per document — the net force the training pairs exert on each document.

Panel C draws these forces on the worked query. The single grade-3 document is under-ranked by the learned scorer — it sits at rank 6 — and it feels by far the strongest upward $\lambda$ , because it loses many preference comparisons it should win. This per-document view is the bridge to LambdaRank: that method keeps the same force structure but multiplies each pairwise $\lambda_{ij}$ by the $|\Delta\text{NDCG}_{ij}|$ that swapping the pair would cause, so swaps near the top of the list — where the rank discount is steep — pull harder than swaps in the tail. LambdaRank and its tree-boosted cousin LambdaMART are the workhorses of production learning-to-rank; we name them here and stop, because the weighting they add is motivated by the theorem of the next section.

Why learning to rank exists: rank metrics are flat

It would seem more direct to skip the surrogate and optimize NDCG or MAP itself. The reason no one does is a theorem.

Theorem 2 (NDCG and MAP are piecewise-constant in the scores).

Regard NDCG@ $k$ (or MAP) as a function of the score vector $\mathbf{s} = (s_1, \dots, s_N)$ , with the grades fixed. This function is piecewise-constant: it is locally constant on every region of score space where no two scores are tied, and it changes value only across the hyperplanes $s_i = s_j$ where two documents swap. Consequently its gradient is zero almost everywhere, with jump discontinuities at the swaps — it cannot be optimized by gradient descent.

Proof. The ranking is the argsort of $\mathbf{s}$ . On any open region of score space where all scores are distinct and no pair is tied, a small perturbation of $\mathbf{s}$ leaves the argsort unchanged, so the ranking — and therefore NDCG, which depends on $\mathbf{s}$ only through the ranking and the fixed grades — is constant. Its gradient is the zero vector throughout that region. The ranking can change only when the perturbation crosses a tie hyperplane $s_i = s_j$ , where two adjacent documents exchange places and NDCG jumps discontinuously. $\square$

This is the reason the field exists. A metric whose gradient is zero almost everywhere carries no descent direction; you cannot follow it downhill. So learning to rank optimizes a surrogate — a smooth loss whose minimization tends to improve the metric — and RankNet’s pairwise logistic is the canonical choice. Panel B makes the contrast visible. As the weights morph from the weakest single leg toward the learned combiner, NDCG climbs as a staircase — flat stretches punctuated by jumps, from $0.514$ up to $0.767$ — while the pairwise loss we actually optimize falls smoothly the whole way. The metric is the thing we want; the surrogate is the thing we can descend.

Ranking is not regression

The pointwise reduction looks like the safe, simple choice — least squares, a closed-form solution, a calibrated prediction of the grade. But fitting the grade is not the same as fitting the order, and the two objectives can genuinely disagree.

Proposition 2 (A pointwise-optimal model can lose on NDCG (demonstrated)).

There exist labeled rankings on which the least-squares (pointwise) model has strictly lower mean-squared error than a pairwise RankNet model, yet strictly worse NDCG. Order beats calibration.

Construction. Take two queries. Query 1 has high grade magnitude but few documents; query 2 has low magnitude but more documents, hence more preference pairs. A shared order feature has a contested best sign: query 1 (whose large grade dominates the squared error) wants a positive weight, query 2 (whose many pairs dominate the pairwise count) wants a negative one. A second feature is constant within each query — order-irrelevant, but useful to least squares for fitting each query’s grade level, and exactly zero in the pairwise gradient because every within-query difference of a constant vanishes. Least squares therefore spends the second feature on calibration and picks the magnitude-favored sign on the first, mis-ordering query 2; RankNet ignores the second feature and picks the count-favored sign, fixing query 2 at the cost of query 1 and winning on the mean. The numbers: pointwise mean-squared error $0.80$ versus pairwise $1.21$ , yet pointwise mean NDCG $0.715$ versus pairwise $0.815$ . The pointwise model is the global MSE minimizer — so its MSE is unbeatable — and it still ranks worse. $\square$

On the real finance corpus the two reductions come out nearly tied (pointwise NDCG $0.768$ , pairwise $0.767$ ), because the corpus is forgiving enough that calibration and order mostly agree — so the constructed witness, not the corpus, is the rigorous anchor. But the lesson it isolates is the one that motivates the whole pairwise program: a model can be perfectly calibrated and still rank badly, so optimizing for the grade is the wrong objective when the grade is only a means to an order.

Learned fusion of the three legs

The finance payoff is a fusion problem. The three legs read different token windows of each document, so each is a partial view: the lexical leg catches exact term matches, the dense leg catches pooled semantic similarity, the late-interaction leg catches token-level alignment, and each recalls relevant documents the others miss. The capstone combined them with reciprocal-rank fusion — a fixed, unsupervised rule that adds $1/(c + \text{rank})$ across the legs. Learning to rank is the supervised alternative: instead of fixing the combination, learn it from labeled preferences.

Algorithm 1 (Learning a fusion ranker over the three legs).

Each document’s feature vector is the triple of standardized leg scores $\mathbf{x}_d = [\,s_{\text{lex}}, s_{\text{dense}}, s_{\text{late}}\,]$ . Fit a RankNet weight vector $\mathbf{w} \in \mathbb{R}^3$ on the preference pairs of the training queries by minimizing the convex pairwise loss, then score every document of a held-out query by $s_d = \mathbf{w}^\top\mathbf{x}_d$ and rank by score. The learned weights say how much to trust each leg.

Trained on 24 queries and evaluated on the 16 held out, the learned weights are $[\,0.70,\ 1.44,\ 0.68\,]$ — it leans hardest on the dense leg but keeps real weight on the other two. On the held-out queries the learned ranker reaches recall@10 of $0.637$ and NDCG@10 of $0.767$ , against the best single leg (dense, recall $0.588$ ) and reciprocal-rank fusion (recall $0.619$ , NDCG $0.733$ ). It beats every single leg comfortably and edges out RRF. Panel D shows the bars and a training-set-size slider: with a single labeled query the learned ranker is shaky and sits below RRF; by two queries it has overtaken RRF, and it plateaus from there — a three-feature linear ranker is data-efficient, but it still has to earn its advantage over the first labeled examples.

§ Learning to fuse a filings, transcripts, and news retriever text audio news

The production system this series tracks retrieves across 10-K and 10-Q filing text, earnings-call transcripts, and market news — heterogeneous sources that a single retriever serves unevenly. Lexical matching nails a named entity or a reported figure; dense retrieval catches a paraphrased disclosure; late interaction aligns a query phrase to a specific clause. Reciprocal-rank fusion combines their rankings with one fixed rule and no training. A learned RankNet, given a few dozen labeled queries, instead learns how much to weight each leg — and on this corpus it beats RRF and every single leg on held-out queries. The honest caveat is in the size of the win: the $+0.019$ recall margin over RRF sits inside the confidence interval (standard error $0.040$ on 16 test queries), so the defensible claim is that the learned ranker matches RRF and clearly beats any single leg — not that learned fusion is universally better. The corpus here is a synthetic complementary-view token cloud standing in for those documents; every value illustrates the mechanism, not a benchmark.

Beyond pairwise: the listwise preview

The pairwise reduction has a blind spot that the rigor flag named: it is position-blind. A swap at ranks 1–2, where the rank discount is steep and the user is looking, costs the same one unit of pairwise loss as a swap at ranks 99–100, where no one will ever look. RankNet treats them identically.

Remark (What the pairwise reduction misses).

Two families of methods address the position blindness. The listwise approach defines a loss on the whole ranked list rather than on isolated pairs: ListNet maps the scores to a probability distribution over permutations (via the Plackett–Luce model) and minimizes the cross-entropy against the ideal list’s distribution. The $\Delta\text{NDCG}$ -weighted approach keeps the pairwise structure but scales each force by the metric change it would cause — LambdaRank and LambdaMART. And the modern listwise direction lets a large language model read the candidate list and emit a permutation directly — the RankGPT-style listwise rerankers. Each is a response to the same observation: the ranking metric weights the top of the list, and a loss that wants to optimize it should too.

These are the topics that build on this one. What this topic established is the foundation they share: ranking reduces to supervised learning; the rank metric is piecewise-constant and so cannot be optimized directly; the pairwise logistic is the convex surrogate that can; and its gradient factorizes into the per-document forces that every method downstream reshapes.

⚠ Rigor flag

The honest summary. RankNet’s pairwise logistic is a surrogate — minimizing it does not guarantee minimizing NDCG or MAP, the consistency gap that motivates LambdaRank’s $\Delta\text{NDCG}$ reweighting. Its convexity holds only for the linear scorer; a deep RankNet is non-convex and is trained by SGD. The headline that learned fusion beats RRF is one synthetic corpus, pinned to the observed winner after running, and its margin over RRF is within sampling noise — the robust claim is that the learned ranker matches RRF and clearly beats every single leg. The pairwise reduction is position-blind, the gap listwise objectives address. And the relevance grades are exact-MaxSim oracle tertiles, a neutral stand-in for human editorial judgments, not a benchmark.