Late Interaction and Learned Sparse Retrieval: ColBERT and SPLADE

Overview & motivation

The previous topic closed with a wall. A dual encoder pools each text into a single vector and scores by an inner product, and the relevance patterns such a model can represent are bounded by the sign-rank of the relevance matrix — for combinatorial patterns, a dimension that grows faster than any fixed embedding can follow. That is a property of the single pooled vector, not of neural retrieval. This topic is about the two architectures that route around it.

The first keeps the dense geometry but drops the pooling. Instead of one vector per document, late interaction (ColBERT) keeps one vector per token, and scores a query against a document by MaxSim: each query token finds its best-matching document token, and the matches are summed. The second keeps the lexical, inverted-index world of BM25 but learns the weights. SPLADE expands each text into a high-dimensional sparse vector over the vocabulary, with neural term weights and expansions, scored by a sparse dot product. Both escape the single-vector ceiling, by different mechanisms — one by becoming a max of many inner products rather than one, the other by living in a space whose dimension is the vocabulary size — and the honesty of this topic is in being precise about what is proved (a clean reduction) and what is demonstrated (the escape itself).

MaxSim score (sum of per-token maxima)

2.913

single pooled vector score

0.000

Each query token (row) matches its best document token (the green-bordered cell), and MaxSim sums those maxima. Three query tokens land on three distinct document tokens — a many-to-many match. A single pooled document vector (the token average) cannot point at three things at once, so it scores 0.00. With one vector per item the max is over a singleton and MaxSim is exactly the dual-encoder dot product — the provable anchor.

The first panel is the MaxSim grid: a query’s tokens against a document’s tokens, with each query token’s best match highlighted, and the contrast to a single pooled vector. The second is the escape — the single-vector wall from the previous topic and the same all-pairs pattern realized by a two-vector MaxSim model at the same per-vector dimension. The third is SPLADE: a query’s learned expansion into vocabulary terms it never mentioned, and the sparsity–quality trade-off the FLOPS regularizer controls.

Notation: tokens, MaxSim, and the sparse vocabulary

A dual encoder maps a text to one vector. A multi-vector encoder maps a text to a set of vectors, one per token: a query $q$ becomes $\{q_1, \dots, q_{m_q}\} \subset \mathbb{R}^d$ and a document $d$ becomes $\{d_1, \dots, d_{m_d}\} \subset \mathbb{R}^d$ . Late interaction scores the pair by

S(q, d) \;=\; \sum_{i=1}^{m_q} \max_{1 \le j \le m_d} \langle q_i, d_j\rangle,

the MaxSim operator: every query token contributes the similarity of its single best-matching document token. A learned sparse encoder instead maps a text to a vector $w \in \mathbb{R}^{|V|}_{\ge 0}$ over the vocabulary $V$ , almost all of whose entries are zero, and scores by the ordinary dot product $\langle w_q, w_d\rangle$ — a weighted term-overlap, computed in an inverted index. Throughout, $\langle \cdot, \cdot\rangle$ is the inner product and $\|w\|_0 = |\{j : w_j \ne 0\}|$ counts the nonzero (active) entries.

Movement 1 — MaxSim, and the m = 1 collapse

The first thing to establish is the bridge back to the dual encoder, because it is the one statement here we can prove exactly, and it pins MaxSim as a strict generalization of the score the previous topics used.

Theorem 1 (MaxSim reduces to the dot product at one vector per item).

If a query and a document each have a single token vector, $m_q = m_d = 1$ , then $S(q, d) = \langle q_1, d_1\rangle$ — exactly the dual-encoder score of DPR.

Proof (Proof).

With $m_d = 1$ the inner maximum is over a single index, so $\max_{1 \le j \le 1}\langle q_1, d_j\rangle = \langle q_1, d_1\rangle$ . With $m_q = 1$ the outer sum has one term, $S(q,d) = \langle q_1, d_1\rangle$ , which is the dual encoder’s inner-product score. $\blacksquare$

∎

The notebook makes this the load-bearing anchor: it builds a MaxSim score matrix with one vector per item and asserts it equals the imported DPR dot-product matrix to within floating-point error. Late interaction is therefore not a different kind of score; it is the dual encoder with the pooling step removed, and the pooling is exactly what the rank ceiling was about. The reason MaxSim escapes that ceiling is structural.

Proposition 1 (MaxSim is a max of linear forms, not a bilinear form).

For fixed document vectors, $S(q, d) = \sum_i \max_j \langle q_i, d_j\rangle$ is a sum of pointwise maxima of linear functions of the query tokens — a piecewise-linear, convex function of each $q_i$ . It is not a bilinear form $q^\top W d$ in pooled vectors, so the rank- $\le d$ bound that constrains a dual encoder’s score matrix does not apply to it.

The rank ceiling of the previous topics was a fact about matrices $S = Q G^\top$ : a product through a $d$ -dimensional bottleneck has rank at most $d$ . MaxSim breaks the product. Each query token picks a different document token to match, so the score is a maximum over $m_d$ linear scorers, and a maximum of linear functions is not linear — it is the building block of piecewise-linear expressivity, the same primitive a one-hidden-layer max-network uses. The laboratory’s first panel shows this concretely: three query tokens against four document tokens, each query token’s maximum landing on a distinct document token (cosines $0.978$ , $0.995$ , $0.940$ ) for a MaxSim of $2.913$ , where a single pooled document vector — the average of the four token vectors — scores $0$ , because it cannot point at three different things at once.

Movement 2 — multi-vector escapes the single-vector wall

Proposition 1 says MaxSim can exceed the rank ceiling. Whether it does, at a useful number of vectors, is a separate and harder question — and here we are careful, because the honest answer is that it is demonstrated, not proved.

Proposition 2 (The multi-vector escape (demonstrated)).

On the all-pairs relevance pattern of the embedding-dimension topic — one query per document pair, relevant to both — a MaxSim model with two vectors per document realizes, at per-vector dimension $d = 4$ , corpora that a single vector cannot. With free, perfectly optimized embeddings and the same optimizer for both, the single-vector model’s row-order accuracy collapses past a critical corpus size ( $1.0$ through $8$ documents, then $0.93$ at $10$ and $0.86$ at $12$ ) while the two-vector MaxSim model stays at $1.0$ throughout; its critical $n$ is at least $12$ against the single vector’s $8$ .

The comparison is deliberately a single cloud: the single-vector baseline is the same MaxSim optimizer run with one vector per item, so the gap is the effect of the extra vectors and not a difference between two pieces of code — the $m = 1$ case is, by Theorem 1, literally the dual encoder. We state this as a proposition about what the laboratory measures, not as a theorem, and the honest accounting below is blunt about why: there is no known multi-vector sign-rank lower bound. The paper that proved the single-vector ceiling explicitly leaves the multi-vector case open, and the critical- $n$ values here are best-case demonstrations on free embeddings, optimizer-dependent, not the true capacities. What is solid is the direction — more vectors per document lift the wall — and the mechanism, Proposition 1.

The escape is not free, and the cost is storage. A single-vector index keeps one vector per document; late interaction keeps one per token, so at a representative $32$ tokens per document and a $128$ -dimensional per-token embedding the index is $32\times$ larger — in the laboratory’s figures, $128{,}000$ floats become $4{,}096{,}000$ . ColBERTv2’s residual compression and PLAID’s centroid pruning, the subject of the multi-vector ANN topic that follows, are the engineering that makes this affordable; here it is the honest price of the expressivity.

Movement 3 — SPLADE, the lexical escape

The other escape never enters the dense low-dimensional space at all. It goes the opposite way: into the vocabulary, a space of tens of thousands of dimensions, kept tractable by sparsity.

Definition 1 (The SPLADE representation and the FLOPS regularizer).

For input tokens $t_1, \dots, t_n$ and a model that produces a logit $\ell_{ij}$ for vocabulary term $j$ at position $i$ , the SPLADE weight of term $j$ is

w_j \;=\; \max_{1 \le i \le n} \log\!\big(1 + \mathrm{ReLU}(\ell_{ij})\big),

a non-negative, log-saturated, max-pooled activation over the vocabulary. Sparsity is trained in by the FLOPS regularizer $\mathcal{L}_{\text{FLOPS}} = \sum_{j \in V} \bar{a}_j^{\,2}$ , where $\bar{a}_j$ is the mean weight of term $j$ over a batch — the expected number of nonzero query–document term products, the quantity that sets the cost of an inverted-index traversal.

Two things distinguish this from BM25. The $\log(1 + \mathrm{ReLU}(\cdot))$ activation is a learned re-weighting: the model decides each term’s importance from context, where BM25 reads it off corpus statistics. And the max over positions allows expansion — a term can receive weight even if it never appears in the text, because some input token activated its logit. Expansion is the cure for the disease BM25 cannot treat: vocabulary mismatch, when a query and a relevant document mean the same thing in different words. The laboratory’s third panel makes it concrete on the finance corpus. The query “borrowing costs” shares no term with the on-point filing, which speaks of interest, rate, and exposure; BM25 scores that filing exactly $0$ and cannot rank it at all. SPLADE expands the query into $\{\text{interest}: 1.39,\ \text{rate}: 1.25,\ \text{exposure}: 0.92,\ \text{margin}: 0.69\}$ — every weight a learned expansion, none of them a literal query term — and retrieves the filing at rank one.

Sparsity is the knob that makes this deployable, and it trades against quality. Raising the pruning threshold removes low-weight terms, so the active-term count and the FLOPS value both fall ( $\|w\|_0$ from $100$ down to $79$ active entries across the corpus, FLOPS from $16.1$ to $12.0$ in the laboratory’s sweep) — but past a point the bridging expansion terms themselves are pruned, and the vocabulary-mismatch fix breaks. That is the entire engineering tension of learned sparse retrieval: expand enough to bridge mismatches, stay sparse enough to keep the inverted index fast.

Serving at scale, and the relation to the rest of the track

Both escapes have a serving story this topic only points at. A learned sparse query is a weighted term query, so SPLADE rides exactly the inverted index and the dynamic-pruning machinery already built — the FLOPS regularizer is what keeps that traversal cheap. Late interaction needs a different engine: with many vectors per document, even the candidate-generation step is a multi-vector nearest-neighbor problem, solved by centroid pruning and residual compression (PLAID). That multi-vector ANN problem is its own topic; here it is enough to know the escape from the rank ceiling creates a new indexing problem the next topics solve.

Proposition 3 (What the laboratory measures).

The panels run on deterministic synthetic setups, not trained transformers. Panel A’s token vectors are designed unit vectors; its MaxSim and pooled scores are exact. Panel B’s escape curve and critical- $n$ values come from free-embedding optimization with the same optimizer for the single-vector ( $m=1$ ) and MaxSim ( $m=2$ ) models, reusing the all-pairs construction and the qrel loss of the embedding-dimension topic. Panel C’s SPLADE weights come from a synthetic association matrix standing in for an MLM head, scored over BM25’s actual finance-corpus vocabulary. Every measured number — the escape accuracies, the critical- $n$ , the expansion weights, the sparsity trade-off — is owned by the companion notebook and mirrored here to the decimal; the laboratory recomputes only closed forms (the active-term counts and the MaxSim of the baked token vectors).

Finance case study

§ Finance case study text pdf

The two escapes are the two answers to the two ways a single-vector financial retriever fails. The first failure is combinatorial, and the embedding-dimension topic measured it: analyst queries that ask for a combination of documents — “issuers exposed to both rising rates and supply-chain disruption” — induce relevance patterns whose sign-rank outruns any fixed embedding dimension, and a single pooled vector that perfectly answers single-company retrieval cannot answer them. Late interaction is the dense answer. Because each query token matches its own best document token, a query that names two distinct risk concepts can have one token land on the rate-exposure passage and another on the supply-chain passage of the same filing — a many-to-many match a single pooled vector, which must average both concepts into one direction, cannot make. The escape demonstrated on the all-pairs pattern is exactly this token-level conjunction.

The second failure is lexical mismatch, and it is older than embeddings. A filing discloses “net interest margin sensitivity”; an analyst asks about “borrowing costs”. They are about the same risk, share no words, and BM25 — for all its calibration — scores their overlap at zero. SPLADE is the lexical answer: it expands the query into the filing’s own vocabulary (interest, rate, exposure) and retrieves it, while staying in the inverted index the production system already runs, so the FLOPS regularizer keeps the expanded queries cheap enough to serve. A production retrieval stack over filings, transcripts, and analyst notes therefore tends to run both — a late-interaction dense leg for combinatorial semantic queries and a learned-sparse lexical leg for term-level precision and mismatch — fused into one ranking, because the single-vector retriever has exactly the two blind spots these two architectures were built to cover.

Honest accounting

⚠ Rigor flag

One result here is proved and the headline is honestly empirical. MaxSim reducing to the dual-encoder dot product at one vector per item (Theorem 1) is exact, and MaxSim being a max of linear forms rather than a bilinear form (Proposition 1) — hence not bound by the rank ceiling — is immediate. But that the escape happens at a small number of vectors (Proposition 2) is a best-case demonstration by free-embedding optimization, not a theorem: there is no known multi-vector sign-rank lower bound, and the LIMIT paper that proved the single-vector ceiling explicitly defers the multi-vector case. The single-vector baseline is re-derived with the same optimizer as the MaxSim model, so the measured gap is the multi-vector effect and not a difference between optimizers — but the absolute critical- $n$ values are optimizer-dependent best cases, not true capacities. SPLADE’s FLOPS regularizer is an empirically effective heuristic, not derived from a principle, and its relation to BM25 is conceptual — a learned re-weighting and expansion in the same lexical space — not a formal generalization. The token embeddings and the MLM logits driving SPLADE expansion are synthetic stand-ins built on the InfoNCE, DPR, and BM25 topics, not trained transformers, so the finance numbers are illustrations of the mechanisms, not benchmark results. And the expressivity of late interaction is bought with storage: one vector per token, $32\times$ a single-vector index in the laboratory’s figures, which ColBERTv2 and PLAID mitigate but do not erase.

Late interaction lifts the rank ceiling and, in doing so, creates a new problem: a document is now a set of vectors, and retrieving over many such sets at scale is a multi-vector nearest-neighbor problem that ordinary single-vector indexes do not solve. That problem — centroid pruning, residual compression, the PLAID engine — is where the track goes next, the indexing counterpart to the expressivity gain proved here. The cross-encoder, which this topic and DPR both kept invoking as the rank-free but un-precomputable extreme, is the other branch: where late interaction keeps the matching cheap by deferring it to a sum of token dot products, a cross-encoder pays a full joint forward pass for the most expressive score of all.