Bayesian Relevance Estimation

Every vector-based retrieval system in production today uses cosine similarity. It has been the default since word2vec in 2013. Nobody questions it. But if you sit down and derive the mathematically optimal scoring function for retrieval from first principles using Bayes' theorem, you get something different.

You get the dot product minus a magnitude penalty term.

This is not a minor tweak. It changes the ranking of results in any corpus where embedding magnitudes vary, which is every real corpus. And the derivation reveals something deeper: cosine similarity was never the right metric. It is a special case of the optimal score that only holds under conditions that never exist in practice.

This post walks through the full derivation, proves the optimality result, and extends the framework with structural document priors and adaptive retrieval stopping. All the math is derived from scratch. No existing framework is copied.

The problem: similarity is not relevance

The fundamental assumption behind vector retrieval is that semantic similarity in embedding space correlates with relevance. You embed your query, embed your documents, and rank by cosine similarity:

But this is answering the wrong question. Retrieval is not asking "which documents are similar to my query?" It is asking "which documents contain the answer to my query?" These are different questions, and the mathematical objects that answer them are different.

The correct object is the posterior probability:

where $R_i = 1$ means document $d_i$ is relevant (contains information needed to answer the query). Cosine similarity is an ad hoc proxy for this posterior. The question is: what does the actual posterior look like, and how much does it differ from cosine similarity?

Deriving the optimal score

Setup

We have a query $q$ with embedding $\mathbf{e}_q \in \mathbb{R}^n$ and documents $d_1, \dots, d_N$ with embeddings $\mathbf{e}_1, \dots, \mathbf{e}_N \in \mathbb{R}^n$. Each document is either relevant ($R_i = 1$) or not ($R_i = 0$).

We need a probabilistic model that connects query embeddings to document relevance.

The likelihood model

If document $d_i$ is relevant to query $q$, the query embedding should be "close" to the document embedding in the vector space. We model this as a Gaussian:

This says: if $d_i$ is relevant, the query embedding is drawn from a Gaussian centered on $\mathbf{e}_i$ with variance $\sigma^2$ in each dimension. The parameter $\sigma^2$ controls how tightly queries cluster around their relevant documents in embedding space.

If $d_i$ is not relevant, the query embedding comes from a background distribution. We model this as a zero-centered Gaussian with larger variance:

The zero-centered background is a standard assumption. It means irrelevant documents tell you nothing about where the query lands in embedding space. The variance $\sigma_0^2 > \sigma^2$ is larger because the query could be anywhere.

Applying Bayes' theorem

For each document, we want:

For ranking, we only need the log posterior odds (the denominator $P(\mathbf{e}_q)$ is constant across documents):

where $\pi_i = P(R_i = 1)$ is the prior probability that document $i$ is relevant.

Expanding the log-likelihood ratio

Write out the Gaussian log-PDFs:

The log-likelihood ratio is:

Now expand $\|\mathbf{e}_q - \mathbf{e}_i\|^2 = \|\mathbf{e}_q\|^2 - 2(\mathbf{e}_q \cdot \mathbf{e}_i) + \|\mathbf{e}_i\|^2$:

The first term and the last term do not depend on $i$. They are constants for a given query. For ranking documents, we can drop them:

Since $\sigma^2$ is a positive constant, dividing by it does not change the ranking. The Bayesian Relevance Estimation (BRE) score is:

That is the dot product minus half the squared magnitude of the document embedding.

What this formula means

The dot product, not cosine similarity

The first term is the dot product $\mathbf{e}_q \cdot \mathbf{e}_i$, not cosine similarity. The dot product equals $\|\mathbf{e}_q\| \cdot \|\mathbf{e}_i\| \cdot \cos\theta$ where $\theta$ is the angle between the vectors. It preserves magnitude information from both vectors.

Cosine similarity divides out both magnitudes. The Bayesian derivation says you should not do that. Magnitude carries signal.

This matches the 2025 empirical findings that embedding magnitude correlates with informativeness, specificity, and retrieval quality. The Bayesian framework explains why: magnitude is part of the likelihood ratio. Discarding it loses information.

The magnitude penalty

The second term $-\|\mathbf{e}_i\|^2 / 2$ penalizes documents with very large embedding magnitudes. Without this term, the dot product alone would always favor high-magnitude embeddings, regardless of direction. The penalty creates a balance: you want documents whose embeddings are large enough to carry signal, but not so large that they dominate by magnitude alone.

The optimal magnitude for a document embedding (maximizing the score for a perfectly aligned query) is found by taking the derivative with respect to $\|\mathbf{e}_i\|$ and setting it to zero:

The optimal document magnitude is the projection of the query onto the document direction. Documents whose magnitude matches this projection score highest. This is a meaningful quantity: it says the embedding magnitude should reflect how much of the query's information content the document covers.

Theorem: when cosine similarity is optimal

Theorem. BRE ranking and cosine similarity ranking produce identical document orderings if and only if all document embeddings have equal magnitude.

Proof. If $\|\mathbf{e}_i\| = c$ for all $i$, then:

The term $c^2/2$ is constant across documents. Ranking by BRE is equivalent to ranking by $\mathbf{e}_q \cdot \mathbf{e}_i$. Since $\|\mathbf{e}_i\| = c$:

The factor $\|\mathbf{e}_q\| \cdot c$ is constant. Ranking by dot product is equivalent to ranking by $\cos(\mathbf{e}_q, \mathbf{e}_i)$.

Conversely, if magnitudes are not all equal, there exist query-document pairs where the dot product and cosine similarity disagree on ranking. Let $\|\mathbf{e}_a\| > \|\mathbf{e}_b\|$ with $\cos(\mathbf{e}_q, \mathbf{e}_a) < \cos(\mathbf{e}_q, \mathbf{e}_b)$ but $\mathbf{e}_q \cdot \mathbf{e}_a > \mathbf{e}_q \cdot \mathbf{e}_b$. Cosine ranks $b$ above $a$; the dot product (and BRE, for appropriate magnitudes) ranks $a$ above $b$. QED.

Corollary. In every real embedding corpus, document magnitudes vary. Therefore cosine similarity is never the optimal ranking function for retrieval.

Optimality via Neyman-Pearson

The BRE score is not just "better than cosine." It is provably optimal.

The Neyman-Pearson lemma states that for binary hypothesis testing (relevant vs. irrelevant), the likelihood ratio test is the most powerful test at any significance level. No other test achieves higher recall at the same precision, or higher precision at the same recall.

The BRE score is the log-likelihood ratio (plus log prior odds, which are constant under uniform prior). Therefore:

under the Gaussian embedding model. Any deviation from BRE, including using cosine similarity, results in a ranking that is strictly less powerful when the conditions of the theorem hold (non-uniform magnitudes).

In information-theoretic terms, the Fisher information about relevance contained in the BRE score decomposes as:

Cosine similarity uses only $I_{\text{direction}}$. The magnitude component $I_{\text{magnitude}} \geq 0$, with equality only when magnitudes are constant. For any corpus with magnitude variance, you are leaving discriminative information on the table by using cosine similarity.

Extension: structural priors

The base BRE score uses a uniform prior ($\pi_i = K/N$ for all documents). But documents have structure. Sections in the same chapter are more likely to be co-relevant. Cross-referenced sections are more likely to be co-relevant. We can encode this as a non-uniform prior.

Markov Random Field on document structure

Let $G = (V, E)$ be the document structure graph where vertices are documents/chunks and edges connect structurally related pairs (same section, adjacent paragraphs, cross-references). Define a joint prior on relevance using a pairwise Markov Random Field:

where:

• $w_{ij} > 0$ is the coupling strength between structurally related documents (encourages co-relevance)
• $b_i$ is a document-level bias (e.g., higher for sections whose headings match query terms)
• $Z$ is the partition function ensuring normalization

The marginal prior $P(R_i = 1 \mid G)$ gives each document a structural relevance score. For tree-structured documents (which covers most hierarchical content: books, reports, documentation), exact marginals can be computed in $O(N)$ time using belief propagation.

The full BRE score with structural prior becomes:

The $\sigma^2$ scaling ensures the prior and likelihood are on the same scale. When $\sigma^2$ is small (embeddings are precise), the likelihood dominates. When $\sigma^2$ is large (embeddings are noisy), the structural prior has more influence.

Extension: adaptive retrieval stopping

Fixed top-k retrieval is a design flaw, not a feature. Easy queries might need 2 documents. Hard queries might need 15. The right number depends on the posterior distribution, and BRE gives us that distribution.

From scores to posteriors

Apply the sigmoid function to convert BRE scores to calibrated probabilities:

where $\sigma(x) = 1/(1 + e^{-x})$ is the sigmoid and $\pi$ is the base rate prior.

Stopping criterion

Rank documents by posterior probability. After retrieving the top $k$ documents, the probability that no relevant document was missed is:

This is the product of "probability of irrelevance" for every unretrieved document. Retrieve documents in order until:

where $\theta$ is the desired confidence (e.g., 0.95).

For a concrete example with sorted posteriors $[0.92,\; 0.78,\; 0.45,\; 0.12,\; 0.06,\; 0.03,\; 0.01]$:

At $k = 5$ with $\theta = 0.95$, the system stops. It retrieved exactly what it needed. A fixed top-3 would have missed relevant context. A fixed top-10 would have included noise.

The beauty of this approach: the retrieval budget adapts to query difficulty automatically. Simple factual queries (one document has posterior > 0.9, all others < 0.05) stop at $k = 1$. Complex multi-source queries keep retrieving until confidence is met.

The complete BRE algorithm

Putting it all together:

Indexing (once):

1. Embed all documents to get $\mathbf{e}_1, \dots, \mathbf{e}_N$
2. Precompute $\|\mathbf{e}_i\|^2$ for each document
3. (Optional) Build structure graph $G$ and compute marginal priors via belief propagation

Retrieval (per query):

1. Embed query to get $\mathbf{e}_q$
2. Compute BRE scores: $\text{BRE}(q, d_i) = \mathbf{e}_q \cdot \mathbf{e}_i - \|\mathbf{e}_i\|^2 / 2 + \sigma^2 \log P(R_i \mid G)$
3. Rank documents by score
4. Convert top scores to posteriors via sigmoid
5. Retrieve documents in rank order until $P(\text{complete}) \geq \theta$

The computational overhead vs. standard cosine similarity is minimal. The dot product is already computed by any vector database (it is cheaper than cosine similarity since you skip the normalization). The magnitude penalty is a precomputed constant per document. The structural prior is precomputed at index time. The adaptive stopping adds a product computation over posteriors, which is $O(k)$.

When does this matter?

Not every corpus benefits equally from BRE. The improvement over cosine similarity is a function of how much the conditions of the theorem are violated.

Magnitude variance. Compute $V_m = \text{Var}(\|\mathbf{e}_i\| : i = 1, \dots, N)$. If $V_m \approx 0$, BRE and cosine give nearly identical rankings. If $V_m$ is large, BRE has significantly more discriminative power. In practice, most embedding models produce magnitude variance. OpenAI's text-embedding-3-large has coefficient of variation around 0.15-0.25 on typical corpora. Cohere's embed-v3 is similar. Only models that explicitly L2-normalize their output (forcing all vectors to unit length) eliminate magnitude variance, and in doing so, they discard information that the BRE framework says you should keep.

Structural information. If your documents have hierarchical structure (headings, sections, cross-references), the structural prior improves retrieval for multi-hop questions where relevant information is spread across related sections. If your corpus is flat (isolated snippets with no structure), the structural prior adds nothing.

Query difficulty distribution. If most queries are simple (one relevant document, obvious match), fixed top-k and cosine similarity work fine. If queries vary in difficulty, adaptive stopping prevents both under-retrieval (missing context for hard queries) and over-retrieval (adding noise for easy queries).

What this does not solve

BRE operates within the embedding paradigm. It assumes your embedding model produces vectors where relevance correlates with proximity in the vector space. If the embedding model is bad (poor training data, wrong domain, insufficient model capacity), BRE will rank garbage more optimally, but it is still garbage.

BRE also does not solve the query-document mismatch problem where the query asks about intent but the document describes content. That is a fundamental limitation of any embedding-based approach. Reasoning-based systems like PageIndex address this by sidestepping embeddings entirely.

And BRE does not eliminate the chunking problem. If relevant information is split across chunks with no structural connection, neither the magnitude term nor the structural prior can recover the missing context. Better chunking strategies (semantic chunking, hierarchical chunking) are orthogonal improvements.

The takeaway

Cosine similarity became the default for vector retrieval through historical momentum, not mathematical justification. When you derive the optimal scoring function from Bayes' theorem, you get the dot product minus a magnitude penalty. This is a stronger result than "here is a slightly better metric." It is a proof that cosine similarity is suboptimal for any corpus with non-uniform embedding magnitudes, and a constructive derivation of what you should use instead.

The practical changes are small: replace cosine with dot product in your vector search, subtract a precomputed magnitude term, and optionally add structural priors. The theoretical foundation is what matters: retrieval is an inference problem, not a similarity problem, and treating it that way gives you a principled framework for every design decision in the pipeline.

Sources

• Neyman-Pearson Lemma (original 1933 paper)
• Cosine Similarity Limitations in Embedding Retrieval (2025)
• LaRA: Benchmarking RAG and Long-Context LLMs (2025)
• MIRAGE: Metric-Intensive RAG Evaluation Benchmark (2025)
• Markov Random Fields in Information Retrieval
• PageIndex: Vectorless Reasoning-Based RAG (2025)
• Embedding Vector Magnitude and Semantic Signals (2025)