Retrieval systems do not compare text the way humans do. They first convert text into vectors, then compute similarity using geometry. Computers do not read meaning; they compute orientation, distance, and density within geometric coordinates known as a Vector Space.
Whether you are optimizing search relevancy boundaries or debugging a Retrieval-Augmented Generation (RAG) system, understanding the mathematical mechanics of vector similarity is vital. Let's break down the foundational models of vector representation, moving from foundational geometry to token-based lexical metrics, and finally evaluating the modern era of dense semantic embedding spaces.
1. Cosine Similarity from Scratch
To evaluate text inside a vector space, we represent documents and search queries as multi-dimensional directional vectors. The closer two vectors align in space, the more contextually similar their underlying text segments are considered.
To determine this alignment without getting tripped up by document length variances, systems rely on Cosine Similarity. Before looking at the code, let's break down the individual mathematical operations that make this possible.
The Component Primitives
- The Dot Product (): This tracks both the structural alignment and the scale metrics of two vectors. It is calculated by multiplying matching coordinate dimensions and summing the total:
- The Vector Norm ( ): Specifically the Euclidean () norm, this represents the absolute spatial distance from the vector's tip back to the coordinate origin. It functions as a measure of magnitude:
- Normalization: This takes a vector and rescales it so its total spatial length equals exactly 1.0, projecting it cleanly onto a unit hypersphere:
The Equation
By combining these primitives, we arrive at the standard formula for Cosine Similarity:
We prioritize Cosine Similarity over Euclidean distance to handle document length variance. In text processing, if a document is simply duplicated back-to-back, its core semantic meaning remains identical, but its raw vector representation stretches deeply out into Euclidean space, yielding a massive distance distortion.
Cosine similarity elegantly bypasses this by normalizing the dot product against the product of the vector lengths ( norms). This completely isolates the directional angle on a unit hypersphere, making the comparison size-invariant.
I accumulate the dot product and squared L2 norms in one pass. Then I guard against zero vectors to avoid division by zero. Finally, I divide the dot product by the product of the vector magnitudes. I also validate that the two vectors have the same dimensionality, because otherwisezipwould silently truncate the longer vector. I've added an explicit guard against zero-magnitude vectors to prevent runtime division-by-zero exceptions if the upstream retrieval pipeline passes unindexed or completely empty document arrays.
2. Token-Based Representation: TF-IDF from Scratch
TF-IDF (Term Frequency-Inverse Document Frequency) is the definitive statistical baseline for token-based Sparse Representations.
In a sparse vector space, every unique word in an overall corpus vocabulary is assigned an entirely separate, orthogonal geometric axis. If your database contains a vocabulary of 50,000 unique terms, every single document is converted into a 50,000-dimensional vector. For any given document, the vast majority of these dimensions sit at zero—hence the term sparse.
Deconstructing the Weights
To populate these coordinates without letting generic stop words drown out valuable information, TF-IDF splits the calculations into two primary steps:
- Term Frequency (TF): Raw term counts can be misleading because relevance does not increase linearly with repetition. A document containing a term twenty times is not necessarily twenty times more relevant than one containing it once. To account for this effect, term frequency is often normalized by document length:
Another common variation is log-scaled term frequency, which reflects the intuition that each additional occurrence of a term contributes progressively less information than the previous one. By applying a logarithmic transformation to term counts, log-scaled TF reduces the influence of excessively frequent terms and prevents highly repetitive words from disproportionately dominating the document representation.
- Inverse Document Frequency (IDF): While term frequency measures how often a term appears within a document, document frequency (DF) measures how many documents in the corpus contain that term. Terms with high DF values (e.g., "the", "and", or "system") appear in many documents and therefore provide little discriminative value. IDF reduces their influence by measuring how rare a term is across the entire corpus collection of size
N:
By multiplying these metrics (), we highlight terms that occur frequently within a specific document while remaining relatively uncommon across the broader corpus. The resulting scores help identify the document's most distinctive and informative keywords.
Build the Corpus TF-IDF Matrix
I’ll start by building the TF-IDF representation for the corpus. The first step is to create a vocabulary, where each unique token becomes one dimension in the vector space.
Then I build a term-frequency matrix with shape
(num_docs, vocab_size). For each document, I count how often each word appears and normalize by the document length, so longer documents do not automatically get larger scores just because they contain more tokens.After that, I compute document frequency, which tells me how many documents contain each term. From that, I compute a smoothed IDF value using
log((N + 1) / (df + 1)) + 1. The smoothing avoids division-by-zero issues and keeps the IDF value well-defined.Finally, I multiply the TF matrix by the IDF vector. The result is a document-level TF-IDF matrix, where each row represents one document and each column represents one vocabulary term. I also return the vocabulary, the word-to-index mapping, and the IDF vector because I’ll need the same indexing scheme when vectorizing queries later.
Encode the Query in the Same Vector Space
Now I’ll implement the query-side logic. The key point is that I should not build a new vocabulary for the query. The query has to be projected into the exact same vector space as the documents, using the same
word_to_index mapping and the same IDF values.I initialize a zero vector with the same dimensionality as the corpus vocabulary. Then I count the query terms and compute normalized term frequency for the query. If a query word was never seen in the corpus, I skip it, because there is no corresponding dimension in the document matrix.
I also guard against zero vectors. This can happen if the query is empty or if none of the query terms exist in the corpus vocabulary. In that case, returning zero similarity is safer than dividing by zero.
Rank Documents with Cosine Similarity
Finally, I’ll put the pieces together into a retrieval function.
In previous sections, I build the TF-IDF matrix for the corpus. That gives me one vector per document, plus the vocabulary and IDF values needed to encode future queries consistently. Then I vectorize the query using the same vector space.
Once the query vector is built, I can compare it against each document vector. For similarity, I use cosine similarity because TF-IDF vectors can differ in magnitude, and I care more about directional alignment than raw vector length. Each similarity score represents how well a document matches the query in TF-IDF space.
I store each result as a pair of document index and similarity score, then sort all documents in descending order by score. If
top_k is provided, I return only the top results; otherwise, I return the full ranking.Conceptually, this mirrors a simple retrieval pipeline: build document vectors, encode the query into the same space, score query-document similarity, and return the highest-ranked documents.
3. Semantic Synthesis: Dense Embeddings vs. Lexical Sparsity
While sparse frameworks like TF-IDF or BM25 excel at pinpoint keyword lookups, direct text matching breaks down under the vocabulary mismatch problem. If a client searches for 'cardiac arrest' and our document corpus uses 'myocardial infarction', a sparse matrix maps them to entirely separate, orthogonal axes—yielding a similarity score of zero despite their identical semantic intent.
Modern dense neural embeddings (such as foundational models from OpenAI or Google Research) solve this by mapping strings into compressed, static vector spaces (typically 768 to 1536 dimensions). Here, coordinates represent abstract, real-valued latent concepts rather than literal text tokens, capturing true semantic density and synonymity.
However, transferring purely to dense retrieval introduces an engineering trade-off: you sacrifice exact token precision. If an application requires filtering by strict alphanumeric serial IDs, a dense model can over-generalize and surface false positives. Consequently, industry-grade production systems implement a Hybrid Retrieval Blueprint: we run a sparse inverted index and a dense vector database in parallel, merging their outputs via Reciprocal Rank Fusion (RRF) to maintain hard constraint boundaries without losing semantic depth.
The Core Architectural Differences
Operational Matrix | Sparse Representations (TF-IDF / BM25) | Dense Representations (Neural Embeddings) |
Dimensionality | Massive and dynamic (vocab size: 10^4 to 10^6) | Compressed and static (typically 768 to 3072) |
Vector Density | >99% of values are exactly zero | 100% of values are non-zero floating points |
Matching Axis | Strict character tokens / keyword intersection | Hidden latent concepts / contextual meaning |
Out-of-Vocabulary | Fails completely on raw synonym variations | Maps overlapping conceptual values gracefully |
Compute Footprint | Extremely low; optimal for inverted index lookups | Requires dedicated model inference and vector engines |
Engineering Trade-Offs: Precision vs. Generalization
Does this mean dense embeddings make lexical search obsolete? Not at all. In production systems, leaning exclusively on dense embeddings introduces a new set of challenges:
- Loss of Exact Precision: Dense models trade keyword precision for abstract generalization. If a user searches for an exact serial part number like
"X-702-B", a dense encoder might generalize it as being close to other mechanical part strings, missing the exact matching document.
- The "Hallucinated Match" Risk: Because dense architectures compress strings into continuous spaces, they can occasionally return false positives—yielding high similarity scores for documents that share a similar writing tone or stylistic structure but completely lack the specific factual data required.
Hybrid Recovery
To mitigate these limitations, production enterprise search platforms rarely rely on a single representation layer. Instead, architectures deploy a Reciprocal Rank Fusion (RRF) system that runs a two-pronged, parallel execution strategy:
- The Lexical Track: An inverted index handles exact token queries, phrase filters, code identifiers, and product serial keys using BM25 or sparse vector structures.
- The Semantic Track: A dense passage retriever maps broad thematic concepts, ambiguous queries, and conversational intent variations.
By combining both outputs through an application-layer reranking engine (such as a Cross-Encoder model), systems get the best of both worlds: the strict, unyielding precision of sparse lexical constraints backed by the conceptual depth of dense semantic spaces.
- Author:Fan Luo
- URL:https://fanluo.me/article/text-similarity-and-retrieval-basics
- Copyright:All articles in this blog adopt BY-NC-SA agreement. Please indicate the source!
上一篇
模型训练的方法与实践
下一篇
Essential Evaluation Metrics for Applied ML Systems