What is semantic similarity, and how is it measured?

In this article

1. Before you start
2. Why interviewers ask this
3. Semantic similarity from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the embeddings and vector search learning path.

Why interviewers ask this

Semantic similarity sits under search, recommendation, duplicate detection and retrieval-augmented systems, so a question about it quickly becomes a question about how a whole retrieval stack behaves. Interviewers use it to see whether a candidate can explain a score instead of only calling a library function.

A weak answer says "we embed the texts and take the cosine." A strong answer says what the cosine ignores, why the encoder must be trained for the job, and how the score is checked. A Staff-level answer adds cost and failure: each text is encoded once so comparison is cheap, the score is a proxy for human judgment, and the best embedding model depends on the task. The questions below are the ones that are reported in candidate accounts of AI and machine learning loops: how to evaluate embeddings, and how to combine keyword search with similarity.

Semantic similarity from first principles

Semantic similarity

Semantic similarity is a number that says how close in meaning two pieces of text are, computed by comparing the vectors an embedding model produces for them, and judged good when it agrees with what people say about the same pairs.

Cosine similarity

Cosine similarity is a measure of two vectors that divides their dot product by the product of their lengths, so it depends only on the angle between them: the same direction scores the top value and unrelated directions score zero.

Semantic Textual Similarity task

The Semantic Textual Similarity (STS) task is a shared evaluation in which a system gives each sentence pair a real-valued similarity score and is graded on how well its scores agree with human ratings of the same pairs.

In plain words

Think of every text as a pin on a huge map of meaning. Texts about resetting a password sit near each other, and a text about pizza sits far away. Semantic similarity is the closeness score between two pins. The analogy breaks in one place. The map has hundreds of directions, not two. The closeness the model measures is learned from data, so it can be wrong in ways a real map cannot.

In one precise sentence, semantic similarity compares two embedding vectors and returns one number that is meant to track human judgments of meaning. Reimers and Gurevych (2019) describe the goal as "semantically meaningful sentence embeddings that can be compared using cosine-similarity".

Step 1: An encoder turns each text into a vector

The encoder is a neural network, usually a transformer, that reads a text and outputs one fixed-size vector for it. Input: a text. Output: a list of numbers of the same length for every text, whatever its length.

Not every encoder is good for this. Sentence-BERT (SBERT), where BERT means Bidirectional Encoder Representations from Transformers, adds siamese and triplet network structures so that the vectors can be compared with cosine similarity. A classic baseline is simpler: Cer et al. (2017) describe baselines that average the word embeddings of a sentence and then use cosine. An encoder that encodes each text on its own is called a bi-encoder. You now have two vectors, one per text; the next step compares them.

Step 2: Cosine similarity compares direction, not length

Cosine similarity compares two vectors by the angle between them. The formula is cos(a, b) = (a · b) / (‖a‖ × ‖b‖), where a · b is the dot product and ‖a‖ is the length of a. Dividing by both lengths removes length from the result, so only direction is left. The score runs from −1 (opposite directions) through 0 (unrelated, at right angles) to 1 (the same direction).¹

That is the property a follow-up question probes. A vector and the same vector made twice as long score exactly 1, even though the Euclidean distance between them is not zero and their dot product is larger.¹ A score of 1 therefore means the same direction, not identical text. Sentence encoders such as SBERT are built so that their vectors can be compared with cosine similarity, so direction is the part of the vector that training targets (Reimers and Gurevych, 2019). The similarity search page shows how the score is used to find neighbours.

Step 3: The scores rank candidates or fill a pairwise table

A similarity score is used in one of two ways: to rank candidate texts against one query, or to fill a table of scores for every pair, for clustering or duplicate detection. Input: two vectors, or one query vector and many candidate vectors. Output: one number per pair.

The score is not a calibrated probability. A cosine of 0.96 in the toy example below does not mean "96% the same meaning"; it only means the pair is closer than a pair that scores 0.5 under the same model.¹ Compare scores from one model, never across models, and choose any threshold from labelled examples of your own data. A dot product on vectors scaled to length one gives the same ranking as cosine, which is why many systems normalise once and store the result. For picking a metric inside a retrieval system, see choosing similarity metrics for retrieval-augmented generation.

Step 4: Human ratings and benchmarks check the score

Semantic similarity is judged against people. In the SemEval-2017 task, systems give each sentence pair a continuous similarity score, and performance is measured by the Pearson correlation of machine scores with human judgments. The Pearson correlation is a number from −1 to 1 that is high when two lists of scores rise and fall together. The same paper introduces the STS Benchmark, a shared training and evaluation set drawn from the task data of 2012 to 2017.

For a wider view, the Massive Text Embedding Benchmark (MTEB) spans 8 embedding tasks, 58 datasets and 112 languages, and its authors find that no particular text embedding method dominates across all tasks. So evaluation has two layers: agreement with human ratings on pairs, and performance on the real task, such as retrieval, that uses the scores.

1 · Encode2 · Compare3 · Score4 · CheckReplay

Step-by-step description

1. Each text becomes a vector.
2. Cosine compares direction, not length.
3. One number per pair.
4. Human ratings test the score.

• Each text is encoded into a vector.
• Cosine compares the direction of the two vectors.
• The comparison gives one score.
• Human ratings check the score.

Worked example

Real embeddings have hundreds of dimensions, so this toy uses illustrative two-number vectors you can check by hand. Let the query "reset my password" be a = (3, 4), a close paraphrase "recover my login" be b = (4, 3), an unrelated text "best pizza recipe" be c = (4, −3), an opposite-direction vector be d = (−3, −4), and a text with a's direction but twice the length be e = (6, 8).¹

For a and b the dot product is 3 × 4 + 4 × 3 = 24, each vector has length 5, so the cosine is 24 / 25 = 0.96.¹ For a and c the dot product is 0, so the cosine is 0. For a and d it is −1, and for a and e it is 1, although the straight-line distance between a and e is 5 and a · e is 50 against a · a of 25.¹

Text description of the diagram

• The dot product of a and b is 24.
• Each length is 5, so the cosine is 0.96.

The squared-distance row shows why cosine and distance agree once vectors have length one: the squared distance equals 2 minus twice the cosine, so the closest vector by distance is also the one with the highest cosine.¹

Trade-offs and edge cases

The central trade-off is speed against precision. Scoring every pair with a model that reads both texts together costs n(n − 1) / 2 model runs, which is 49,995,000 for 10,000 texts.¹ Reimers and Gurevych (2019) report that this cuts the effort of finding the most similar pair in 10,000 sentences from 65 hours with BERT to about 5 seconds with SBERT, while keeping the accuracy of BERT on their tests.

The price is that two texts never interact inside a bi-encoder, so fine distinctions can be lost. A cross-encoder reranker re-scores a short list when precision matters, and the two-tower model uses the same split between encoding once and comparing cheaply. Even with each text encoded once, comparing a query with millions of stored vectors by brute force is slow, so large systems use approximate nearest-neighbour search, which checks only part of the collection and may miss a few true neighbours; see vector indexing.

Text description of the diagram

• Cosine ignores the length of the vectors.
• Dot product and Euclidean distance depend on length.
• On vectors of length one they give the same ranking as cosine.

Edge cases follow. First, similar in meaning is not the same as relevant: a question and its answer can have a modest cosine, and two opposite claims about one topic can have a high one, so test on your own labelled pairs. Second, an embedding model that is best on one task need not be best on another, as the MTEB authors report, so evaluate the candidates on your data.

Third, exact terms such as product codes or names may need keyword matching, which is why systems merge both signals in hybrid search. Fourth, raw cosine values from many models sit in a narrow, high band, so unrelated texts can still score well above 0. This crowding of vectors into a narrow cone is called anisotropy, and Ethayarajh (2019) documents it for contextual word representations. It is one more reason to treat absolute scores as relative and to pick thresholds from your own data.

How to answer it in an interview

The spoken structure: say what the score is, say how it is computed, say why it is cheap, then name the proxy trade-off and the reranker.

Common mistakes

• "Semantic similarity is keyword overlap." (wrong) Two texts can share no words and still be close, and share many words and differ in meaning. (right) "It compares meaning vectors; keyword search is a separate signal that I can combine with it."
• "A cosine of 1 means the texts are identical." (wrong) It means the same direction. (right) "A score of 1 says the vectors point the same way; it does not say the words match."
• "Any language model output works as a similarity vector." (wrong) Encoders are tuned for the job; SBERT adds siamese and triplet training for exactly this. (right) "I use an encoder trained so that cosine tracks meaning."
• "The best model on a leaderboard is best for me." (wrong) MTEB reports no method dominating all tasks. (right) "I shortlist models and test them on my own labelled pairs."
• "A score of 0.9 means 90% similar." (wrong) The scale is not a probability. (right) "I pick a threshold from labelled examples."

Follow-up questions

How do you evaluate the quality of embeddings for semantic similarity?

Use two layers. For pair similarity, compare model scores with human ratings on a labelled set such as the STS Benchmark, using Pearson correlation. For the real task, such as retrieval, measure that task's own metric on held-out queries, and check a broad benchmark like MTEB when choosing a model.

How would you combine keyword search with semantic similarity?

Run both searches, then merge the two result lists into one, for example with reciprocal rank fusion, which scores each result by its rank positions in the lists. Keyword search catches exact terms and similarity catches paraphrases; the hybrid search page covers merging keyword and vector results.

Is semantic similarity the same as keyword matching?

No. Keyword matching compares the words themselves, while semantic similarity compares meaning vectors, so it can match a paraphrase that shares no words and can also score two opposite statements as close.

What is the difference between cosine similarity and dot product?

Dot product grows with vector length and cosine does not. When every vector is scaled to length one, the two give the same ranking, which is why systems often normalise once at indexing time.

Does a cosine similarity of 1 mean the texts are identical?

No. It means the two vectors point in the same direction. Different texts can map to nearly the same direction, and the toy vector with twice the length scores 1 against the original.¹

Why not score every pair with a cross-encoder?

The cost grows with the square of the number of texts: 49,995,000 pairs for 10,000 texts.¹ Encode once with a bi-encoder, search, and spend the expensive model only on a short list.

Sources

1. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (arXiv 1908.10084) — arXiv (Reimers and Gurevych, EMNLP 2019), 27 Aug 2019
2. SemEval-2017 Task 1: Semantic Textual Similarity, Multilingual and Cross-lingual Focused Evaluation (arXiv 1708.00055) — arXiv (Cer, Diab, Agirre, Lopez-Gazpio, Specia), 31 Jul 2017
3. MTEB: Massive Text Embedding Benchmark (arXiv 2210.07316) — arXiv (Muennighoff, Tazi, Magne, Reimers, EACL 2023), 19 Mar 2023
4. How Contextual are Contextualized Word Representations? (arXiv 1909.00512) — arXiv (Ethayarajh, EMNLP 2019), 1 Sep 2019