In this article
Before you start
This page is one step of the embeddings and vector search learning path. In a RAG system, the metric decides which stored chunks are fed to the language model.
Why interviewers ask this
A wrong metric quietly returns the wrong chunks, and the language model then writes a fluent answer from them. That is why this question separates people who have only called a vector database from people who understand what it computes.
The question tests three things: whether you can name the common metrics, whether you know they are related, and whether you know the rule for choosing. A weak answer says "cosine, because everyone uses it." A strong answer says to match the metric to the one the embedding model was trained with, and explains what the metric measures. A Staff-level answer adds when the three metrics give identical rankings, when they do not, and what to check before trusting a score. The question is reported by candidates in AI and machine learning interviews.
Similarity metrics in RAG from first principles
- Similarity metric
- A similarity metric is a function that turns two embedding vectors into one number saying how close their meanings are, so a retrieval system can sort stored text chunks from best match to worst for a query. Common choices are cosine similarity, the dot product and Euclidean distance.
- Vector normalization
- Vector normalization is the step that divides a vector by its own length so the result has length one and keeps only its direction. After it, every stored vector lies on the same sphere, and length can no longer favor one chunk over another.
In plain words
Picture each chunk of text as an arrow starting at the same point. Each arrow points somewhere and has a length. A metric answers one question about two arrows: how alike are they?
Cosine asks only whether they point the same way. Euclidean distance asks how far apart the tips are. The dot product mixes both: it grows when the arrows align and also when they are long. The analogy breaks in one place: real embeddings have hundreds of dimensions, so you cannot draw them.
In one precise sentence, a RAG system embeds the query, scores every candidate chunk with one metric, and keeps the best few.
Step 1: Three metrics score a pair of vectors
The three common metrics are cosine similarity, the dot product and Euclidean (L2) distance, and Pinecone's guide describes all three. Cosine similarity is a measure of the angle between two vectors: it is the dot product divided by the product of the two lengths, and it runs from −1 (opposite) through 0 (unrelated) to 1 (same direction). The wider idea of ranking stored items by closeness is covered in similarity search.
The dot product is positive when the angle is under 90 degrees, zero at exactly 90 and negative above it, and it also depends on length. Euclidean distance depends on both direction and length too.
The direction of "better" differs: larger is better for cosine and the dot product, smaller is better for Euclidean distance. Input: two vectors. Output: one number. You now have the three scores; the next step shows when they agree.
Other metrics exist, such as Hamming distance, which counts differing bits and suits binary-quantized vectors. For how similarity is measured more broadly, see semantic similarity.
Step 2: Normalized vectors make the three metrics rank the same
Normalized vectors make cosine, dot product and Euclidean distance produce the same ranking, which is the answer to the follow-up. Cosine is the dot product divided by the two lengths, so on length-one vectors the two scores are identical.
Euclidean distance joins them by arithmetic. For two vectors of length one, the squared distance equals 2 minus 2 times the cosine, so the closest vector by distance is the one with the highest cosine.1 In the toy run below, the squared distance to the chunk pointing the opposite way is 4.0, and 2 − 2 × (−1) is also 4.0.1
Because the orderings match, an engineer may pick whichever metric the index computes fastest without changing which chunks come back. This holds only after normalizing. Input: unit vectors. Output: one shared ranking. The next step shows what happens without it.
Step 3: Without normalization the metrics can disagree
Raw vectors of different lengths can make the three metrics rank the same chunks differently. The numbers are illustrative toy vectors: the query is (1, 2), and the chunks are b = (2, 4), c = (3, 1), d = (0.5, 1.2) and e = (−1, −2).1
Chunk b points exactly along the query but is twice as long. Its cosine is 1.0 and its dot product is 10.0, yet its Euclidean distance is 2.2361, the same as chunk c's. Chunk d is short and nearly parallel: its cosine is 0.9976 and its distance is only 0.9434, so Euclidean distance ranks d first, cosine ranks b first, and the dot product puts c ahead of d because c is longer.1
| Chunk | Cosine | Dot product | Euclidean distance |
|---|---|---|---|
| b = (2, 4) | 1.0 | 10.0 | 2.2361 |
| c = (3, 1) | 0.7071 | 5.0 | 2.2361 |
| d = (0.5, 1.2) | 0.9976 | 2.9 | 0.9434 |
| e = (−1, −2) | −1.0 | −5.0 | 4.4721 |
On normalized vectors all three metrics give the order b, d, c, e, as the figure below shows.
Euclidean distance is sensitive to scale, and so is the dot product. Cosine ignores length, so it is not suitable when length carries meaning.
Step 4: Choose the metric the embedding model was trained with
The rule is to match the index metric to the one used to train the embedding model, which Pinecone calls the basic rule of thumb. Two examples: the all-MiniLM-L6-v2 model card describes a model scored with cosine similarity (as in Reimers and Gurevych, 2019, read 2026-10-26), while the usage example on the msmarco-bert-base-dot-v5 model card scores with the dot product.
So read the model card, then configure the index the same way, for both indexing and querying. If the model card names no metric, evaluate cosine and dot product on a small labelled query set and keep the better one.
In a RAG system the metric only orders candidates. An approximate index returns the top k chunks quickly, and hybrid search or a cross-encoder reranker can fix the order afterward. The vectors themselves come from encoders such as those in the transformer architecture.
Step-by-step description
- Cosine, dot product and Euclidean distance.
- All three give the same ranking.
- The three metrics can disagree.
- Match the training metric.
- On normalized vectors the three metrics give the same ranking.
- On raw vectors the three metrics can disagree.
- The rule is to match the metric the embedding model was trained with.
Text description of the diagram
- On raw toy vectors the three metrics give different orders.
- On normalized vectors all three give the same order.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
What similarity metric would you choose in a RAG system?
I match the metric to how the embedding model was trained, which is usually cosine similarity. I check whether the vectors are normalized to length one. If they are, cosine, dot product and Euclidean distance give the same ranking, so I use whichever my index supports. If the model was trained with the dot product and vector length carries meaning, I keep the dot product and do not normalize. I use the same metric for indexing and querying. Then I test retrieval on a small set of labeled queries, because the metric only orders candidates, and a reranker can fix the top few.
What makes it Staff-level: it states the rule, the condition under which the choice does not matter, the case where it does, and how to verify.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | Says cosine, with no reason | Names cosine, dot product and Euclidean distance correctly | Also states the direction of "better" for each |
| Depth | Only lists metrics | Match the model's training metric | Explains the equivalence on normalized vectors |
| Trade-offs | None | Cosine ignores length; dot product uses it | Names the case where length carries meaning |
| Communication | Jargon without definitions | Defines each metric in one sentence | Gives a tiny numeric example |
| Follow-up readiness | Stuck on "why equivalent?" | Knows cosine equals dot on unit vectors | Shows squared distance equals 2 minus 2 cosine |
Three variants of this question
Cosine similarity or dot product for embeddings in a vector database?
Use the one the model was trained with. If the vectors are normalized, they rank identically, so the dot product is a fine stand-in for cosine.
Why not just use Euclidean distance?
Euclidean distance is sensitive to scale, so on raw vectors it can rank differently from cosine. On normalized vectors it ranks like cosine, so it works there too.
What if the embedding model returns unnormalized vectors?
Either use cosine, which divides out the length, or normalize first and use the dot product. Keep the dot product on raw vectors only if the model was trained that way.
Common mistakes
- "Cosine is always the best metric." (wrong) It ignores length, which can matter. (right) "I use the metric the model was trained with; cosine is the common case."
- "Higher dot product always means more similar." (wrong) It also grows with vector length. (right) "On raw vectors, a long chunk can win the dot product without pointing the same way."
- "I can index with one metric and query with another." (wrong) The ranks can change. (right) "Indexing and querying use the same metric."
- "Smaller score is better for every metric." (wrong) That holds only for distance. (right) "Smaller is better for Euclidean distance, larger for cosine and the dot product."
Follow-up questions
Do cosine similarity and Euclidean distance give the same ranking?
Only on normalized vectors, where the squared distance equals 2 minus 2 times the cosine. On raw vectors the toy run above shows they can put different chunks first.
Is a higher dot product always more similar?
No. The dot product reflects direction and length, so a long vector can outscore a better-aligned short one.
Does the metric matter more than the embedding model?
No: the embedding model matters far more, because it decides what the vectors mean, and the metric follows from how the model was trained. Match the training metric, then compare retrieval quality on labeled queries.
Where do hybrid search and a reranker fit after the metric?
Hybrid search and a reranker come after the metric. Hybrid search merges keyword and vector results, and a reranker re-sorts the short list. The two-tower model shows how the vectors are trained.
- Match the similarity metric to the one the embedding model was trained with.
- Cosine uses direction only; the dot product and Euclidean distance also use length.
- On normalized vectors, cosine, dot product and Euclidean distance rank the same.
- On raw vectors the three can disagree, so check normalization.
- The metric orders candidates; hybrid search and a reranker refine the top few.
Sources
- Vector Similarity Explained — Pinecone (Roie Schwaber-Cohen); accessed 2026-10-26, 30 Jun 2023
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (arXiv 1908.10084) — arXiv (Reimers and Gurevych), 27 Aug 2019
- sentence-transformers/all-MiniLM-L6-v2 model card — Hugging Face (sentence-transformers); accessed 2026-10-26, 26 Oct 2026
- sentence-transformers/msmarco-bert-base-dot-v5 model card — Hugging Face (sentence-transformers); accessed 2026-10-26, 26 Oct 2026
