What are embeddings? Embeddings explained from scratch

In this article

1. Before you start
2. Why interviewers ask this
3. Embeddings from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

Embeddings sit under search, recommendations, retrieval-augmented generation (RAG) and clustering, so a candidate who cannot explain them cannot design any of those systems. The question arrives as "what is an embedding", "how are embeddings trained", "how do you compare two of them" or "how big is the index". A weak answer says "a vector that represents a word". A strong answer says who produces the vector, what training rewards, how closeness is measured and what the vector length costs. The structure to practise is: what it is, how it is made, how it is compared, what it costs. This page builds each part from zero, then runs real vectors through a small model so every number can be checked.

Embeddings from first principles

Embedding model

An embedding model is a trained neural network that maps one input, such as a sentence, to a fixed-length vector so that inputs with similar meaning receive vectors that lie close together.

Contrastive training

Contrastive training is a way of teaching a model by showing it pairs that belong together and pairs that do not, and rewarding it when the first kind end up close and the second kind end up far apart.

In plain words

Picture a city map where every shop is a pin. Similar shops sit in the same neighbourhood. To find a place like this one, look at nearby pins. An embedding is the pin's coordinates. A real embedding has hundreds of coordinates, not two. Its neighbourhoods are meanings, not streets. The analogy breaks in one place. Surveyors draw a city map. The embedding map is learned from examples, and its axes have no names a person could read.

In one precise sentence: an embedding is the output vector of a trained model, trained so that distance between vectors tracks similarity of meaning. The embeddings and vector search hub defines the term and shows how vectors are stored and searched. This page explains how one is made.

Step 1: Turn the input into tokens the model can read

A model cannot read letters or pixels directly; it reads numbers. For text, the first job is to cut the input into tokens, small chunks of text, and replace each chunk with an integer id. The sentence "How do I reset my password?" becomes a short list of ids. Byte pair encoding is the usual way those chunks are chosen.

The input of this step is raw text. The output is a list of integers. Nothing about meaning exists yet: the id for "password" is just a label, and the id 5 is not "closer" to the id 6 in any useful sense. Meaning gets added in the next two steps, and the first thing the model does with the ids is look each one up in a table of learned vectors.

Step 2: Let the model turn the tokens into one vector

The embedding model reads all the token vectors together and outputs one vector for the whole input. Most embedding models are transformers: attention layers let every token look at every other token, and the model then combines the results into a single fixed-length vector. Fixed length is the point. A three-word query and a three-paragraph passage both come out as, for example, 384 numbers for the small open model used later on this page, as its model card states.¹

The output is the embedding. A network with random weights would output noise; the structure comes from training, which is why the vector alone, without the model that made it, is not interpretable, and two different models produce vectors that cannot be compared.

Step 3: Training decides what each number means

Training makes the vectors useful, and it usually happens in two stages. First a transformer is pretrained on large amounts of text to predict missing words, which teaches it language in general. Then comes fine-tuning with contrastive training: the model sees pairs that should match, such as a question and its answer, or two paraphrases, and pairs that should not. Its loss, the number it tries to lower, is small when matching pairs are close and non-matching pairs are far apart.

Nobody picks what coordinate 17 means; it is whatever helped lower the loss. The all-MiniLM-L6-v2 model card says it starts from a pretrained MiniLM model, followed by fine-tuning with a contrastive objective on more than one billion sentence pairs. Sentence-BERT (Reimers and Gurevych, 2019) showed that fine-tuning a pretrained BERT as a siamese network on labelled sentence pairs gives sentence vectors that cosine similarity can compare. Earlier, Mikolov et al. (2013) trained word vectors by predicting a word from its neighbours, or the neighbours from the word, in under a day on 1.6 billion words.

1 · Tokens2 · Vector3 · Training4 · Cosine5 · SearchReplay

Step-by-step description

1. The input is cut into tokens with integer ids.
2. The model outputs one fixed-length vector.
3. Training, done once before use, pulls related items close.
4. Closeness is the angle between two vectors.
5. An index returns the nearest stored vectors.

• Text is cut into tokens before the model reads it.
• The model outputs one fixed-length vector for the whole input.
• Training happens once, before use, and fixes what each number means.
• Cosine similarity compares the angle between two vectors.

Step 4: Measure closeness with the angle

Once every item is a vector, "similar" becomes arithmetic. The standard measure is cosine similarity: multiply the two vectors entry by entry, add the products, and divide by the product of their lengths. The result depends only on direction. OpenAI's embeddings guide recommends cosine similarity and says its embeddings are normalised to length 1, which means a dot product alone gives the cosine and the ranking by Euclidean distance is identical.

The check in our script confirms the identity for unit vectors: the squared Euclidean distance between two of them equals two minus twice their cosine, so both measures rank neighbours the same way.¹ If your model's vectors are not unit length, use the score the model was trained with: most sentence models are trained for cosine, but some retrieval models are trained for the raw dot product, where the vector's length carries signal, and normalising them changes the ranking (Sentence-Transformers pretrained models, read October 2026).

Step 5: Store the vectors and search them

To answer "find items like this one", embed the query with the same model and return the stored vectors with the highest cosine. A vector database holds the vectors with an approximate nearest-neighbour index so that the search does not compare the query with every row. Sentence-BERT's paper explains why this split matters: comparing every pair with a full BERT pass would need about 50 million inference passes for 10,000 sentences, roughly 65 hours, while precomputed embeddings reduce the search to about 5 seconds.

The output of this step is a ranked list. Two refinements are common and covered elsewhere: hybrid search combines embeddings with keyword matching, and a cross-encoder reranker re-scores the top results more carefully.

Worked example

The script embeds six short sentences with all-MiniLM-L6-v2 and compares every pair. The sentences are ours: two about resetting a password, two about France, two about a cat.¹

Text description of the diagram

• The password paraphrase pair scores 0.794.
• The cat and kitten pair scores 0.509 with almost no shared words.
• Unrelated pairs score close to zero.

The cat pair is the instructive one: "cat" and "kitten", "warm windowsill" and "sunny ledge" share no words except "on" and "the", yet the pair scores 0.509, because the model learned that these phrases occur in similar contexts. A keyword search would score that pair at nearly zero. Each vector has 384 numbers, and the first five numbers of the password question are 0.007, -0.063, -0.072, -0.032 and -0.043; they mean nothing on their own.¹

The same script prices the storage. One float32 number takes 4 bytes, so a 384-number vector takes 1,536 bytes and one million of them take 1.43 GiB.¹ At the 1,536 default length of OpenAI's text-embedding-3-small, one million vectors take 5.72 GiB, and at the 3,072 default of text-embedding-3-large they take 11.44 GiB, per the lengths in OpenAI's guide.¹

Text description of the diagram

• One million 384-number float32 vectors take 1.43 GiB.
• At 1,536 numbers they take 5.72 GiB and at 3,072 numbers 11.44 GiB.

Trade-offs and edge cases

Longer vectors cost memory and time, and help less than you expect. Storage and comparison cost grow with the vector length, as the figure above shows. OpenAI's guide says its newer models accept a dimensions setting that shortens the vector while keeping its concept-representing properties, which is the vendor's own trade-off lever; whether the quality loss matters is something you measure on your own data.

Vectors from different models are not comparable. Each model builds its own space. Switching model, or even model version, means re-embedding the whole collection, and that cost is one reason teams pin a version.

An embedding compresses. One vector cannot hold every detail of a long passage. This model truncates input longer than 256 word pieces, per its model card, so text after that point does not affect the vector at all.¹ Chunking long documents before embedding is the usual fix.

Similar is not the same as correct. The nearest vector to a question may be on the topic and still not contain the answer. That gap is why retrieval systems add reranking and keyword search.

The distance measure follows the model. OpenAI's guide says the choice typically does not matter much for its unit-length vectors, because cosine, dot product and Euclidean distance then rank identically. Sentence-Transformers publishes models tuned for the dot product, so for vectors that are not unit length, check the score the model card names.

How to answer it in an interview

The spoken structure is: what it is, how it is trained, how it is compared, then what it costs and where it fails.

Common mistakes

"The numbers are features a person designed." (wrong) Nobody assigns a meaning to coordinate 17; training does, and the axes have no readable names. (right) "The model learns the layout so that distance tracks similarity."

"You can compare vectors from two different models." (wrong) Each model has its own space, so a cosine between spaces is meaningless. (right) "Both sides must come from the same model and version."

"Higher dimensions are always better." (wrong) Storage and search cost grow with length, and quality gains flatten. (right) "I choose the shortest length that holds retrieval quality on my own test set."

"Similarity of 0.8 means 80 percent similar." (wrong) The score is an angle measure whose scale depends on the model; in our run 0.794 was a paraphrase and 0.669 was two related facts. (right) "I compare scores within one model and set a threshold from examples."

Follow-up questions

What is an embedding model?

An embedding model is the neural network that maps an input to a vector. Open examples such as all-MiniLM-L6-v2 output 384 numbers, and hosted models output more. It is trained for similarity, not for writing text, so it cannot answer questions itself.

What is embedding in generative AI?

In generative AI an embedding is the vector used to represent text so it can be searched or compared, most often to fetch relevant passages for a language model in retrieval-augmented generation. The generator then answers from those passages. See the AI engineer interview questions for the neighbouring topics.

What is an embedding in ML?

In machine learning an embedding is a learned vector for a discrete or complex item, such as a word, a user, a product or an image, trained so that useful relations show up as distance. Recommendation systems embed users and items in the same space.

Is an embedding the same as the input layer of an LLM?

An embedding model's output and a language model's input table are related but not the same. A language model starts with a table of learned vectors, one per token, which is an embedding table; an embedding model goes further and outputs one vector for the whole input. The table is part of the pipeline of tokenization and the transformer, while the final vector is what retrieval uses.

Can you make an embedding shorter?

Yes. Some models accept a shorter output length, as OpenAI's dimensions option does, and you can also compress vectors with lower precision. Measure retrieval quality before and after, because the loss depends on your data.

Sources

1. Efficient Estimation of Word Representations in Vector Space (arXiv 1301.3781 v3) — arXiv (Mikolov, Chen, Corrado, Dean), 7 Sep 2013
2. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (arXiv 1908.10084) — arXiv (Reimers, Gurevych), 27 Aug 2019
3. sentence-transformers/all-MiniLM-L6-v2 model card — Hugging Face (date = read date), 24 Oct 2026
4. Sentence-Transformers pretrained models — sbert.net (date = read date), 24 Oct 2026
5. Vector embeddings guide — OpenAI (date = read date), 24 Oct 2026