In this article
Before you start
This page sits inside the embeddings and vector search learning path; the full treatment of the idea is in what an embedding is.
Why interviewers ask this
The question separates candidates who use "embedding" and "encoding" as synonyms from candidates who know what each one gives a model. A reply that treats the two words as the same thing is a common weak answer, and interviewers follow it with a question about why one-hot vectors fail. Google's machine learning course teaches the contrast in its embeddings module, which presents embeddings as lower-dimensional representations of sparse data.
A weak answer says "they are basically the same, both turn things into numbers." A strong answer says a fixed encoding is a rule chosen before training, an embedding is a learned vector where distance means similarity, and gives one example of each. A Staff-level answer adds what it costs: the size of the vectors, the loss of meaning in one-hot, and the fact that learned embeddings drift when the model or data changes.
Embedding versus encoding from first principles
- Encoding
- An encoding is a mapping that turns data, such as a word, a category or a character, into numbers or symbols so a computer can store or process it. Some encodings follow a rule fixed in advance; others, called embeddings, are learned from data.
- One-hot encoding
- One-hot encoding is an encoding that represents each item of a vocabulary as a vector as long as the vocabulary, with a single 1 at the item's position and 0 everywhere else, so no two different items share a position.
In plain words
Think of a coat check. Every coat gets a ticket number by a fixed rule: the next number in the pile. The number says nothing about the coat, so a red wool coat and a black raincoat are as far apart as two numbers can be. Now imagine instead that the attendant hangs coats on a long rail, placing similar coats next to each other, and the position on the rail is the label. The rail position carries meaning. The analogy breaks in one place: a real embedding has many positions at once, not one line. A model also learns where to place each item from data.
In one precise sentence, an encoding maps data to numbers. A fixed encoding uses a rule chosen in advance. An embedding is a learned encoding whose numbers form a dense vector, so that similar items get nearby vectors. The vocabulary of the field is explained in what an embedding is.
Step 1: A fixed encoding is a rule from data to numbers
A fixed encoding is an encoding whose rule is chosen before any learning happens and does not look at what the data means. Character codes are one example: the word "cat" becomes the three numbers 99, 97 and 116, and "cap" becomes 99, 97 and 112.1 One-hot encoding is another, and Google's course introduces it as a way to encode categorical data in feature vectors.
The rule never looks at what the word means, so the numbers carry no similarity of meaning. Cosine similarity is the cosine of the angle between two vectors, which is 1 when they point the same way and 0 when they are at right angles; semantic similarity covers how it is used. Treated as vectors, the character codes of "cat" and "cap" have a cosine similarity of about 0.9999, because they share two letters, and those of "cat" and "dog" about 0.9655, though a cat is closer to a dog than to a cap.1 You now know what a fixed encoding is; the next step shows why the best-known one fails.
Step 2: A one-hot encoding puts every pair of words equally far apart
A one-hot vector is a sparse vector, meaning a list that is almost entirely zeros, with exactly one 1 at the position of its item. For a vocabulary of 50,000 words (illustrative), each word becomes a vector of length 50,000 with one 1, so 99.998% of its entries are zero.1
Take any two different words. Their vectors have the 1 in different places, so their dot product is zero and their cosine similarity is exactly 0. Their straight-line distance is always the square root of 2, about 1.4142.1 The one number the lead promised is therefore 0: a hot dog, a shawarma and a salad are all equally unrelated. Google's course states the same problem: one-hot vectors lack meaningful relationships between items, such as hot dogs and shawarmas being more similar than hot dogs and salads.
Input: a word id. Output: a vector with one 1. What you now know: one-hot is a valid encoding that tells a model nothing about similarity.
Step 3: An embedding is a learned dense vector where nearness means similarity
An embedding replaces the long sparse vector with a short dense vector, a list in which almost every entry is a non-zero number, and trains those numbers so similar items land close together. Google's course describes embeddings as lower-dimensional representations of sparse data that fix both the size problem and the missing-similarity problem of one-hot.
Mechanically, an embedding is a lookup table with one row per item. Multiplying a one-hot vector by the table picks one row, so the "crepe" row is [0.85, 0.15, 0.25] in a toy table (illustrative).1 Training adjusts the rows with a neural network, so the numbers come from data, not from a rule. In the toy table, cosine similarity is 0.996 for pancake and crepe, 0.992 for hot dog and shawarma, but 0.507 for hot dog and salad and 0.283 for pancake and salad, while the one-hot version scores 0 for every pair.1
Input: an item id. Output: a short vector. The how-it-is-trained detail is in the two-tower model page.
Step-by-step description
- A fixed encoding maps data to numbers by a set rule.
- Every different pair is equally far apart.
- Similar items get nearby vectors.
- A fixed encoding turns data into numbers by a rule chosen before training.
- A one-hot encoding puts every pair of different words equally far apart.
- A learned embedding places similar items near each other.
| Property | Fixed encoding (for example one-hot) | Embedding |
|---|---|---|
| How the numbers are made | by a fixed rule | learned from data |
| Vector shape | sparse and as long as the vocabulary | dense and short |
| Does distance mean similarity? | no | yes, that is the training goal |
| Needs training data | no | yes |
What does the size difference look like in numbers?
A one-hot vector for a 50,000-word vocabulary, written out in full, takes 200,000 bytes at 32-bit precision, while a dense vector of 768 numbers takes 3,072 bytes, about 65 times less (sizes illustrative).1 That compares the written-out vector. In practice a one-hot vector is kept as its index, one integer of about 4 bytes, so its real cost is the width of the model's input, not storage.1
The width reaches the model: a first layer of 256 hidden units needs 12,800,000 weights for a 50,000-long one-hot input but 196,608 for a 768-long dense input.1 The embedding table itself holds 50,000 × 768 = 38,400,000 parameters, because an embedding layer is a one-hot input times a weight matrix with one row per word. With the table, the total is 38,596,608, so the saving is not in the parameter count.1 It comes in the layers after the lookup and in the similarity the vectors carry. Google's course notes that large one-hot inputs mean a huge number of weights and more data, computation and memory.
Text description of the diagram
- A one-hot vector for a 50,000-word vocabulary, written out in full, takes 200,000 bytes at 32-bit precision.
- A dense vector of 768 numbers takes 3,072 bytes.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
How does an embedding differ from an encoding?
An encoding is any mapping from data to numbers. Fixed encodings, such as one-hot, character codes or label ids, follow a rule chosen before training. An embedding is a learned encoding: a dense, low-dimensional vector that a model trains so similar items land close together. So a fixed encoding only has to be consistent, while an embedding has to carry meaning.
One-hot puts every pair of words the same distance apart and grows with the vocabulary, so a model gets no similarity from it. An embedding table gives each word a short trained vector, so cat and kitten end up near each other. The trade-off is that embeddings depend on training data, so they drift and must be recomputed when the model changes.
What makes it Staff-level: it names learned versus fixed as the real axis, says what breaks in the alternative (no similarity, growing size), and ends on the maintenance cost.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Definition | "They are the same: text to numbers." | A fixed encoding is a rule; an embedding is a learned dense vector. | Adds that every embedding is a learned encoding, not the reverse. |
| Depth | Names neither example. | Gives one-hot as the encoding example. | Shows one-hot gives zero similarity for every different pair. |
| Trade-offs | No cost mentioned. | Mentions vector length and memory. | Adds drift, retraining and re-indexing when the model changes. |
| Communication | Jargon without definitions. | Defines each term in one sentence. | Gives a concrete pair, such as hot dog and shawarma. |
| Follow-up readiness | Stalls on "how is it trained?" | Explains a lookup table trained with a network. | Mentions contextual embeddings and non-neural methods. |
Three variants of this question
What is the difference between an embedding and one-hot encoding?
One-hot encoding is the fixed, sparse case; an embedding is the learned, dense one. One-hot vectors grow with the vocabulary and put all words the same distance apart, while embeddings stay short and place similar words close.
Is the output of an encoder model an embedding?
Not automatically. "Encoder" also names a model component, as in the transformer; its output vectors are embeddings in the loose sense, but a vector is only a good embedding of a whole sentence if the model was trained for that. Reimers and Gurevych (2019) built sentence vectors with siamese networks and report cutting a search for the most similar pair among 10,000 sentences from about 65 hours to about 5 seconds, because each sentence is encoded once and the vectors are compared.
What happens to each when the vocabulary doubles?
A one-hot vector doubles in length, so the model's input and its first-layer weights double. An embedding vector keeps its length and every layer after the lookup stays the same. The table gains rows, so its parameters grow with the vocabulary, but the lookup cost per item does not.
Common mistakes
- "Embedding and encoding mean the same thing." (wrong) They overlap in everyday speech but differ in meaning. (right) "An embedding is a learned encoding; a fixed encoding is a rule chosen before training."
- "One-hot vectors capture meaning because each word is unique." (wrong) Unique is not similar. (right) "One-hot makes all different words equally far apart, so there is no similarity to use."
- "Embeddings always come from a neural network." (wrong) Matrix-factorisation methods also produce dense vectors. (right) "Most are learned by a network, but any method that yields dense, similarity-preserving vectors counts."
- "I can reuse old vectors after changing the model." (wrong) Vectors from two models live in different spaces. (right) "I re-embed the data and rebuild the index when the model changes."
Follow-up questions
How is an embedding trained?
A network adjusts the rows of the lookup table so that a training objective improves, for example pulling matched query and item vectors together. The two-tower model page walks through one such loss.
Can an embedding be made without a neural network?
Yes. Matrix-factorisation methods such as truncated singular value decomposition, used in latent semantic analysis, compress sparse word-count data into dense vectors; they are less common for text today but still count as embeddings.
How do you compare two embeddings?
Use cosine similarity or a dot product, then search many vectors with an index; see similarity search. For the geometry behind it, read what a vector space is.
What is a contextual embedding?
A contextual embedding gives the same word different vectors depending on the sentence around it, unlike a static table with one row per word. Transformer models produce contextual embeddings; the transformer overview shows how.
If embeddings capture meaning, why keep keyword search?
Embeddings can blur exact terms such as names and codes, which keyword matching catches. Hybrid search merges both, and a cross-encoder reranker can then reorder the short list.
- An encoding maps data to numbers; a fixed encoding uses a rule chosen before training, and an embedding is a learned dense vector.
- Every embedding is a learned encoding, but not every encoding is an embedding.
- One-hot vectors give cosine similarity 0 for every different pair, so they carry no meaning.
- Embeddings are shorter, and similar items sit close together.
- Learned vectors drift, so re-embed and re-index when the model changes.
Sources
- Machine Learning Crash Course: Embeddings — Google for Developers, 26 Oct 2026
- Machine Learning Crash Course: Categorical data, vocabulary and one-hot encoding — Google for Developers, 26 Oct 2026
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (arXiv 1908.10084) — arXiv (Reimers and Gurevych), 27 Aug 2019
