What is CBOW (continuous bag of words) and how does it work?

In this article

1. Before you start
2. Why interviewers ask this
3. CBOW from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

CBOW is where word embeddings began for many practitioners, and the question tests whether you understand what an embedding is trained to do, not only how to call a library. Interviewers use it in three forms: explain CBOW, compare it with skip-gram, or explain why a transformer's token embeddings differ from word vectors. A weak answer says "it predicts a word from its neighbours".

A strong answer states what is averaged, what the output layer computes, what the training cost depends on, and what is lost. The answer to practise has three parts: the prediction task, the averaging step, and the limit that one vector per word cannot capture word order or several meanings. This page builds each part from a toy corpus you can follow by hand.

CBOW from first principles

CBOW

CBOW, the continuous bag-of-words model, is a log-linear model (one with no hidden non-linear layer between the averaged input and the output scores) that learns word vectors by averaging the vectors of the words around a position and using that average to predict the word at that position.

Word vector

A word vector is a list of numbers, typically a few hundred, learned during training so that words used in similar contexts get similar lists.

In plain words

Imagine a sentence with one word covered by your thumb. You read the words on both sides and guess the hidden one. CBOW does this millions of times, and each time it nudges the number lists of the neighbours so that the guess improves. After enough rounds, words that fit the same gaps end up with similar lists. The analogy breaks in one place. A person remembers the order of the neighbours. CBOW mixes them into a single average first, so "dog bites man" and "man bites dog" give the same input. The paper calls it a bag-of-words model for that reason: "the order of words in the history does not influence the projection" (Mikolov et al., 2013, section 3.1).

Step 1: Slide a window to make training examples

Each position in the text becomes one training example: the word there is the target, and the words within the window are the context. Take the sentence "the quick brown fox jumps over the lazy dog". It has 8 distinct words and 9 positions, so 9 examples with a window of 2.¹ The first is context "quick, brown" with target "the". The third is context "the, quick, fox, jumps" with target "brown", and this is the example we follow.

The original paper reports its best results on its test using four history and four future words as context, so the window looks forward as well as back (Mikolov et al., 2013). The gensim library defaults to a maximum distance of 5 (gensim documentation, read October 2026). The output of this step is a list of pairs; no learning has happened yet.

1 · Slide a window2 · Average the context3 · Score the vocabulary4 · Update the vectorsReplay

Step-by-step description

1. Pick a target word and its neighbours.
2. Four vectors become one.
3. Softmax gives each word a probability.
4. Raise the true word's probability.

• A window picks one target word and its neighbours as the context.
• The context vectors are averaged into one vector.
• Every vocabulary word gets a score and softmax turns the scores into probabilities.
• Gradient descent then updates the vectors and raises the target's probability.

Step 2: Average the context vectors

Each word owns one input vector (a word vector, defined above), here three numbers set at random (illustrative, fixed seed). Look up the vectors of the four context words and take their average, number by number. For "the, quick, fox, jumps" the average is (-0.221, 0.270, -0.010).¹ The average is the model's summary of the neighbourhood, and its length is the vector size, not the number of context words.

Averaging is why CBOW is fast and why it is limited. The cost of reading the context stays small because every position uses the same averaging step, whatever order the words come in. The price is that two contexts with the same words in a different order are indistinguishable. Gensim exposes the choice: by default cbow_mean is 1 and it uses the mean of the context vectors, while 0 uses their sum.

So the "bag" in the name is a statement about what is lost: order, not membership. That is the follow-up trap closed: the context of "bites" in "dog bites man" is dog and man, and in "man bites dog" it is man and dog. Both average to the same vector, so CBOW cannot tell who bit whom. Skip-gram, the paper's other model, also ignores word order within the window, so this is a property of both, not the reason for the test scores in the trade-offs section.

Step 3: Score every word and apply softmax

Multiply the average by an output weight matrix (a second table of numbers, one row per vocabulary word) to get one score per vocabulary word, then apply softmax to turn the scores into probabilities that sum to one. In the toy model the 8 scores range from -0.087 to 0.209, and the probability of the true word "brown" comes out at 0.109, slightly below the uniform guess of 0.125 for 8 words.¹ The loss is the negative natural logarithm of that probability, 2.219.

This is the expensive step in a real model: the output layer has one score per vocabulary word. The paper gives the training cost per word as Q = N × D + D × log2(V), where N is the number of context words, D the vector size and V the vocabulary size, because it replaces the full softmax with a hierarchical one (Mikolov et al., 2013, equation 4). A later paper offers negative sampling as "a simple alternative to the hierarchical softmax" (Mikolov et al., 2013b).

Two terms need unpacking. Hierarchical softmax is a way to score a word without scoring them all: it arranges the vocabulary as the leaves of a Huffman tree (a binary tree that gives frequent words short paths) and scores a word by the chain of left-or-right decisions on its path, so about log2 V decisions replace V scores (Mikolov et al., 2013, section 2.1). Negative sampling is a second shortcut: it skips most of the vocabulary: each update scores the true word against only k randomly drawn noise words and trains the model to tell them apart, and gensim's negative parameter defaults to 5 (Mikolov et al., 2013b; gensim documentation, read October 2026).

Step 4: Update the vectors and repeat

Gradient descent now changes the numbers, raising the probability of the target word. The error signal at the output is the probability vector minus a one-hot vector (a list that is 1 at the true word's position and 0 everywhere else). It adjusts the output weights, and a share of it flows back to each of the four context vectors, so those words move toward being useful for predicting "brown".

The update raises the target's probability. The learning rate is the size of each adjustment, and real runs use much smaller values than the 1.0 used here. With a learning rate of 1.0, one step raised the probability of "brown" from 0.109 to 0.132 and cut the loss from 2.219 to 2.024.¹

Then the window slides and the process repeats over the whole corpus, for several passes. Gensim's default is 5 passes over the data. The vectors that remain at the end are the product: one row per word, and words that appear in similar contexts have similar rows. Nothing in the loop looks up a meaning; similarity emerges from the prediction task alone.

Worked example

The toy corpus has 8 words and a vector size of 3. The table lists what the script computed for the example "the, quick, fox, jumps" predicting "brown".¹

Text description of the diagram

• Before the update the target brown has probability 0.109.
• After one update the probability of brown is 0.132.
• The loss falls from 2.219 to 2.024.

The loss near zero after 200 more steps¹ shows memorisation of this one example. A real corpus has millions of examples that disagree, so the model cannot memorise them and must learn what generalises. The central lines of the script are the average and the update:

h = [sum(W_in[i][d] for i in ids) / len(ids) for d in range(D)]   # average context vector
dz = [p[j] - (1.0 if j == target else 0.0) for j in range(V)]      # probabilities minus one-hot

Scale matters. With 300 numbers per vector, 8 context words and a vocabulary of 1,000,000 words, equation 4 gives about 8,379 operations per training word, against about 300,002,400 for a full softmax, a ratio of about 35,800.¹ That gap is why the paper uses the hierarchical form.

Trade-offs and edge cases

CBOW against skip-gram. Skip-gram is the paper's second model, and it reverses the task: it predicts the surrounding words from the current word. The analogy test is the paper's scoring method, a set of 8,869 semantic and 10,675 syntactic questions (Mikolov et al., 2013, section 4.1). Each question asks for the word whose vector is closest to vector(X) - vector(Y) + vector(Z); a semantic example is that Athens is to Greece as Oslo is to Norway, and a syntactic one is that big is to biggest as small is to smallest.

Only an exact match counts. In the paper's comparison with 640-dimensional vectors on the same data, CBOW scored 24% semantic and 64% syntactic, and skip-gram 55% and 59%.² Order loss is not what separates them, since skip-gram also ignores order. Neither dominates; skip-gram led on the semantic questions and CBOW on the syntactic ones in that test.

Text description of the diagram

• CBOW predicts the current word from its averaged context.
• Skip-gram predicts the surrounding words from the current word.
• Semantic accuracy was 24% for CBOW and 55% for skip-gram.
• Syntactic accuracy was 64% for CBOW and 59% for skip-gram.

One vector per word. The model has a single row per word, so "bank" gets one vector for the river and the money. Transformers instead compute a new representation of each token from its context, which is why a transformer's embedding table is only the first layer (see how transformers work). The vocabulary is a choice. Words below a count threshold are dropped (gensim's min_count defaults to 5); subword methods such as byte pair encoding avoid dropping rare words by splitting them. Frequent words. The second Mikolov paper reports that randomly discarding frequent words speeds up training and gives more regular representations.

How to answer it in an interview

The spoken structure is: the task, the averaging, the cost trick, then the limit.

Common mistakes

"CBOW predicts the next word." (wrong) It predicts a word at a position from words on both sides, including future words. (right) "CBOW predicts the centre word from its surrounding context."

"CBOW concatenates the context vectors." (wrong) That would preserve order and make the input depend on the window size. (right) "It averages them, so order is discarded."

"A word vector captures all meanings of a word." (wrong) There is one row per word, so senses are blended. (right) "One vector per word means senses blend; contextual models fix that."

"Training needs a softmax over the whole vocabulary every step." (wrong) That is what hierarchical softmax and negative sampling avoid. (right) "The cost per word falls from about D times V to about D times log V."

Follow-up questions

What is the difference between CBOW and skip-gram?

CBOW predicts the centre word from the average of its context; skip-gram predicts context words from the centre word. In the paper's test with 640-dimensional vectors, CBOW scored 24% semantic and 64% syntactic, skip-gram 55% and 59%.²

Why does CBOW average the context vectors?

Averaging gives a fixed-size input however many neighbours there are and makes the input cheap to compute at every position. The cost is that order is lost, so the paper describes it as a bag-of-words model.

Does CBOW keep word order?

No. The projection averages the context, so permuting the neighbours gives the same input; the paper states that the order of words in the history does not influence the projection.

How is CBOW trained efficiently on a large vocabulary?

By replacing the full softmax. The paper's cost per word is N × D + D × log2(V) with hierarchical softmax, and a later paper adds negative sampling as an alternative.¹ Gensim sets 5 negative samples by default.

Is CBOW still used today?

It remains a baseline and a teaching model, but modern language models learn context-dependent representations instead. For what replaced it, read the transformer hub, the deep learning hub, the loss behind the update in cross-entropy loss, and how a neural layer transforms vectors in activation functions.

Sources

1. Efficient Estimation of Word Representations in Vector Space (arXiv 1301.3781 v3) — arXiv (Mikolov, Chen, Corrado, Dean), 7 Sep 2013
2. Distributed Representations of Words and Phrases and their Compositionality (arXiv 1310.4546) — arXiv (Mikolov, Sutskever, Chen, Corrado, Dean), 16 Oct 2013
3. models.word2vec: Word2Vec embeddings (parameter reference) — Gensim documentation (date = read date), 24 Oct 2026