In this article
Before you start
This builds on how transformers read text.
Why interviewers ask this
The question sounds like trivia, but a wrong answer shows up in real systems: costs, context limits and odd failures on rare words all trace back to how text is cut into tokens. Interviewers use it to see whether a candidate treats a token as a design choice rather than a synonym for "word".
A weak answer gives one number ("about four characters") and stops. A strong answer says why there is no single ideal length and names the two ends: characters make long sequences, whole words make a huge vocabulary. A Staff-level answer adds what each end costs the model, says that the vocabulary size is the knob that sets average length, and notes that the four-character figure is an English estimate that changes with language and tokenizer. The same question appears among the large language model interview questions reported by candidates.
Token size from first principles
- Token
- A token is a unit of text that a language model reads and writes, such as a character, a part of a word, a whole word or a punctuation mark, so a token is not a fixed number of characters and the token count of a text is not its word count.
- Subword token
- A subword token is a token that is a frequent piece of a word, so common words stay whole while rare words split into known pieces and unknown tokens become rare.
In plain words
Think of building sentences from a box of labelled tiles. If every tile is one letter, the box is tiny. But every sentence needs a long row of tiles, and each tile says almost nothing. If every tile is a whole word, a sentence is short. But you need a tile for every word that exists, and a new word has no tile. Tiles for common syllables and word pieces sit in the middle.
The analogy breaks in one place. A model learns which tiles to have from its training text, so the box is built from data, not by hand.
In one precise sentence: the vocabulary size you choose sets token length. A larger vocabulary means longer tokens and shorter sequences. A smaller one means shorter tokens and longer sequences.
Step 1: Character tokens cost attention pairs
At the character level every letter, digit and space is a token. The Hugging Face tokenizer summary says this gives a small vocabulary and no <unk> token (the placeholder for an unknown word), but sequences become much longer, and a single character carries far less meaning than a word.
Input: text. Output: one token per character. Take an illustrative 144-character sentence: it becomes 144 tokens.1 Attention compares every token with every other token, so the number of pairs grows with the square of the sequence length; the quadratic attention page explains why and how to cut it. Character tokens give 20,736 pairs, and about 36 subword tokens (Step 3) give 1,296, sixteen times fewer.1 Sequence length also fills the context window, the limit on how many tokens a model can read at once. You now know the first end: small vocabulary, long sequence, many attention pairs.
Step 2: Word tokens cost embedding rows
At the word level each whole word is a token. Word-level schemes usually split punctuation off too, so splitting on spaces is the simplest case; the same illustrative sentence has 27 space-separated tokens.1 Sequences are short, but the vocabulary must hold every word.
The embedding matrix is the table with one row per vocabulary entry and one column per embedding number, called d_model. The pinned Hugging Face documentation says a big vocabulary forces an enormous embedding matrix, which increases memory and time. Its example is a word-level model whose space and punctuation tokenization gave a vocabulary of 267,735 entries.
At an illustrative d_model of 768, that is 205.6 million embedding numbers. A 50,257-entry vocabulary needs 38.6 million. A 40,478-entry vocabulary, that of the original generative pre-trained transformer (GPT; see what GPT stands for), needs 31.1 million embedding numbers at width 768.1 The word-level matrix is 5.3 times the 50,257-entry one. Per the current Hugging Face page, words missing from the vocabulary map to <unk>, so such a model cannot handle new words.
Text description of the diagram
- A 267,735-entry vocabulary needs 205.6 million embedding numbers at width 768.
- A 50,257-entry vocabulary needs 38.6 million.
- A 40,478-entry vocabulary needs 31.1 million.
Step 3: Subwords sit between the two costs
A subword vocabulary keeps frequent words whole and splits rare ones into known pieces. The Hugging Face summary says subword tokenization gives a reasonable vocabulary size while still learning meaningful representations, and its example splits "annoyingly" into "annoying" and "ly".
The pinned version adds that transformer models rarely have a vocabulary above 50,000, especially when trained on one language. Meta states that Llama 3 uses a tokenizer with a vocabulary of 128K tokens (April 2024). A larger vocabulary gives shorter sequences, most visibly for multilingual text and code, and costs a bigger embedding matrix.
The English estimate from OpenAI is that one token is about four characters, about three-quarters of a word, and that 100 tokens are about 75 words. These are estimates, not exact counts. Applied to the 144-character sentence, both rules give about 36 tokens.1
Step-by-step description
- One token per character: 144.
- One token per word: 27.
- About 36 tokens, between the two.
- The illustrative sentence is 144 tokens at character level.
- The sentence is 27 tokens at word level.
- The sentence is about 36 subword tokens by the four-character estimate.
Step 4: Byte pair encoding sets the length by merging
Most subword vocabularies come from byte pair encoding (BPE), which keeps merging the most frequent pairs of pieces until the vocabulary reaches the target size. Byte-level BPE starts from 256 byte tokens, so it needs no <unk>. The target size is the knob: more merges mean longer average tokens. The Hugging Face tokenizer summary gives the arithmetic for the 50,257-entry vocabulary used above: 256 byte tokens, plus 50,000 merges, plus one special end-of-text token. Each merge adds one longer piece to the vocabulary, so a larger merge budget keeps more common words whole, while a smaller one splits more words into pieces and makes sequences longer. This is why the vocabulary size is chosen once, before training, and then fixed.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
In terms of token size, how long do you think each token should be?
There is no single ideal length. It is a trade-off set by the vocabulary size. Characters give a tiny vocabulary but very long sequences, and attention gets more expensive as sequences grow. Whole words give short sequences but a huge embedding matrix and unknown words. So I would use subwords from byte pair encoding, with a vocabulary from tens of thousands to a few hundred thousand entries, which in English averages about four characters per token. I would check that on my own data and languages, because the average moves with the tokenizer and the language.
What makes it Staff-level: it names both costs, treats vocabulary size as the control, and says the average must be measured on the real data.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | "A token is a word." | A token is a piece of text, often a subword. | Gives the English estimate and says it is only an estimate. |
| Depth | One number, no reason. | Names the character and word extremes. | Explains how the vocabulary size sets average length. |
| Trade-offs | None. | Long sequences against a large vocabulary. | Adds the attention cost and the embedding-matrix cost with numbers. |
| Communication | Rambling or a bare figure. | Definition, then trade-off. | Definition, trade-off, then how to measure on the data. |
| Follow-up readiness | Stuck on "why that size?" | Knows rare words split into pieces. | Knows other languages and tokenizers change the average. |
Three variants of this question
What is the best vocabulary size for a language model?
No single size is best. The vocabulary size is a hyperparameter, a setting chosen before training. The Hugging Face documentation says transformer models rarely exceed 50,000 entries for a single language, while Llama 3 uses a 128K-token vocabulary. A larger vocabulary shortens sequences but enlarges the embedding matrix, so pick it by measuring both costs on your data.
Why not just use characters and avoid unknown words?
Characters do remove the <unk> problem, but sequences become much longer and each token carries little meaning. Byte-level BPE gets the same no-unknown-word benefit: GPT-2 starts from 256 byte tokens, then adds merges, so any text can be written and common pieces stay short in sequence.
Is four characters per token true for every language?
No. OpenAI says the same text can give different token counts depending on the model, its encoding and the language, and that the four-character figure is an estimate. For a non-English language, measure the average with the actual tokenizer instead of assuming four.
Common mistakes
- "A token is a word." (wrong) A token can be a character, part of a word, a whole word or punctuation. (right) "A token is a piece of text, and the token count differs from the word count."
- "Every token is four characters." (wrong) That is an English rule of thumb, an estimate. (right) "About four characters in English, and I would measure it for the model and language in use."
- "There is one ideal token length." (wrong) The length follows from the vocabulary size you choose. (right) "It is a trade-off between sequence length and vocabulary size."
- "A bigger vocabulary is always better." (wrong) It enlarges the embedding matrix and adds memory and time cost. (right) "Bigger shortens sequences but costs embedding parameters, so I balance the two."
Follow-up questions
How many tokens is 750 words?
About 1,000 tokens, because OpenAI's rule of thumb is that 100 tokens are about 75 words. It is an estimate for English and the exact count depends on the tokenizer.
Does a bigger vocabulary make a better model?
Not automatically. It makes each token longer and the sequence shorter, but the embedding matrix grows with it; the 267,735-entry word-level vocabulary needs about 5.3 times the embedding numbers of a 50,257-entry one at the same width.1
What happens to a word the model has never seen?
A subword tokenizer splits it into known pieces, as "annoyingly" becomes "annoying" and "ly" in the Hugging Face example. Byte-level BPE can write any text, while a word-level tokenizer would map the word to <unk>, losing it.
How does token length affect the context window?
Longer average tokens mean fewer tokens for the same text, so more text fits in the context window. The relationship between words and tokens is only approximate, so budget with a safety margin.
Why do non-English languages use more tokens?
Usually because the vocabulary has fewer whole-word pieces for them, so their words split into more pieces. OpenAI says token counts vary by language, so measure with the real tokenizer.
- A token is a piece of text, not a fixed number of characters; in English it averages about four characters.
- Token length is set by the vocabulary size, a design choice, not a constant.
- Short tokens give long sequences and more attention work; long tokens give a bigger embedding matrix.
- Subword tokens sit between the extremes; byte-level BPE, which starts from 256 bytes, never needs an unknown token.
- Measure the real average on your own tokenizer and language.
Sources
- What are tokens and how to count them? — OpenAI Help Center (accessed 2026-10-26), 26 Oct 2026
- Summary of the tokenizers (current page) — Hugging Face Transformers documentation (accessed 2026-10-26), 26 Oct 2026
- Summary of the tokenizers (pinned version v4.44.0) — Hugging Face Transformers documentation (accessed 2026-10-26), 26 Oct 2026
- GPT (OpenAIGPTConfig, default vocab_size 40478) — Hugging Face Transformers documentation (accessed 2026-10-26), 26 Oct 2026
- Introducing Meta Llama 3 — Meta AI, 18 Apr 2024
