In this article
Before you start
This page is one step of the transformer architecture guide, which follows text from tokens to generated output.
Why interviewers ask this
A wrong answer about tokenization shows up later as a wrong cost estimate, a wrong plan for context length, or a wrong explanation of why a model fumbles spelling and arithmetic. Tokenization is the first step in every large language model (LLM) pipeline, so interviewers use it to check whether you know what the model actually reads. You can compare your answer with the lists of NLP interview questions and LLM interview questions.
A weak answer says "it splits text into words." A strong answer says the pieces are usually subwords from a fixed vocabulary, that the vocabulary is built by a separate step before the model is trained, and that the design trades sequence length against vocabulary size. A Staff-level answer adds what breaks: numbers, code and some languages often split into many tokens, and different models count the same text differently.
Tokenization from first principles
- Tokenization
- Tokenization is a fixed procedure that splits raw text into tokens and maps each token to an integer from a vocabulary, so a language model receives numbers. A decode step reverses it. The procedure is built before the model is trained, and the model's training does not change it.
In plain words
Imagine a box of letter tiles. Some tiles hold a whole common word. Others hold an ending or a single letter. To write a word, you pick the fewest tiles that spell it. The box is the vocabulary, and each tile is a token. Every tile also has a number printed on it, and the model sees only those numbers.
The analogy breaks in one place. Nobody picks the tiles by hand. A counting procedure picks them from a large body of text, so common strings earn their own tile and rare strings are spelled from small ones.
In one precise sentence: tokenization is a fixed, trained mapping from text to a sequence of integer IDs, plus a decode step that maps the IDs back to text. The Stanford information retrieval textbook treats it as cutting a character sequence into pieces, and notes that the cutting rules have pitfalls.
Step 1: Start from single characters and an end marker
The byte pair encoding (BPE) method of Sennrich et al. starts from single characters plus an end-of-word marker. The page uses one small illustrative corpus: the word "low" five times, "lower" twice, "newest" six times and "widest" three times. The marker, written _, shows where a word stops, so an ending such as "est" at the end of a word differs from "est" inside one.
The starting vocabulary is the ten letters l, o, w, e, r, n, s, t, i and d plus the marker, which is 11 tokens. Written in single characters, the whole corpus takes 95 tokens.1 That is the longest possible sequence and the smallest vocabulary. You now know the starting point; the next step shows how pieces grow.
Step 2: Count pairs and merge the most frequent one
The most frequent adjacent pair is merged into one new token. A tokenizer built this way counts every neighbouring pair across the corpus, weighted by how often each word occurs. A merge rule is one recorded instruction: join this adjacent pair into a new token.
In the example, "e" followed by "s" occurs 9 times: six in "newest" and three in "widest". The pairs "s" then "t", and "t" then the marker, also occur 9 times, so three pairs tie.1 The script breaks ties alphabetically, which is a choice of this page; real tools use their own rules. It merges "e" and "s" into the new token "es", and the vocabulary grows to 12 tokens.1 Sennrich et al. adapted this merge idea to split rare words into pieces.
Step 3: Repeat until the vocabulary is the size you chose
Merging repeats until the vocabulary reaches the chosen size. This method is BPE. Each pass recounts the pairs and merges the new winner, and the merge order is the tokenizer's recipe.
Here the six merges, in order, make "es", "est", "est_", "lo", "low" and "ew", with winning counts 9, 9, 9, 7, 7 and 6.1 The vocabulary ends at 17 tokens. A subword is a piece of text shorter than a word but longer than a character, such as the ending "est_", which "newest" and "widest" now share. Writing the corpus now takes 48 tokens, against 95 for single characters and 16 for whole words.1 Whole-word tokens are fewest but cannot spell an unseen word.
Text description of the diagram
- Writing the corpus takes 95 single-character tokens, 48 tokens after six merges and 16 whole-word tokens.
- Whole-word tokens are fewest but cannot spell an unseen word.
Step 4: Turn the pieces into token IDs and decode them back
Each final token maps to an integer ID. The IDs here follow the sorted starting tokens, with the marker _ as 0, then d, e, i, l, n, o, r, s, t and w as 1 to 10, and the merged tokens as 11 to 16 in merge order.1
To encode "lower", replay the merge rules in order: it becomes low, e, r and the marker, which is the IDs 15, 2, 7, 0. "newest" becomes n, ew and est_, which is 5, 16, 13.1 The word "lowest" never appeared in the corpus, yet it encodes as low and est_, which is 15 and 13.1 Decoding looks up each ID and joins the pieces, turning the marker back into a space. The model then learns an embedding for each ID. Many production tokenizers split text first and start from bytes (or fall back to bytes).
Step-by-step description
- Start from single characters and an end marker.
- The most frequent adjacent pair becomes one token.
- Merging repeats until the vocabulary is the chosen size.
- Each final token maps to an integer.
- Byte pair encoding starts from single characters plus an end-of-word marker.
- The most frequent adjacent pair is merged into one new token.
- Merging repeats until the vocabulary reaches the chosen size.
- Each final token maps to an integer ID.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
What is tokenization?
Tokenization converts text to integer IDs before the model sees it. A tokenizer is trained separately, before the model, on a text corpus. The most common method, byte pair encoding, starts from characters or bytes and repeatedly merges the most frequent adjacent pair until it reaches a chosen vocabulary size; WordPiece scores merges by likelihood, and Unigram prunes a large vocabulary down.
Encoding replays the merge rules in order. I use subwords because whole-word vocabularies are huge and fail on unseen words, while single characters make sequences long and attention expensive. The cost is that numbers, code and rare languages often split into many tokens, so counts differ by model, and the tokenizer is frozen: changing it usually means retraining or re-learning embeddings.
It is Staff-level because it names the trade-off and the failure modes, not only the definition.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | Says tokens are words. | Says tokens are subwords with integer IDs from a fixed vocabulary. | Adds that the tokenizer is trained separately and frozen before the model. |
| Depth | Gives no method. | Names BPE, WordPiece or Unigram and describes merging the most frequent pair repeatedly. | Adds that encoding replays merge rules in order and byte bases cover any input. |
| Trade-offs | Names none. | Names sequence length against vocabulary size. | Adds weak spots: numbers, code, rare languages, and counts that differ by model. |
| Communication | Uses jargon without definitions. | Defines token and vocabulary, then the method. | Gives a short merge example, then the trade-off. |
| Follow-up readiness | Stuck on unseen words. | Explains fallback to smaller pieces. | Links tokens to context window, cost and embeddings. |
Three variants of this question
Is a token the same as a word?
No. A token can be a whole word, a word fragment, punctuation or a byte. In the worked example "lowest" is two tokens, low and est_, although the word never appeared in the training corpus.1
Why do two models give different token counts for the same text?
Each model family has its own tokenizer and vocabulary, so the same text splits differently. Count with the tokenizer of the model you call.
Is tokenization learned by the neural network?
No. The tokenizer is trained beforehand on text statistics and then fixed. The network learns an embedding for each token ID, not the splitting rules.
Common mistakes
- "A token is a word." (wrong) Tokens are often subwords, punctuation or bytes. (right) "A token is a unit from the tokenizer's vocabulary; many are word pieces."
- "The model learns how to split text." (wrong) The tokenizer is a separate, frozen step. (right) "The tokenizer is trained first; the model only learns embeddings for its IDs."
- "More merges are always better." (wrong) A bigger vocabulary shortens sequences but adds rare tokens that are seen too little to learn well. (right) "I choose the vocabulary size as a trade-off."
- "All models count tokens the same." (wrong) Vocabularies differ between model families. (right) "I count with the target model's own tokenizer."
Follow-up questions
What is byte pair encoding tokenization?
Byte pair encoding tokenization builds the vocabulary by repeatedly merging the most frequent adjacent pair, as in Steps 2 and 3. The byte pair encoding guide covers the algorithm in full.
What happens if you change the number of merges?
Fewer merges mean a smaller vocabulary and longer sequences; more merges mean the reverse. With zero merges the corpus costs 95 tokens, and after six it costs 48.1
How does tokenization work in an LLM, from input to output?
The prompt is encoded to IDs, each ID is looked up as an embedding, and the layers, including self-attention, process them. The model outputs the next token's ID, and decoding turns the IDs back into text.
How do tokenizers handle languages without spaces, and emoji?
Some tokenizers, such as SentencePiece, work on raw text without splitting on spaces first. Byte-based vocabularies can represent any character because every byte is in the base vocabulary.
What is tokenization of payments or assets?
It is a different idea: replacing a sensitive value with a stand-in, or representing an asset digitally. This page does not cover it; use a payments standards body or a finance source for that meaning.
Why does the token count matter for the context window?
A context window is measured in tokens, so a tokenizer that splits text into more pieces fits less text. See the context window guide.
- Tokenization maps text to integer IDs from a fixed vocabulary, and decoding reverses it.
- Subword tokens balance sequence length against vocabulary size.
- Byte pair encoding, the most common method, builds the vocabulary by repeatedly merging the most frequent adjacent pair; WordPiece and Unigram build it differently.
- The tokenizer is trained separately and frozen; the model learns embeddings for its IDs.
- Token counts differ between models, so count with the model's own tokenizer.
Sources
- Neural Machine Translation of Rare Words with Subword Units (arXiv 1508.07909) — arXiv (Sennrich, Haddow, Birch; ACL 2016), 10 Jun 2016
- Tokenization, in Introduction to Information Retrieval — Stanford NLP (Manning, Raghavan, Schütze), 7 Apr 2009
- SentencePiece: A simple and language independent subword tokenizer (arXiv 1808.06226) — arXiv (Kudo and Richardson; EMNLP 2018), 19 Aug 2018
