What is the LLM vocabulary size tradeoff, and how do you choose one?

In this article

1. Before you start
2. Why interviewers ask this
3. LLM vocabulary size from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the LLM inference path.

Why interviewers ask this

The vocabulary size looks like a cosmetic setting, yet it changes memory, speed and quality at once, so the question separates candidates who know the whole inference path from those who know only the architecture diagram. It comes up in inference and model-design discussions, where a candidate is asked why a newer model might choose a larger tokenizer than an older one.

A weak answer says "bigger is better because there are more words." A strong answer names both sides: fewer tokens per text against larger tables and logits. A Staff-level answer adds that the best size depends on model scale, so a small model can be hurt by a vocabulary that suits a large one, and it puts a number on the cost of the output layer relative to the rest of the network.

LLM vocabulary size from first principles

Vocabulary

A vocabulary is the fixed list of token pieces a language model can read and write, each with an integer id, whose length V is set before training and normally stays fixed; extending it later means adding and training new embedding and output rows.

Tied embeddings

Tied embeddings are a design in which the input embedding table and the output layer share one weight matrix, so the model stores V × d numbers once instead of twice.

In plain words

Think of a phrasebook. A small phrasebook has only short entries. A sentence then needs many lookups. A large phrasebook has long entries for common phrases. A sentence needs few lookups, but the book is heavy. Many entries are almost never used. The analogy breaks in one place: the model does not only look entries up. It must also learn a vector for every entry and score every entry each time it speaks.

A token is one piece from the vocabulary, such as a whole common word, a word fragment or a single byte. In one precise sentence, a larger V trades shorter sequences for larger tables and a wider output at every step.

Step 1: A tokenizer splits text into pieces from a fixed list

A tokenizer cuts text into vocabulary pieces. Byte pair encoding (BPE) is a tokenizer training method that starts from single characters or bytes and repeatedly merges the most frequent adjacent pair into a new entry, so frequent strings become single tokens. Sennrich et al. (2015) used BPE to build subword units so that a translation model could handle rare words. The number of merges decides V. In the SentencePiece tool the vocabulary size is a number you set before training the tokenizer (the vocab_size option; the paper's abstract does not state this, so check the tool's documentation).

Input: raw text. Output: a list of integer ids. The vocabulary is therefore a set of subword units, not a count of words. You now know where V comes from: a setting made once, before the model sees any training step.

Step 2: Fewer tokens per text shrink the cache and the decode loop

A larger vocabulary encodes the same text in fewer tokens, because more strings earn their own entry. Fewer tokens helps in two ways. First, the KV cache holds a fixed number of bytes per token, so a shorter sequence needs less of it. Second, the model generates one token per decoding step, so a shorter answer needs fewer steps.

How much shorter depends on the text. A tokenizer trained mostly on English splits other languages and source code into more pieces, and a larger vocabulary that covers them reduces that gap. The sizes in the worked example use illustrative token counts, not measured ones. You now know the benefit side: shorter sequences. The next step is the bill.

Step 3: Every entry adds a row to the embedding table and the output layer

The embedding table has one row of d numbers for each of the V tokens, where d is the model's hidden width, so it holds $V\times d$ parameters. The output layer maps a hidden vector back to V scores and holds the same count. Vaswani et al. (2017) share one weight matrix between the embedding layers and the pre-softmax linear layer, which is the tied-embeddings design and halves that cost.

Rare entries are the hidden cost. A token that appears seldom in the training text receives few updates, so its vector stays poor; Land and Bartolo (2024) describe how to detect such under-trained tokens in large language models. Tao et al. (2024) argue that larger models deserve larger vocabularies, which implies the best V grows with model size. A small model with a huge vocabulary spends parameters on rows it cannot train well.

Step 4: Every generated token scores the whole vocabulary

At each step the model outputs logits, one score per vocabulary entry. A softmax over V scores gives the next-token probabilities, and sampling picks one. Multiplying a hidden vector of d numbers by a d × V matrix takes about twice d times V floating-point operations (FLOPs), so the cost of this last layer grows in a straight line with V.

In training the cost is memory as well: the logits for every position in the batch exist at once before the loss is computed. In serving, the logits tensor is small because it covers one position per sequence. You now have the whole trade: a smaller sequence on one side, and larger tables, a wider output layer and a larger logits tensor on the other.

1 · Tokenizer2 · Fewer tokens3 · Tables4 · LogitsReplay

Step-by-step description

1. Text is cut into pieces from V entries.
2. A larger V gives shorter sequences.
3. Each entry adds a row of d numbers.
4. Every step scores all V entries.

• The tokenizer splits text into pieces from a fixed vocabulary of V tokens.
• The embedding table has V rows of d numbers.
• The output layer produces V scores at every generation step.

The tokenizer splits text into pieces from a fixed vocabulary of V tokens. The embedding table has V rows of d numbers. The output layer produces V scores at every generation step.

Worked example

Take a model with hidden width d = 4,096, 32 layers, a body of about 12 × L × d² parameters (the standard dense-layer approximation: attention plus a feed-forward block four times wider than d; with 8 key-value heads the true attention count is lower, and the approximation is deliberate), 8 key-value heads of 128 numbers each and 16-bit cache values, all illustrative. Compare three vocabulary sizes: 32,000, 128,000 and 256,000. The tokens per word are also illustrative: 1.6, 1.3 and 1.2.¹ The three sizes match real choices: Llama 2 used 32,000 tokens, which Tao et al. (2024) estimate was well below its optimum; Llama 3 moved to 128,000; and Gemma uses 256,000 (Gemma Team, 2024).

At 128 KiB of cache per token, a 3,000-word document takes 4,800 tokens and 600 MiB of cache at the smallest vocabulary, but 3,900 tokens and 487.5 MiB at 128,000, and 3,600 tokens and 450 MiB at 256,000 (illustrative).¹ For a 500-word answer, the total per-token work of body plus output layer is 13.9% lower at 128,000 and 14.5% lower at 256,000 than at 32,000 (illustrative; counted in FLOPs, which ignores memory traffic).¹

Text description of the diagram

• A 3,000-word document needs 600 MiB of KV cache at a vocabulary of 32,000 tokens and 450 MiB at 256,000 (illustrative).
• The KV cache holds a fixed number of bytes per token.

A 3,000-word document needs 600 MiB of KV cache at a vocabulary of 32,000 tokens and 450 MiB at 256,000 (illustrative).

The training side moves the other way. A batch of 8 sequences of 4,096 tokens has 32,768 positions, and the logits tensor in 32-bit numbers is about 3.9 GiB, 15.6 GiB and 31.3 GiB at the three sizes (illustrative).¹

Text description of the diagram

• The logits tensor of one training batch grows in step with V.
• The logits for every position in the batch exist at once before the loss is computed.

The logits tensor of one training batch grows in step with V.

Trade-offs and edge cases

No single size wins. The table of costs and gains above is the whole answer, and the edge cases decide which column matters.

• Model scale. In a small model the tables are a large part of all parameters, so a big vocabulary crowds out the layers that do the reasoning. In a very large model the same tables are a small share. This matches the argument of Tao et al. (2024) that vocabulary should grow with model size.
• Language and code mix. A tokenizer with poor coverage of a language makes that language's users pay for more tokens in price, latency and cache; Petrov et al. (2023) report that the same text can need many more tokens in some languages than in others. A larger vocabulary reduces this, but only if the tokenizer was trained on that language.
• Decode is memory-bound. In serving, each step reads the weights from memory, so the saving from fewer steps often matters more than the FLOP count suggests; see high bandwidth memory and graphics processing unit (GPU) memory.
• Tying and quantization. Tying halves the table cost; quantization shrinks it per number. Neither removes the per-step scoring of V entries.
• Batching. Shorter sequences let a server fit more concurrent requests, which interacts with continuous batching.
• Under-trained rows. Entries that almost never occur in training can produce erratic behavior if they appear at run time, as Land and Bartolo (2024) show for under-trained tokens.

How to answer it in an interview

The spoken structure: say the benefit, say the cost, then say that model size decides.

Common mistakes

• "A bigger vocabulary is always better." (wrong) Tables and logits grow and rare tokens are under-trained. (right) "It shortens sequences but costs parameters; the best size depends on model scale."
• "Vocabulary size is the number of words." (wrong) Entries are subword pieces. (right) "It is the count of tokens the tokenizer can emit, many of them word fragments."
• "A bigger vocabulary makes generation much slower." (wrong) The output layer is a modest share of work in a large model, and fewer tokens means fewer steps. (right) "I would compute the output layer's share and the token saving before deciding."
• "Vocabulary size sets how long a prompt can be." (wrong) Context length is a separate limit counted in tokens. (right) "A larger vocabulary fits more text into the same context."

Follow-up questions

Does a bigger vocabulary make an LLM slower?

Per token, yes, because the output layer scores more entries. Per piece of text, often no: fewer tokens means fewer decoding steps and a smaller cache, and in the illustrative example the total work fell. Measure on your own text and hardware.

How do you choose the vocabulary size for a new model?

Treat it as a setting made before training: train candidate tokenizers on your data mix, measure tokens per text for each language, and weigh that against the table cost for your model size. Work such as Tao et al. (2024) suggests larger models can afford larger vocabularies.

Why do some languages use more tokens than others?

A tokenizer merges the pairs it saw most often in its training text, so a language that was rare in that text gets fewer merges and splits into smaller pieces. A larger vocabulary trained on a broader mix narrows the gap.

What are tied embeddings and do they change the trade-off?

Tied embeddings share one matrix between the input table and the output layer, as in Vaswani et al. (2017). They halve the stored parameters, which softens the cost of a larger vocabulary, but every step still scores all V entries.

Is vocabulary size the same as context length?

No. Vocabulary size is how many distinct tokens exist; context length is how many tokens fit in one request. A larger vocabulary helps the same text fit in a given context, and the LLM inference guide covers how context length drives memory.

Sources

1. Neural Machine Translation of Rare Words with Subword Units (arXiv 1508.07909) — arXiv (Sennrich, Haddow, Birch), 31 Aug 2015
2. Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies (arXiv 2407.13623) — arXiv (Tao et al.), 18 Jul 2024
3. Attention Is All You Need (arXiv 1706.03762) — arXiv (Vaswani et al.), 12 Jun 2017
4. SentencePiece: A simple and language independent subword tokenizer (arXiv 1808.06226) — arXiv (Kudo and Richardson), 19 Aug 2018
5. Introducing Meta Llama 3 — Meta AI, 18 Apr 2024
6. Gemma: Open Models Based on Gemini Research and Technology (arXiv 2403.08295) — arXiv (Gemma Team), 13 Mar 2024
7. Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models (arXiv 2405.05417) — arXiv (Land and Bartolo), 9 May 2024
8. Language Model Tokenizers Introduce Unfairness Between Languages (arXiv 2305.15425) — arXiv (Petrov et al.), 26 May 2023