In this article
How the questions are ordered
Each group reuses the vocabulary of the groups before it. New to the field? Start with the transformer learning path, then return here to test yourself.
Text description of the diagram
- The index has five groups of questions in study order.
- Each group reuses the vocabulary of the one before.
Text and tokens
What is tokenization, and why do language models use subword tokens?
Tokenization cuts text into units called tokens and maps each one to an integer id. Subword tokens, such as those from byte pair encoding, split a rare word into pieces so a fixed vocabulary can still encode any text. Sennrich, Haddow and Birch encode rare and unknown words "as sequences of subword units" so that translation is open-vocabulary (arXiv v5, June 2016).
Why it is asked: tokenization sits under cost, context length and odd failures, so it separates people who have used models from people who have read about them. Read the full explanation: byte pair encoding.
What is the difference between stemming and lemmatization?
Stemming chops a word to a common form without looking at context, so its output may not be a real word: the Wikipedia article gives "lazy" becoming "lazi". Lemmatization finds the dictionary form, the lemma, using the word's meaning and part of speech, so "better" maps to "good", a link stemming misses (Wikipedia, lemmatization, read 2026-10-24). Stemming is faster; lemmatization is more accurate.
Why it is asked: it checks that you know classical text normalisation, still used in search pipelines. Read the full explanation: classical machine learning.
What is TF-IDF, and when would you still use it?
Term frequency (TF) times inverse document frequency (IDF), written TF-IDF, scores a word by its count in a document and how rare it is across documents. The scikit-learn default IDF is ln((1 + n) / (1 + df)) + 1, so among 3 documents a word in one scores 1.693 and a word in all three scores 1.0 (scikit-learn tf-idf guide, read 2026-10-24).1 It is a strong keyword baseline, but it has weak semantic sensitivity: similar meaning with different words is not linked (Wikipedia, vector space model, read 2026-10-24).
Why it is asked: it tests whether you reach for a cheap baseline before a neural model. Read the full explanation: hybrid search.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Embeddings
What is a word embedding?
A word embedding is a list of numbers that represents a word so that similar words get nearby lists. Mikolov, Chen, Corrado and Dean learn such "continuous vector representations of words" from very large text collections, and report that vector("King") minus vector("Man") plus vector("Woman") lands closest to vector("Queen") (arXiv v3, September 2013). Today the same idea is applied to sentences and documents.
Why it is asked: it is the bridge from words to every neural model that follows. Read the full explanation: what an embedding is.
What is the difference between CBOW and skip-gram?
Both are word2vec architectures. In the paper's own words, the continuous bag-of-words (CBOW) model predicts the current word from its context, and skip-gram predicts the surrounding words from the current word (Mikolov et al., arXiv v3, 2013, Figure 1). The same paper reports skip-gram working slightly worse on the syntactic test and better on the semantic one in its experiments.
Why it is asked: it is the standard check that you know how word vectors are trained. Read the full explanation: continuous bag of words.
What is the difference between static and contextual embeddings?
A static embedding gives a word one vector wherever it appears; a contextual embedding computes a vector from the whole sentence, so "bank" gets different vectors in "river bank" and "bank loan". Peters et al. derive their vectors from the internal states of a deep bidirectional language model so they capture how word use varies across linguistic contexts (arXiv v2, March 2018).
Why it is asked: it explains polysemy and why transformers replaced lookup tables. Read the full explanation: embeddings and vector search.
Sequence models and attention
Why did LSTMs replace plain recurrent networks?
A plain recurrent network suffers from vanishing gradients: the signal that teaches long-range dependencies shrinks as it travels back through many steps. The long short-term memory (LSTM) network, published in 1997 by Hochreiter and Schmidhuber, adds a memory cell with gates that let gradients flow with little attenuation. The common form today has input, output and forget gates; the forget gate was added by Gers, Schmidhuber and Cummins in 1999 (Wikipedia, LSTM, read 2026-10-24).
Why it is asked: it is the first step of the history that leads to attention. Read the full explanation: LSTM.
Why did transformers replace recurrent models?
Recurrence processes tokens one after another, which limits parallel training. Vaswani et al. propose the Transformer, "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely", and report it is more parallelizable and needs significantly less training time (arXiv v7, August 2023). Attention also connects any two positions directly.
Why it is asked: it tests whether you can state the reason for the architecture, not only its name. Read the full explanation: transformer architecture.
What is self-attention?
Self-attention lets each token build its output as a weighted average of the other tokens' values, where each weight comes from comparing that token's query with every token's key, turned into probabilities by a softmax. Vaswani et al. define this as scaled dot-product attention (arXiv v7, 2023, section 3.2).
Why it is asked: it is the core mechanism; weak answers repeat the words "focus on important tokens" with no arithmetic. Read the full explanation: self-attention.
Why are attention scores divided by the square root of d_k?
If the query and key entries are independent with mean 0 and variance 1, their dot product has variance d_k, so scores grow with the key dimension, and the paper suspects large scores push the softmax into regions with extremely small gradients (Vaswani et al., 2017, section 3.2.1 and footnote 4). In a seeded simulation with d_k = 64 the spread was 8.04, and the largest of eight softmax weights averaged 0.874 without scaling; after dividing by the square root of d_k it averages 0.361 (illustrative).1
Text description of the diagram
- Without scaling the largest softmax weight averages 0.874 in the simulation.
- After dividing by the square root of d_k it averages 0.361.
Why it is asked: it separates memorised formulas from understanding of variance. Read the full explanation: self-attention.
Why does a transformer use multi-head attention?
Several heads let the model look at different kinds of relationship at once. The paper says multi-head attention "allows the model to jointly attend to information from different representation subspaces at different positions", and that with a single head, averaging inhibits this (Vaswani et al., 2017, section 3.2.2). Each head works on a smaller projection, and the heads' outputs are concatenated and projected again.
Why it is asked: it checks that you can name the benefit and the mechanism. Read the full explanation: multi-head attention.
Why does a transformer need positional information?
Attention treats its input as an unordered set, so order must be injected. The authors note the model has no recurrence and no convolution, add sinusoidal position encodings and report that learned positional embeddings gave nearly identical results (Vaswani et al., 2017, section 3.5). Many newer models use rotary position embeddings instead.
Why it is asked: it shows you know what attention alone cannot do. Read the full explanation: rotary positional embedding.
What is the difference between encoder-only, decoder-only and encoder-decoder models?
An encoder reads the whole input with context on both sides, a decoder generates text left to right, and an encoder-decoder pairs them. The paper introducing Bidirectional Encoder Representations from Transformers (BERT) notes that the bidirectional version is often called a Transformer encoder, and the left-context-only version a Transformer decoder "since it can be used for text generation" (Devlin et al., arXiv v2, 2019, footnote 1). Masks enforce the left-to-right rule.
Why it is asked: choosing the family is the first design decision in an NLP task. Read the full explanation: causal masks, encoder and decoder.
Pretraining and adaptation
How is BERT pre-trained?
BERT is trained on two tasks. In masked language modelling (MLM) the model hides 15% of the WordPiece tokens at random and predicts them from both sides. In next sentence prediction, half of the sentence pairs are real neighbours and half are random (Devlin et al., arXiv v2, 2019, section 3.1). The abstract says one extra output layer then fine-tunes it for many tasks.
Why it is asked: it tests whether you know how a bidirectional model is trained without seeing the answer. Read the full explanation: transformer architecture.
Fine-tuning, prompting or retrieval: how do you choose?
Prompt first because it is cheapest; add retrieval when the model lacks facts; fine-tune when you need a behaviour change. Retrieval-augmented generation (RAG) pairs a model's weights, its parametric memory, with a searchable index, its non-parametric memory (Lewis et al., arXiv v4, April 2021). That lets you update knowledge by changing the index, not retraining.
Why it is asked: it checks cost-aware design, the usual practical NLP question. Read the full explanation: fine-tuning vs prompt engineering and RAG architecture.
What is LoRA and why does it help?
Low-rank adaptation (LoRA) freezes the pre-trained weights and trains small rank-decomposition matrices added to each layer. Compared with fine-tuning the 175-billion-parameter generative pre-trained transformer (GPT) 3 with Adam, Hu et al. report 10,000 times fewer trainable parameters and 3 times less graphics processing unit (GPU) memory (arXiv v2, October 2021). For one 4,096 by 4,096 matrix and rank 8, that is 65,536 trainable numbers instead of 16,777,216 (illustrative sizes).1
Why it is asked: parameter-efficient tuning is the default way to adapt large models. Read the full explanation: LoRA fine-tuning.
Generation and evaluation
What is perplexity, and how do you read it?
Perplexity is the exponentiated average negative log-likelihood of a sequence, so lower means the model was less surprised. The Hugging Face documentation also notes it is not well defined for masked models such as BERT and that tokenization changes it, so compare models only with the same tokenizer (Hugging Face, read 2026-10-24). A toy sequence with token probabilities 0.5, 0.25, 0.125 and 0.5 has perplexity 3.364 (illustrative).1
Why it is asked: it ties evaluation to the loss you train on. Read the full explanation: cross-entropy loss.
Greedy search, beam search or sampling: how do you decode?
Greedy search takes the likeliest token each step, beam search keeps several likely sequences, and sampling draws from the distribution. Holtzman et al. find that maximising likelihood gives text that is "bland and strangely repetitive", and propose nucleus (top-p) sampling, which samples only from the smallest set of tokens whose probabilities add up to at least p, cutting off the unreliable tail (arXiv v2, February 2020). Use beam search for translation-like tasks and sampling for open text.
Why it is asked: it checks that you know decoding changes quality even with the same model. Read the full explanation: beam search.
What is the difference between BLEU and ROUGE?
The bilingual evaluation understudy (BLEU), introduced for machine translation, is precision based: it counts how many of a candidate's n-grams appear in the references, with a brevity penalty so very short output is not rewarded (Papineni et al., 2002). Recall-Oriented Understudy for Gisting Evaluation (ROUGE), introduced for summaries, includes ROUGE-N, an n-gram recall measure whose denominator is the reference side (Lin, 2004). So BLEU asks "is what you said right" and ROUGE asks "did you cover what was there".
Why it is asked: it tests whether you pick a metric to match the task. Read the full explanation: BLEU vs ROUGE.
How do you measure a text classifier or a named-entity tagger?
Use precision, recall and their combination, F1. scikit-learn defines F1 from the counts of true positives (TP), false positives (FP) and false negatives (FN) as 2·TP / (2·TP + FP + FN) (scikit-learn F1 documentation, read 2026-10-24); with 8, 2 and 4 that is 0.727 (illustrative).1 Macro averaging weights each label equally; micro averaging counts every instance. The Jurafsky and Martin draft has a chapter on sequence labeling for parts of speech and named entities.
Why it is asked: label imbalance makes accuracy misleading. Read the full explanation: confusion matrix.
What is hallucination, and how do you reduce it?
Hallucination is generated text that is unfaithful or not grounded in the source; Ji et al. survey it across summarisation, dialogue, question answering and translation, and note that neural generation is "prone to hallucinate unintended text" (arXiv v7, July 2024). Grounding the model in retrieved passages helps: the RAG authors report output that is "more specific, diverse and factual" than a parametric-only baseline. It reduces the problem; it does not remove it.
Why it is asked: reliability is the main production concern. Read the full explanation: RAG architecture.
What does "lost in the middle" mean for long contexts?
Language models use information best when it sits at the start or the end of the input and worse when it is in the middle, even models built for long contexts, according to Liu et al. (arXiv v3, November 2023). In practice, put the most important passages first or last and do not assume more context is free.
Why it is asked: it tests judgment about prompts and retrieval. Read the full explanation: context rot.
Preparing for an NLP interview
How can I prepare for an NLP interview?
Learn the five groups above in order, implement tokenization, attention and one metric yourself, and practise saying each answer in under a minute. Use a primary text for gaps: Jurafsky and Martin's draft third edition covers tokens, embeddings, transformers, masked models and retrieval (Stanford, draft of 19 August 2026). Then work through coding and system questions.
Why it is asked: preparation habits predict how you reason under pressure. Read the full explanation: AI engineer interview questions.
Which transformer questions should I expect or ask?
Expect the mechanism questions above, plus cost questions about the cache: why attention scores are scaled, how positions are encoded, what a key-value (KV) cache stores and why memory limits serving. If you are the interviewer, ask for a worked number, not a definition. The transformer hub lists the learning path.
Why it is asked: it frames the follow-ups after the basic ones. Read the full explanation: transformers and KV cache.
- Learn NLP in order: tokens, embeddings, attention, pretraining, then decoding and evaluation.
- State the reason for each design choice, such as the sqrt(d_k) scale or the subword vocabulary, not only its name.
- Pick a metric to match the task: BLEU for precision, ROUGE for recall, F1 for imbalanced labels, perplexity for language models.
- Retrieval reduces hallucination but does not remove it; decoding choices change quality.
Sources
- Attention Is All You Need (arXiv 1706.03762 v7) — arXiv (Vaswani et al.), 2 Aug 2023
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv 1810.04805 v2) — arXiv (Devlin, Chang, Lee, Toutanova), 24 May 2019
- Efficient Estimation of Word Representations in Vector Space (arXiv 1301.3781 v3) — arXiv (Mikolov, Chen, Corrado, Dean), 7 Sep 2013
- Deep contextualized word representations (arXiv 1802.05365 v2) — arXiv (Peters et al.), 22 Mar 2018
- Neural Machine Translation of Rare Words with Subword Units (arXiv 1508.07909 v5) — arXiv (Sennrich, Haddow, Birch), 10 Jun 2016
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401 v4) — arXiv (Lewis et al.), 12 Apr 2021
- LoRA: Low-Rank Adaptation of Large Language Models (arXiv 2106.09685 v2) — arXiv (Hu et al.), 16 Oct 2021
- The Curious Case of Neural Text Degeneration (arXiv 1904.09751 v2) — arXiv (Holtzman et al.), 14 Feb 2020
- Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172 v3) — arXiv (Liu et al.), 20 Nov 2023
- Survey of Hallucination in Natural Language Generation (arXiv 2202.03629 v7) — arXiv (Ji et al.), 14 Jul 2024
- Bleu: a Method for Automatic Evaluation of Machine Translation — ACL 2002 (Papineni, Roukos, Ward, Zhu), 1 Jul 2002
- ROUGE: A Package for Automatic Evaluation of Summaries — ACL Workshop 2004 (Lin), 1 Jul 2004
- Perplexity of fixed-length models — Hugging Face (date = read date), 24 Oct 2026
- sklearn.metrics.f1_score — scikit-learn (date = read date), 24 Oct 2026
- Feature extraction: tf-idf term weighting — scikit-learn (date = read date), 24 Oct 2026
- Lemmatization — Wikipedia (date = read date), 24 Oct 2026
- Long short-term memory — Wikipedia (date = read date), 24 Oct 2026
- Vector space model — Wikipedia (date = read date), 24 Oct 2026
- Speech and Language Processing, 3rd edition draft — Jurafsky and Martin (Stanford; draft date), 19 Aug 2026
