In this article
Before you start
Why interviewers ask this
The question looks like trivia, and that is why it works. A candidate who has only used a chatbot can recite the three words. A candidate who has read about the models can say what each word commits the model to, and an interviewer hears the difference within the first answer.
The weak answer is the expansion alone: "generative pre-trained transformer." The strong answer attaches a fact to each word: what generating means, what pre-training buys, and what makes the architecture a transformer. The question is also a doorway. The next questions are usually about how a transformer works and when fine-tuning is still needed. This page is built so that you can answer the doorway question and the two behind it.
GPT from first principles
- GPT
- GPT is a family of language models, started by OpenAI, that are transformers pre-trained on unlabeled text and then used to generate text one token at a time.
In plain words
Think of a person who reads a very large library with one habit. They cover the next word, guess it, then check. Nobody marks the guesses; the book itself is the answer key. After enough guesses they know how sentences go. Later you ask them for a specific job, such as summarising a contract. They need only a few examples. That reader is "pre-trained", and the habit of finishing sentences is "generative". The analogy breaks in one place. A person guesses one word at a time, while in training the model guesses every word of a passage at once. What it keeps is only the numbers its guesses adjusted.
In one precise sentence: GPT is a transformer trained to continue text, which it learns from unlabeled text before any task-specific training.
Step 1: Generative means the model writes one token at a time
A generative model produces new output rather than choosing a label for an input. GPT's output is the next token. Given a sequence of tokens, the model outputs a score for every token in its vocabulary, one token is chosen, it is added to the sequence, and the model runs again on the longer sequence.
That loop is the whole of text generation, repeated until the model emits a stop token or hits a length limit. The first GPT cut text into tokens with byte-pair encoding, a method that starts from single characters and merges frequent pairs, and its vocabulary came from 40,000 merges (Radford et al., 2018). Input: a token sequence. Output: one more token. What you now know: "generative" means the output is more text, produced one piece at a time.
Step 2: Pre-trained means it learned from unlabeled text first
Pre-trained means the model's first training used text with no labels, and only afterwards was it adapted. A label is a correct answer that a person attaches to an example, such as "positive" on a film review, and labels are expensive. The 2018 paper starts from the observation that large unlabeled text collections are abundant while labeled data for specific tasks is scarce, and it proposes generative pre-training of a language model on diverse unlabeled text, followed by discriminative fine-tuning on each specific task.
The training text was BooksCorpus, which holds over 7,000 unique unpublished books from many genres, about 800 million words (Devlin et al., 2018) (Radford et al., 2018). The training task needs no labeler, because the next token of any text is its own answer. Input: a large pile of text. Output: a model with general knowledge of language. What you now know: "pre-trained" names a stage, not a finished product.
Step 3: Transformer means attention that only looks back
A transformer is a network architecture built on attention instead of recurrence or convolution, introduced by Vaswani et al. (2017) for translation. In self-attention each position weighs the others, and the transformer architecture stacks layers of it.
The original transformer architecture has two halves. An encoder is the half that reads the whole input at once, and a decoder is the half that writes the output one token at a time. GPT keeps only the decoder. The first GPT was a 12-layer decoder-only transformer with masked self-attention, 768-dimensional states and 12 heads (Radford et al., 2018); the heads run side by side, which is multi-head attention. "Masked" answers the follow-up question.
A mask hides every later token, so a position can use only itself and the tokens before it. That lets the model train on the next-token task at every position at once without peeking, and generate left to right. A causal mask is what separates a decoder, which cannot see later tokens, from an encoder, which sees the whole input. What you now know: the T is the mechanism, the mask the direction.
Step-by-step description
- The output is one more token.
- It first learned from unlabeled text.
- Masked attention looks only backward.
- Generative means the output is one more token, produced one piece at a time.
- Pre-trained means the model first learned from unlabeled text, over 7,000 books for the first GPT.
- A mask hides every later token from each position, so a position can use only itself and the tokens before it.
Where does the name come from?
The name comes from OpenAI's 2018 paper on generative pre-training, although that paper never spells out the letters GPT. The paper, Radford et al. (2018), is titled "Improving Language Understanding by Generative Pre-Training", and its results tables label the model a "Finetuned Transformer" language model. The three-word expansion is in print by October 2018, when the Bidirectional Encoder Representations from Transformers (BERT) paper (Devlin et al., 2018) calls that model "the Generative Pre-trained Transformer (OpenAI GPT)". OpenAI kept the letters for its later models, such as GPT-3. What you now know: GPT is a label for one paper's method, and the label outlived the paper's model.
Did the meaning change as the models grew?
The expansion stayed fixed while the scale and the way the models are used changed. GPT-3, described in May 2020, is "an autoregressive language model with 175 billion parameters" that is applied "without any gradient updates or fine-tuning", with tasks and demonstrations given purely as text (Brown et al., 2020). A gradient update is the step that changes a model's weights during training, so the second quotation says the weights stay fixed while the model is used.
The first GPT, by contrast, learned each new task by fine-tuning, meaning extra training on labeled examples of that task. Size and usage changed together. The first GPT had 12 layers, each 768 numbers wide, about 116 million parameters1 (a parameter is one adjustable number of the network); GPT-3 has 96 layers, each 12,288 wide, and was trained for 300 billion tokens (Brown et al., 2020, §2.1–2.2). The basic design, a decoder-only transformer trained to continue text, did not change, although GPT-3 adds details such as alternating dense and sparse attention layers.
Text description of the diagram
- The first GPT was a 12-layer decoder-only transformer with 768-wide states, about 116 million parameters by our count.
- GPT-3 has 96 layers 12,288 wide and 175 billion parameters, trained on 300 billion tokens.
- The first GPT saw about 800 million words in pre-training (BERT paper), and GPT-3 was trained for 300 billion tokens.
- The first GPT was trained further on labeled examples for each task, and GPT-3 was applied without gradient updates or task-specific training.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
What does GPT stand for, and why does each word matter?
GPT stands for generative pre-trained transformer, and I'd tie one fact to each word. Generative means it produces text one token at a time. Pre-trained means it first learns from a large body of unlabeled text by continuing it, so one model serves many tasks.
Transformer is the architecture: stacked layers of self-attention, and in GPT the attention is masked, so each position sees only itself and earlier tokens, which lets it learn next-token prediction at every position in parallel. The trade-off is that no position sees the words after it, which suits writing better than understanding a whole sentence, and pre-training costs a great deal of compute. The first GPT was a twelve-layer decoder; GPT-3 scaled the same idea to 175 billion parameters.
What makes it Staff-level: each word is tied to a consequence, the mask is named with its cost, and the answer ends on how the idea scaled.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | Gives the three words | Defines each word correctly | Adds that the mask makes parallel next-token training possible |
| Depth | Stops at the expansion | Names next-token output and unlabeled text | Cites the paper's decoder and its training corpus |
| Trade-offs | None | Says pre-training saves labeling | Says what it costs: no view of later words, and the compute of pre-training |
| Communication | Reads out an acronym | Explains in order G, P, T | Explains in order and checks the interviewer's level |
| Follow-up readiness | Stuck on "why masked?" | Answers "why masked?" | Moves on to fine-tuning versus prompting unprompted |
Three variants of this question
"What does GPT stand for in ChatGPT?"
The same three words: generative pre-trained transformer. ChatGPT is OpenAI's chat product, released in November 2022 and built by fine-tuning a model from the GPT-3.5 series so that it follows instructions and holds a dialogue (OpenAI, November 2022).
"Is GPT the same as a large language model?"
No. A large language model is any language model trained at scale; GPT is one family of them. GPT-3's own paper describes it as an autoregressive language model, which means it produces each token from the tokens before it (Brown et al., 2020).
"Does GPT ever mean something else?"
Yes, outside machine learning. In storage, GPT is the Globally Unique Identifier (GUID) Partition Table, a layout for the partitions of a disk that is part of the Unified Extensible Firmware Interface (UEFI) standard (GUID Partition Table article, read October 2026). In economics, GPT means general-purpose technology, a technology that can affect an economy at a national or global level (General-purpose technology article, read October 2026). In an interview, say which one you mean.
Common mistakes
- "GPT is a chatbot." (wrong) A chatbot is a product; GPT is the family of models behind some of them. (right) "GPT is a type of language model; chat is one way to use it."
- "The T stands for training." (wrong) It is transformer, the architecture. (right) "Transformer, an attention-based network."
- "Pre-trained means finished." (wrong) It is the first stage; the paper pairs it with fine-tuning. (right) "Pre-trained means learned general patterns before any task."
- "GPT reads in both directions." (wrong) The masked attention hides later tokens. (right) "Each position attends only to itself and earlier tokens; it writes left to right, one token at a time."
Follow-up questions
What is the difference between GPT and a transformer?
A transformer is an architecture, and GPT is a particular model family built from it. GPT uses the decoder half of the original design, with masked attention, and trains it to continue text. The mask has a cost: no position sees the words after it, which the BERT authors called "sub-optimal for sentence-level tasks" in the OpenAI GPT design (Devlin et al., 2018).
Why is it called pre-trained when it is still trained?
The word names the order of stages, not a lack of training. The model is trained first on unlabeled text, a stage that costs a great deal of computing but needs no labels, and is adapted or prompted afterwards for a specific task.
What did GPT-3 change compared with the first GPT?
It scaled the same idea to 175 billion parameters and showed that tasks could be given as text, with a few examples and no weight updates, instead of fine-tuning on each task (Brown et al., 2020).
Do GPT models need fine-tuning to be useful?
Not always. The first GPT needed fine-tuning for each task, while GPT-3 was applied with examples in the prompt; the choice today depends on cost, data and accuracy, and the serving side is covered in the large language model (LLM) inference path.
- GPT is short for generative pre-trained transformer.
- Generative: the model outputs one token at a time and feeds it back in.
- Pre-trained: it first learns from unlabeled text; the first GPT used over 7,000 books.
- Transformer: attention-based, and in GPT the attention is masked to look only backward.
- The acronym is also a disk layout and an economics term, so say which one you mean.
Sources
- Improving Language Understanding by Generative Pre-Training — OpenAI (Radford, Narasimhan, Salimans, Sutskever); the PDF carries no date, so the date is that of the OpenAI announcement, 11 Jun 2018
- Language Models are Few-Shot Learners — arXiv (Brown et al.), version 1, 28 May 2020
- Attention Is All You Need — arXiv (Vaswani et al.), version 1, 12 Jun 2017
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — arXiv (Devlin et al.), version 1, 11 Oct 2018
- ChatGPT: Optimizing Language Models for Dialogue — OpenAI blog post 'ChatGPT: Optimizing Language Models for Dialogue', dated 30 November 2022; the openai.com page returned HTTP 403, so the text was read in the Internet Archive capture of the same post, 30 Nov 2022
- GUID Partition Table — Wikipedia (date = read date; a pointer for a definition only), 24 Oct 2026
- General-purpose technology — Wikipedia (date = read date; a pointer for a definition only), 24 Oct 2026
