In this article
Before you start
Why interviewers ask this
The wrong choice costs weeks. A team that fine-tunes before it has an evaluation set cannot tell whether the tuning helped, and a team that stuffs a prompt with rules for a problem that needs new knowledge ships a model that is confidently wrong. Interviewers use this question to separate candidates who treat "customise the model" as one decision from those who see three different levers: the instructions, the information the model is given and the weights.
A weak answer says "fine-tuning is better because it is trained on your data". A strong answer names what each lever fixes, the order to try them in, the cost curve of each and the evidence that fine-tuning is a poor way to add facts. The question appears in AI-engineer loops, and the AI engineer interview questions lists the neighbouring ones.
What each one is
- Prompt engineering
- Prompt engineering is the practice of changing the text sent to a language model, its instructions, examples, format and context, so that the same model produces better output, without changing the model's weights.
- Fine-tuning
- Fine-tuning is the process of continuing to train a pretrained model on a smaller set of example inputs and desired outputs, so that its weights change and it produces that behaviour without being told each time.
Prompt engineering in plain words
Prompt engineering is briefing a skilled contractor with a written note every morning. The contractor's skills do not change; the note decides what they do today. Brown et al. (2020) showed that a large model can learn a new task from a few examples or plain instructions "without any gradient updates or fine-tuning", which is the discovery that made prompting a serious tool.
The analogy breaks in one place: a human remembers yesterday's note, while the model reads the whole note again on every request and pays for it in tokens. Anthropic's prompt-engineering overview (read Oct 2026) says to set clear success criteria and a way to test against them before prompting at all, and adds that some failing criteria, such as latency and cost, are fixed more easily by choosing a different model than by editing the prompt.
Fine-tuning in plain words
Fine-tuning is sending the contractor on a training course, a smaller version of the process that trains the base model. Afterwards the habits are built in, so the daily note can be short, but the course teaches a way of working more reliably than it teaches new facts.
OpenAI's supervised fine-tuning guide (read Oct 2026) says the method suits classification, nuanced translation, generating content in a specific format and correcting instruction-following failures, and that you should invest in it only after setting up evaluations. An evaluation set is a collection of test inputs with known good outputs that you score the model against. The same guide says the minimum is 10 examples, suggests starting with 50 and notes that improvements are often seen at 50 to 100 examples. It also says OpenAI is winding down its fine-tuning platform, closed to new users, with eligible existing customers able to create jobs until January 2027, which is a reminder that a tune is tied to a vendor's base model.
How does fine-tuning compare with RAG?
Retrieval-augmented generation (RAG) adds knowledge by fetching documents and placing them in the prompt, while fine-tuning adds behaviour by changing weights. Ovadia et al. (2024) compared unsupervised fine-tuning with RAG on knowledge-intensive tasks and report that fine-tuning gave modest gains while RAG did better on both facts the model had seen in training and entirely new ones. They also found that models struggle to learn new facts through unsupervised fine-tuning on documents, and that many differently worded versions of each fact could alleviate this.
For supervised fine-tuning, Gekhman et al. (2024) report that large language models struggle to acquire new factual knowledge through fine-tuning, and that new-fact examples raise the model's tendency to hallucinate as they are learned. OpenAI's optimization guide (read Oct 2026) frames the split as two axes: context optimization for missing or outdated knowledge, which maximizes accuracy, and large language model (LLM) optimization for inconsistent behaviour, which maximizes consistency. The two add together and can be combined. See the embeddings and vector search guide for the retrieval side.
Text description of the diagram
- Prompting changes what the model reads at request time; fine-tuning changes its weights using example outputs.
- Retrieval-augmented generation adds knowledge by fetching documents and placing them in the prompt.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Take a model shaped like Llama 2 7B: 4,096 hidden units, 32 layers and a feed-forward width of 11,008, which its published configuration lists, giving 6,738,415,616 parameters in total.1 Full fine-tuning updates all of them. LoRA, which Hu et al. (2021) report cuts trainable parameters by 10,000 times for generative pre-trained transformer (GPT) 3 with 175 billion parameters, trains for each adapted matrix of size in by out only rank times (in plus out) numbers. At rank 8 on the query and value projections of every layer that is 8 times 8,192 per matrix, 65,536, over 64 matrices: 4,194,304 parameters.1
| Setup | Trainable parameters | Share of 6.74 billion |
|---|---|---|
| Query and value, rank 8 | 4,194,304 | 0.062% |
| Query and value, rank 16 | 8,388,608 | 0.124% |
| All seven linear layers, rank 16 | 39,976,960 | 0.593% |
| All seven linear layers, rank 64 | 159,907,840 | 2.373% |
Now the cost side, which uses illustrative numbers because real prices change and differ by vendor. Suppose a task needs five worked examples of 200 tokens each in every prompt: 1,000 extra input tokens per request, or a billion for a million requests. A tuning run on 500 examples of 300 tokens for 3 passes is 450,000 training tokens. Training tokens usually cost the same as or more than input tokens, so let the price ratio be the cost of one training token divided by the cost of one input token. The tuning run breaks even when requests times 1,000 equals 450,000 times the ratio: 450 requests at a ratio of 1, 1,350 at 3 and 4,500 at 10.1
Prompt caching moves these numbers. Anthropic's prompt-caching documentation (read Oct 2026) says cache read tokens are 0.1 times the base input price, with writes at 1.25 times for a 5-minute cache. A static example block is exactly what a cache discounts, so at a 90% discount the same break-evens become 4,500, 13,500 and 45,000 requests.1 Two more things this leaves out are the effort of building the 500 examples and the fact that the tuned model may cost more per request. The point is the shape: a prompt's cost repeats on every request unless it is cached, and a tune's training cost is paid once, so volume and the tuned model's serving price decide the break-even.
Text description of the diagram
- A tuning run on 500 examples of 300 tokens for 3 passes is 450,000 training tokens.
- The tuning run breaks even when requests times 1,000 equals 450,000 times the ratio: 450 requests at a ratio of 1, 1,350 at 3 and 4,500 at 10.
The reading: fine-tuning is a way to move repeated instruction cost out of the prompt and into the weights, and it pays only at volume and only for behaviour.
Trade-offs and edge cases
The recommended order follows cost and reversibility.
| Dimension | Prompt engineering | Fine-tuning |
|---|---|---|
| What changes | the text sent per request | the model's weights |
| Fixes best | format, tone, task framing, steps | consistent style, format, narrow tasks |
| Adds new facts | by putting them in the prompt | poorly (Ovadia et al., 2024, for unsupervised tuning on documents; Gekhman et al., 2024) |
| Needs | a prompt and an evaluation set | examples plus an evaluation set |
| Cost shape | repeats on every request, less if cached | paid once, plus any per-token premium for the tuned model |
| Undo | edit the text | retrain, or keep the old model |
Prompt first. OpenAI's guide calls prompt engineering "typically the best place to start", and notes that for tasks like summarization or translation it is often the only method needed. A good prompt also gives you a baseline score, without which you cannot show that a tune helped. See the how prompt engineering works and getting reliable structured output from an LLM, which is often solved without any tuning.
Add retrieval for facts. When the model lacks or has outdated knowledge, put the documents in the context window; a context window is the amount of text a model can read in one request.
Fine-tune for repeatable behaviour. Tune when a measured gap persists across many inputs: the wrong format, an inconsistent tone, a classifier that should be cheaper and smaller. OpenAI's guide says tuning can reach similar accuracy with fewer tokens or a smaller model.
Edge cases. Tuning on a small, narrow set can make a model worse outside that set, so keep a regression evaluation of the general tasks the base model already handled. A tune also inherits the biases of its examples. And tuned weights belong to one base model: when the base is retired, the effort is repeated.
Text description of the diagram
- A team that fine-tunes before it has an evaluation set cannot tell whether the tuning helped.
- Tune when a measured gap persists across many inputs.
Monitoring matters after the choice: see LLM observability for tracking whether prompts, retrieval or a tune is the weak link in production, and LLM inference for how longer prompts raise serving cost.
How to answer it in an interview
When would you fine-tune instead of just prompting?
I start with prompting, because it is cheapest to change and gives me a baseline on an evaluation set. If the model lacks facts, I add retrieval, since research finds retrieval beats unsupervised fine-tuning on documents for new information, and supervised tuning on new facts raises hallucination. I fine-tune when a behaviour gap stays after good prompts: a fixed output format, a consistent style, or a narrow task where tuning lets me shorten the prompt or use a smaller model. At high volume the one-off training cost can beat repeating examples in every request. I would not tune without evaluations or a stable base model.
The spoken structure is: prompt first, retrieval for facts, tune for behaviour, then the volume argument.
Common mistakes
"Fine-tune it so it knows our documents." (wrong) Tuning teaches behaviour better than facts, and research finds retrieval beats unsupervised fine-tuning on documents for new knowledge. (right) "Retrieve the documents; tune only the behaviour."
"Prompting is just a stopgap." (wrong) For many tasks it is all that is needed, and it is the baseline any tune is judged against. (right) "Prompting is the first lever and often the last."
"Fine-tuning needs millions of examples." (wrong) Vendor guidance starts with tens of examples. (right) "Start small, measure, and add examples where the evaluation shows gaps."
"We will compare by feel." (wrong) Without an evaluation set nobody can tell whether a change helped. (right) "I would build the evaluation first."
Follow-up questions
Fine-tuning vs RAG: which one adds new knowledge?
RAG does, because it puts the documents in front of the model on each request. Ovadia et al. report that RAG outperformed unsupervised fine-tuning for both existing and new knowledge.
How do you fine-tune or train a large language model?
Collect input and desired-output examples, hold out an evaluation set, train with full fine-tuning or a parameter-efficient method such as LoRA, then compare against the prompt baseline. Full training from scratch is a different, far larger project; the guide to fine-tuning covers it, and LoRA fine-tuning covers the parameter-efficient route.
When is fine-tuning worth the effort?
When a behaviour gap is measured, repeatable and survives good prompts, or when volume is high enough that moving instructions into the weights saves more tokens than the tuning run costs.
How many examples do you need to fine-tune?
Vendor guidance varies by task: OpenAI's guide gives a minimum of 10 examples, suggests starting with 50 and notes gains at 50 to 100. More varied examples help; check on a held-out set.
What if prompt caching cuts the input price?
The break-even moves out. At a 90% discount on the repeated example block, the worked example's break-evens rise tenfold, to 4,500, 13,500 and 45,000 requests.1
Can you combine prompting, RAG and fine-tuning?
Yes; OpenAI's guide says the approaches are additive. A common pattern is a tuned model for format and tone, retrieval for facts and a short prompt for the task.
- Prompt engineering changes the text the model reads; fine-tuning changes its weights.
- Start with an evaluation set and a prompt baseline; retrieval fixes missing facts; tuning fixes repeatable behaviour gaps.
- Research finds retrieval beats unsupervised fine-tuning for new knowledge.
- At ranks 8 to 16, LoRA trains 0.06% to 0.6% of a Llama-2-7B-sized model; rank 64 on every linear layer reaches 2.4%.
- A prompt's cost repeats on every request unless cached; volume and the tuned model's serving price decide the break-even.
Sources
- Model optimization (accuracy optimization guide) — OpenAI, 24 Oct 2026
- Supervised fine-tuning — OpenAI, 24 Oct 2026
- Prompt engineering overview — Anthropic, 24 Oct 2026
- Language Models are Few-Shot Learners (arXiv 2005.14165 v4) — arXiv (Brown et al.), 22 Jul 2020
- LoRA: Low-Rank Adaptation of Large Language Models (arXiv 2106.09685 v2) — arXiv (Hu et al.), 16 Oct 2021
- Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (arXiv 2312.05934 v3) — arXiv (Ovadia et al.), 30 Jan 2024
- Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (arXiv 2405.05904) — arXiv (Gekhman et al.), 9 May 2024
- Prompt caching — Anthropic, 24 Oct 2026
- Llama-2-7b-hf config.json (model architecture figures) — Hugging Face (mirror of the Meta Llama 2 7B configuration), 24 Oct 2026
