In this article
How this list is ordered
The page groups its questions into eight topics, from language-model basics to the interview process, and the study path ends with serving and then the interview process, which covers projects and the offer. For how these topics map onto the rounds of a real loop, read interview loops.
Text description of the diagram
- The page groups its questions into eight topics, from language-model basics to the interview process.
- The page ends with the serving topic and the interview process, which covers projects and the offer.
Language-model basics
What is a token and why does it matter?
A token is a piece of text, often a word or part of a word, that a language model reads and writes as one unit. Limits, speed and cost are all counted in tokens, so a longer prompt costs more and leaves less room in the context window for everything else.
Why it is asked: it checks that you reason about cost and limits in the model's own units. Read the full explanation: tokens and byte-pair encoding.
What is a context window?
The context window is the amount of text a model can read in a single call, covering the instructions, history, retrieved documents and the answer. A larger window does not mean the model uses all of it well: the Lost in the Middle paper found accuracy degraded when relevant information sat in the middle of a long input (Liu et al., 2023).
Why it is asked: it separates raw capacity from usable capacity. Read the full explanation: AI agent architecture.
What does temperature do?
Temperature scales how random the model's choice of the next token is. Low values make outputs more repeatable, which suits extraction and code; higher values make them more varied, which suits brainstorming. It does not make a model more truthful, and a low temperature can still produce a confident wrong answer.
Why it is asked: it tests whether you pick settings by task, not habit. Read the full explanation: prompting.
What is attention, and why did transformers replace recurrence?
Attention lets each token weigh the other tokens it can see when building its representation; in a decoder-only model such as the Generative Pre-trained Transformer (GPT) family, a causal mask limits that to earlier tokens. The Transformer paper dropped recurrence and convolutions and relied solely on attention, which made it more parallelisable and faster to train than the recurrent and convolutional models it replaced (Vaswani et al., 2017).
Why it is asked: it is the foundation question; weak answers stop at the name. Read the full explanation: transformers.
What is a hallucination and how do you reduce it?
A hallucination is fluent output that is not supported by the input or by fact. Reduce it by grounding the model in retrieved documents, asking for citations, allowing it to say it does not know, and checking claims against a source. You cannot remove it entirely, so design for detection and review.
Why it is asked: it tests honesty about limits and practical mitigations. Read the full explanation: prompting.
What is a KV cache?
A KV cache stores the attention keys and values of tokens already processed, so the model does not recompute them for every new token during generation. It speeds up decoding but uses memory that grows with sequence length and batch size, which is why long contexts are expensive to serve.
Why it is asked: it is the standard serving-cost question. Read the full explanation: KV cache.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Prompting and structured output
What is the difference between zero-shot and few-shot prompting?
Zero-shot prompting gives the model only instructions; few-shot prompting adds a few worked examples of the input and desired output. Examples fix format and style quickly, but they take context space and can bias the model toward the examples' quirks, so test them on held-out cases.
Why it is asked: it checks you know the cheapest lever before heavier ones. Read the full explanation: prompt engineering interview questions.
How do you get reliable structured output from a model?
Specify a schema, use the provider's structured-output or constrained-decoding feature where available, validate the result in code, and retry with the validation error on failure. Never trust unvalidated text: a downstream system should reject anything that does not parse or fit the schema.
Why it is asked: many production bugs are malformed output. Read the full explanation: getting reliable JavaScript Object Notation (JSON) from a large language model (LLM).
What is prompt injection?
Prompt injection is text in untrusted input, such as a web page or an email, that contains instructions the model may follow instead of yours. Reduce the damage by limiting what tools the model can call, separating trusted instructions from untrusted data, and requiring approval for sensitive actions. No prompt alone is a complete defence.
Why it is asked: it tests security thinking for model applications. Read the full explanation: human in the loop for AI agents.
How do you version and test prompts?
Treat a prompt like code: keep it in version control, run it against a fixed set of example inputs with expected properties after each change, and compare results to the previous version before shipping. Record the model name and settings with each version, because a model update can change behavior with no prompt change.
Why it is asked: it separates demos from maintainable systems. Read the full explanation: prompting.
Retrieval and embeddings
What is retrieval-augmented generation?
Retrieval-augmented generation (RAG) retrieves relevant documents at question time and gives them to the model, so answers can be grounded and updated without retraining. The original paper paired a generator with a dense vector index of Wikipedia and reported state-of-the-art results on three open-domain question-answering tasks (Lewis et al., 2020).
Why it is asked: it is the most common applied-AI design question. Read the full explanation: embeddings and vector search.
How do you choose chunk size for a retrieval system?
Choose chunks that hold one self-contained idea: small enough that the retrieved text is relevant, large enough to keep its meaning. Start from the document structure, such as sections, add a small overlap, and measure retrieval quality on real questions instead of fixing a number in advance.
Why it is asked: it checks you tune by measurement. Read the full explanation: embeddings and vector search.
What is an embedding and how does vector search use it?
An embedding is a list of numbers that represents the meaning of a piece of text so that similar texts are close together. Vector search embeds the query, then returns the stored items whose vectors are nearest by a similarity measure such as cosine similarity, often using an approximate index for speed.
Why it is asked: it tests the base of every retrieval answer. Read the full explanation: embeddings and vector search.
Why add a reranker after retrieval?
A first-stage retriever is fast but coarse. A reranker, often a cross-encoder that reads the query and a candidate together, rescores the top results more accurately before they reach the model. It adds latency, so apply it to a short candidate list and measure whether the quality gain justifies the cost.
Why it is asked: it tests staged-retrieval design. Read the full explanation: cross-encoder reranker.
How do you debug a retrieval system that gives bad answers?
Separate retrieval from generation. First check whether the right passage was retrieved for failing questions; if not, fix chunking, embeddings or query handling. If it was retrieved but the answer is wrong, fix the prompt, the ordering of passages or the model. Measuring each stage prevents tuning the wrong one.
Why it is asked: debugging method matters more than the fix. Read the full explanation: embeddings and vector search.
Choosing an adaptation method
When do you use prompting, retrieval or fine-tuning?
Use prompting when instructions and examples are enough, retrieval when the model needs knowledge that changes or is private, and fine-tuning when you need a consistent style, format or behavior that prompting cannot reach. Try them in that order, because each step costs more to build and maintain.
Why it is asked: it tests judgment and cost awareness. Read the full explanation: prompting.
What is LoRA?
Low-rank adaptation (LoRA) freezes the pretrained weights and trains small low-rank matrices added to each transformer layer. The paper reports about 10,000 times fewer trainable parameters and about three times less graphics processing unit (GPU) memory than fully fine-tuning a 175-billion-parameter Generative Pre-trained Transformer (GPT) model, GPT-3, with Adam, with quality on par or better on the models tested (Hu et al., 2021).
Why it is asked: it tests whether you know efficient fine-tuning. Read the full explanation: linear algebra for machine learning, including low-rank matrices and LoRA.
When is the right answer not to use a language model?
When the task is deterministic, has a simple rule, needs exact arithmetic, or cannot tolerate occasional errors without a check. A rule, a database query or classical code is cheaper, faster and testable. Anthropic's guidance is to start with the simplest solution and add complexity only when it is needed (Anthropic, December 2024).
Why it is asked: restraint is a senior signal. Read the full explanation: AI agent architecture.
Evaluation
How do you evaluate a language-model application?
Build a test set of real inputs with expected properties, score outputs with code checks where possible and a rubric-based judge where not, and sample outputs for human review. Report results by category, not one average, and run the set on every change so regressions show up before users see them.
Why it is asked: evaluation is the hardest part of the job. Read the full explanation: LLM observability.
What are the pitfalls of using a model as a judge?
A judge model can favor longer answers, its own style or the first option shown, which the study of model judges by Zheng et al. calls verbosity, self-enhancement and position bias (Zheng et al., 2023), and it can be wrong with confidence. Use a clear rubric, calibrate against human labels on a sample, and keep humans testing. Anthropic describes using a rubric-based model judge with human testing alongside (Anthropic, June 2025).
Why it is asked: it tests skepticism about automated scoring. Read the full explanation: LLM observability.
How do you catch regressions after a model or prompt change?
Keep a fixed regression set that includes past failures, run it before and after every change, and compare per-category scores and a sample of raw outputs. Pin the model version so a provider update does not change behavior silently, and monitor live traffic for drift.
Why it is asked: it tests release discipline. Read the full explanation: machine learning operations (MLOps).
Agents and tools
What is an AI agent, and when should you build one?
An agent is a system where a model chooses its own next action, calls tools and loops until the task is done. Build one only when the steps cannot be written in advance; Anthropic recommends workflows for well-defined tasks and agents for open-ended problems, noting agents cost more and can compound errors (Anthropic, December 2024).
Why it is asked: it checks you can argue for the simpler design. Read the full explanation: AI agent architecture.
How does tool calling work?
You describe functions to the model with names and argument schemas; the model returns a structured request to call one; your code runs it and returns the result as input to the next model turn. The model never executes the function itself, which is what lets you validate arguments and add approvals.
Why it is asked: it is the mechanism under agent questions. Read the full explanation: tool use, function calling and the Model Context Protocol (MCP).
How do you keep an agent from causing harm?
Give it the fewest tools needed, validate every argument, cap steps and spend, require human approval for irreversible actions, and log every call so you can audit and replay. Layer several checks, because any single guardrail can fail, and review the logs regularly to see which tool calls the agent actually makes.
Why it is asked: it tests safety-by-design for actions. Read the full explanation: agentic AI interview questions.
Serving, cost and operations
How do you reduce the latency of a language-model feature?
Shorten the prompt, stream tokens so users see output sooner, use a smaller model for easy cases, cache repeated work, and run independent calls concurrently. Measure the slow step first, since time may be spent in retrieval or tools rather than in the model.
Why it is asked: it tests profiling before optimising. Read the full explanation: LLM inference.
How do you reduce the cost of a language-model feature?
Cut tokens: shorter prompts, only the retrieved passages that matter, and a cap on output length. Route simple requests to a cheaper model, cache repeated requests, and batch work that does not need an immediate reply. Track cost per successful task, not per call.
Why it is asked: it tests unit-economics thinking. Read the full explanation: LLM inference.
What would you monitor for a language-model feature in production?
Latency and error rates, token use and cost, the rate of failed or empty answers, tool-call failures, user feedback and quality scores on sampled traffic. Store traces of each request so a bad answer can be replayed. Alert on changes in these over time, not only on outages.
Why it is asked: it tests production ownership. Read the full explanation: LLM observability.
How would you design a question-answering system over company documents?
Clarify users, document types, freshness and access rules first. Then ingest and chunk the documents, embed and index them, retrieve and rerank for each question, generate an answer with citations, and evaluate on real questions. Enforce access control at retrieval time so users only see documents they may read.
Why it is asked: it is a common design prompt. Read the full explanation: machine learning system design.
Talking about your work
Walk me through a project you built.
Open with the problem and who it was for, then your specific part, the key decision and the alternative you rejected, the result with a number if you have one, and what you would change. Keep it to about two minutes and be ready for deeper questions on every claim.
Why it is asked: project deep dives are in most loops. Read the full explanation: tell me about yourself and behavioral interviews.
Follow-up questions
How do I prepare for an AI engineer interview?
Build one small end-to-end project, such as a retrieval question-answering app with an evaluation set, so you have real stories. Review the fundamentals on this page and practise explaining each aloud. Ask the recruiter which round types your loop includes; interview loops shows what each round tests.
How do I become an AI prompt engineer?
Learn software engineering first, then the language-model basics above, then build and evaluate small applications and publish what you learn. Prompt writing alone is a narrow base, so show retrieval, evaluation and deployment work as well. See prompting and AI agent architecture.
What are your salary expectations?
Recruiters often ask this early. Pay varies by country, company, level and equity, so use dated, current sources for your market, compare total compensation rather than base alone, and prepare your reasoning before the offer arrives. Salary negotiation covers how to answer.
- Answer in the order the topics build: basics, prompting, retrieval, adaptation, evaluation, agents, serving.
- Argue for the simplest design that works, and name how it fails.
- Measure each stage separately: retrieval, generation, tools and cost.
- Prepare one real project end to end; it supplies the stories for every round.
Sources
- Attention Is All You Need (arXiv 1706.03762 v7) — arXiv (Vaswani et al.), 2 Aug 2023
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401 v4) — arXiv (Lewis et al.), 12 Apr 2021
- LoRA: Low-Rank Adaptation of Large Language Models (arXiv 2106.09685 v2) — arXiv (Hu et al.), 16 Oct 2021
- Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172 v3) — arXiv (Liu et al.), 20 Nov 2023
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685 v4) — arXiv (Zheng et al.), 24 Dec 2023
- Building effective agents — Anthropic, 19 Dec 2024
- How we built our multi-agent research system — Anthropic, 13 Jun 2025
