Model serving API take-home: how to build and explain it

In this article

1. Before you start
2. Why interviewers ask this
3. Model serving API take home from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the large language model (LLM) inference guide, which covers how language models are run in production.

Why interviewers ask this

A serving take home moves the question from "does the model work" to "does the service survive real traffic". The model itself is often a given, such as an open-weights language model or a small classifier. The assignment tests everything around it: how the service starts, what a request looks like, what happens when twenty requests arrive together, and what the caller sees when something fails.

A strong submission shows a clear request contract, a model that loads once, honest health checks, a bounded queue and a short note on what was not measured. A weak one loads the model inside the request handler, has no health endpoint and treats every error as a crash. A Staff-level submission adds the trade-offs in writing: which latency it gave up for throughput, and why. Take-home formats vary, so read the prompt first and treat this page as a map of what a serving service involves, not a prediction of a particular prompt.

Model serving API take home from first principles

Model server

A model server is a long-running program that loads a trained model into memory once and answers many requests over a network interface, so callers send inputs and receive outputs without running the model themselves.

Readiness probe

A readiness probe is a check that a platform runs against a container to decide whether it may receive traffic. A failing readiness probe removes the container from routing but does not restart it, which suits a model that is still loading.

Dynamic batching

Dynamic batching is a server feature that combines requests arriving close together into one batch before running the model, so the hardware does more work per pass. It raises throughput, and a waiting period adds latency to each request.

In plain words

A model server is like a restaurant kitchen. The chef is hired and ready before the doors open; that is loading the model. Orders arrive on tickets with a fixed format; that is the request contract. The cook may wait a moment to make several matching dishes together; that is batching. Each dish leaves the kitchen as soon as it is done; that is streaming.

The analogy breaks in one place. A real dish cannot be served half-made, but a language model server sends partial output on purpose.

In one precise sentence, a model serving API turns "run this model on this input" into an HTTP call that is safe to repeat, easy to monitor and cheap enough to run for many callers.

Step 1: Load the model once and report readiness honestly

A model server loads its weights once at startup, before the server reports ready, because loading is slow and paying it on every request would ruin latency. Keep the loaded model in a variable that every request handler reads.

Health checks come next. A liveness probe is a check that decides when to restart a container (Kubernetes v1.32 docs). A readiness probe decides whether it may accept traffic. Liveness probes do not wait for readiness probes to succeed (Kubernetes probes), so a slow model load can look like a dead container and be killed. If the model loads before the server starts accepting connections, the liveness endpoint cannot answer during the load, so a liveness probe with default settings fails three times and the kubelet restarts the container.

A startup probe is a check that holds back the other two probes until it succeeds, which gives the application time to finish initialising, and Kubernetes describes it as useful for containers that take a long time to come into service. Expose /health/live for the process, /health/ready that passes only after the weights are loaded, and cover the load with a startup probe. You now have a service that can start safely; next it needs a contract.

Step 2: Accept a typed, validated request

A model server defines its request as a typed schema and rejects bad input before it touches the model. In FastAPI, to declare a request body you use Pydantic models, and a parameter of that type is read as the body. A parameter of a singular type, such as an integer or a string, is read as a query parameter instead.

Use POST for inference. The FastAPI documentation says that to send data you should use one of POST, PUT, DELETE or PATCH, with POST the most common, so a GET carrying a body is the wrong tool. A typed request is validated before it reaches the model, and that check protects your expensive hardware from garbage.

Input: a JSON (JavaScript Object Notation) body with a prompt and a few limits such as a maximum number of new tokens. Output: a response with the generated text. Set a ceiling on input length and on tokens requested. If you serve through vLLM instead, its server follows the OpenAI request shape, covered in the trade-offs below.

Step 3: Batch requests in a short queue

A short batch queue raises throughput but adds waiting time. Dynamic batching combines requests on the server so a batch is created dynamically, and Triton's guide says creating a batch of requests typically results in increased throughput. The same guide says the dynamic batcher suits stateless models.

By default Triton's batcher creates batches as large as possible up to the maximum batch size and does not delay when forming them. To raise throughput further you can increase the maximum batch size or set a non-zero queue delay, which trades latency for throughput. In its optimization guide Triton calls dynamic batching, for most models, the feature that gives the largest performance improvement (optimization guide).

A generation request runs for many decoding steps and requests finish at different lengths, so a batch formed once per request would make short requests wait for the longest; language model servers therefore use continuous batching, which adds and removes requests at every decoding step. A take home still benefits from the idea: a bounded queue, a cap on wait time and a measured effect. The worked example below puts numbers on it.

Step 4: Stream tokens as they are produced

A language model server sends tokens as a stream of server-sent events, so the caller sees the first words early. The server answers with the content type text/event-stream, and Mozilla's web documentation says each notification is sent as a block of text terminated by a pair of newlines.

The HTML standard names the same type, and says the connection fails if the response status is not 200 or its content type is not text/event-stream. Also check that no proxy in front of your server buffers the response and holds the stream back.

Streaming does not need WebSockets. Server-sent events (SSE) is one-way, server to client, over plain HTTP, which is all token streaming needs. Input: one request. Output: many small events, then a final event that marks the end. Decide what the client sees if generation fails halfway: send a final error event rather than silently closing.

Step 5: Fail loudly and cleanly

A model server returns a clear error, not a hang, when it is overloaded or given bad input. Bound the queue so that new work is refused when it is full, set a timeout on each generation, and map each failure to an HTTP status the caller can act on. Use 422 for a body that fails validation, which is FastAPI's default; 429 Too Many Requests when the queue is full (Request for Comments (RFC) 6585 §4); 503 Service Unavailable while the model is not ready, optionally with a Retry-After header (RFC 9110 §15.6.4); and 504 Gateway Timeout when a generation timeout is exceeded behind a gateway (RFC 9110 §15.6.5).

Log one line per request with its queue wait, generation time and token count. Those numbers are the evidence for the trade-off note in your README. A reviewer reading it should see what you measured and what you only reasoned about.

1 · Load once2 · Typed request3 · Batch queue4 · Stream5 · Clean errorsReplay

Step-by-step description

1. Weights load once; ready comes after.
2. Input is validated first.
3. Short wait, more throughput.
4. Tokens leave as events.
5. Overload gets a clear status.

• The model loads once at startup, before the server reports ready.
• A typed request is validated before it reaches the model.
• A short batch queue raises throughput but adds waiting time.
• A model server sends tokens as a stream of server-sent events.

Worked example

The numbers below are illustrative, not measurements. Assume one forward pass over a batch of n requests takes 20 plus 2 times n milliseconds, a fixed cost plus a per-request cost, and that the queue delay is at most 5 milliseconds.¹

Throughput is 45.45 requests per second at batch 1 and 380.95 requests per second at batch 32. If 100 requests per second arrive, a server that runs one request at a time cannot keep up, because 45.45 is below 100. A batch of 4 gives 142.86 and keeps up.¹

The last column adds the 5 millisecond queue delay to the pass time, so latency rises with batch size even as throughput rises.¹ A request that arrives while a batch is already running also waits for that pass, so real tail latency is higher than this column. At 100 requests per second a batch of 32 would not fill inside the 5 millisecond window either, so the table shows what each batch size costs, not what a given arrival rate achieves.

Text description of the diagram

• Throughput is 45.45 requests per second at batch 1.
• Throughput is 380.95 requests per second at batch 32.

Now the startup budget. Kubernetes considers a probe failed after it fails undefined times in a row, and the default is 3. With a probe period of 10 seconds (illustrative), that default allows 30 seconds. If the model takes 120 seconds to load (illustrative) and the server only starts listening after the load, the container is restarted before it is ready. A startup probe with a threshold of 30 gives 300 seconds, which leaves 180 seconds of margin.¹

Text description of the diagram

• The default budget allows 30 seconds.
• The model takes 120 seconds to load.
• A startup probe with a threshold of 30 gives 300 seconds, which leaves 180 seconds of margin.

Trade-offs and edge cases

Write your own server or use an existing one. vLLM provides an HTTP server that implements the OpenAI Completions, Chat and Embeddings APIs (vLLM v0.8.3 documentation); newer versions list more endpoints, so read the current page. You start it with the vllm serve command or through Docker, and the project publishes an official image, undefined. If the take home allows it, say so and explain what your own thin layer adds. If it asks you to build the service, building it shows more.

OpenAI-compatible is not identical. vLLM supports parameters OpenAI does not, such as top_k, passed through the client's extra_body. By default it also applies the model repository's generation_config.json, so default sampling values can differ from what a caller expects.

Batching suits stateless work. An embedding model or classifier fits a dynamic batcher. A generation endpoint is usually served with continuous batching and a managed KV cache; paged attention is how vLLM stores it. If inputs differ in length, padding wastes compute on empty positions, so cap lengths or group similar ones.

Blocking calls and worker processes. FastAPI's concurrency page explains that a handler declared with async def runs on the event loop, so a blocking model call inside it stalls every other request; declare the handler with plain def, which FastAPI runs in a threadpool, or hand the call to a separate worker. Starting several server worker processes also loads one copy of the weights per process, so GPU memory use multiplies; keep one process per GPU unless you have measured otherwise.

Repeated prompts. When many callers send the same prefix, caching LLM calls can cut cost. Sharing one server between customers adds fairness and isolation problems, covered in multi-tenant model serving. For a hard latency target on a large model, see serving a 70B model under a latency budget, and for what the engine does under the API, see what an inference engine does.

Liveness failure restarts the container. Readiness failure removes the container from traffic. A startup probe holds back the other two probes until it succeeds.

How to answer it in an interview

The spoken structure: say how it starts, what a request looks like, how it batches, how it streams, then how it fails and what you did not measure.

Common mistakes

• "I load the model inside the request handler." (wrong) Every call pays the loading cost. (right) "I load once at startup and share the model across requests."
• "Liveness and readiness are the same check." (wrong) They do different things. (right) "Liveness restarts the container; readiness only removes it from routing."
• "Batching always lowers latency." (wrong) Dynamic batching raises throughput, and a non-zero queue delay adds latency per request. (right) "I batch for throughput and cap the wait to bound latency."
• "OpenAI-compatible means identical." (wrong) Extra parameters and sampling defaults differ. (right) "I check the server's own documentation for parameters and defaults."
• "I call the model inside an async def handler and run several worker processes." (wrong) A blocking call stalls the event loop, and each process loads its own copy of the weights. (right) "I use a plain def handler or a worker for the model call, and one process per GPU."
• "Streaming needs WebSockets." (wrong) Token streaming is one-way. (right) "Server-sent events over plain HTTP are enough."

Follow-up questions

How do you serve a machine learning model as an API?

Load the model once at startup, expose a POST endpoint with a typed request body, run the model, and return the output. Add health endpoints, a bounded queue and clear error statuses, then package it in a container.

What is the difference between liveness and readiness probes?

A liveness probe decides when to restart a container, and a readiness probe decides whether it should receive traffic. Liveness probes do not wait for readiness, so use a startup probe or an initial delay for a slow model load.

Does batching reduce latency?

No, it raises throughput. In the illustrative model above, throughput grows from 45.45 to 380.95 requests per second across batch sizes 1 to 32, while latency with the full queue wait grows from 27 to 89 milliseconds (a request arriving during a running batch waits longer still). Batching helps latency only indirectly, by keeping a queue from building up when one-at-a-time serving cannot keep up.

Do you need WebSockets to stream tokens?

No. Server-sent events carry text from server to client over an ordinary HTTP response with the content type text/event-stream, which is enough for generated tokens. Use WebSockets only if the client must also send messages on the same connection.

Is an OpenAI-compatible server the same as OpenAI's API?

No. It accepts the same style of request, but vLLM adds parameters such as top_k through extra_body and can take default sampling values from the model's generation_config.json. Test the exact behaviour you rely on.

Sources

1. OpenAI-Compatible Server (vLLM documentation, latest) — vLLM project, 26 Oct 2026
2. OpenAI-Compatible Server (vLLM documentation, v0.8.3) — vLLM project, 26 Oct 2026
3. Using Docker (vLLM documentation) — vLLM project, 26 Oct 2026
4. Request Body (FastAPI documentation) — FastAPI, 26 Oct 2026
5. Liveness, Readiness, and Startup Probes (Kubernetes documentation) — Kubernetes, 26 Oct 2026
6. Liveness, Readiness and Startup Probes (Kubernetes documentation, v1.32) — Kubernetes, 26 Oct 2026
7. Batchers: Dynamic Batcher (Triton Inference Server user guide) — NVIDIA, 26 Oct 2026
8. Optimization (Triton Inference Server user guide) — NVIDIA, 26 Oct 2026
9. Handling Errors (FastAPI documentation) — FastAPI, 26 Oct 2026
10. Concurrency and async / await (FastAPI documentation) — FastAPI, 26 Oct 2026
11. RFC 9110: HTTP Semantics — IETF, 6 Jun 2022
12. RFC 6585: Additional HTTP Status Codes — IETF, 1 Apr 2012
13. Using server-sent events (MDN Web Docs) — Mozilla, 26 Oct 2026
14. Server-sent events (HTML Living Standard) — WHATWG, 26 Oct 2026