How do you design a batch inference pipeline for an LLM?

In this article

1. Before you start
2. Why interviewers ask this
3. Batch inference pipeline design from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one part of the LLM inference optimization guide, and it builds on the transformer architecture.

Why interviewers ask this

A prompt such as "run a model over millions of documents" can appear in a machine learning design round, and a batch pipeline is the shape of its answer. The consequence of a weak design is concrete: a job that ends with some answers missing and no way to tell which ones.

A weak answer says "put the documents in a queue and call the model in a loop." A strong answer separates the work from the waiting. It splits the job to fit limits, tags every request, expects some requests to fail, and reruns only those. A Staff-level answer adds cost and deadline reasoning: what the discount is worth, how the 24-hour window shapes the schedule, and when your own GPUs beat a provider. The billion-document batch inference design takes the same ideas to extreme scale.

Batch inference pipeline design from first principles

Batch job

A batch job is a group of independent model requests that a provider or an engine accepts as one submission, runs asynchronously and returns as one set of results, so no caller waits on any single answer and the provider can price the work lower.

Request id

A request id is a caller-chosen label attached to each request in a batch job that comes back unchanged with its result, so the pipeline can join every result to its input and find the requests that need a retry, whatever order results arrive in.

In plain words

Think of sending a large photo order to a print shop. You do not stand at the counter while each photo prints. You hand over a box, get a receipt, and come back later. The shop prints the box when its machines are quiet, so it charges less. The analogy breaks in one place: a print shop returns the whole box, while a batch job can return most photos and a list of the ones that failed.

In one precise sentence, a batch pipeline submits independent requests asynchronously, joins results to inputs by id, and reruns only the failures.

Providers sell this as a deal. OpenAI describes its Batch API as asynchronous groups of requests with 50% lower costs, a separate pool of higher rate limits and a 24-hour turnaround. Anthropic charges 50 percent of standard prices for batch work.

Step 1: Shard the job into batches that fit the limits

A batch is limited to 100,000 requests or 256 MB, whichever comes first, according to the Anthropic batch documentation. A job larger than that is split into shards, and the pipeline submits each shard as its own batch job. Other providers set their own caps, so read the vendor's current page before you fix a shard size.

Input: the full list of requests. Output: shards, each with a recorded list of the ids it contains. The byte cap often binds before the request cap, because long prompts make large files; the worked example below shows a case where it does. Keep shards well inside the limits, because an oversized batch is refused as a whole.

You now know the pipeline's first move: cut the job to fit the provider. The next step makes the pieces trackable.

Step 2: Give every request an id and expect partial results

Each request carries a custom_id that joins its result to its input. The Anthropic documentation says each request is handled independently, and that custom_id is 1 to 64 characters from letters, digits, underscores and hyphens. Because requests are independent, a batch ends with a mix of outcomes, not a single pass or fail.

The documentation lists four result types: succeeded, errored, canceled and expired. Only succeeded requests are billed. Join on the id, never on line position, so the pipeline needs no assumption about the order of results.

Choose ids that you can rebuild from the input, such as a document id plus a prompt version. Then a retry run can ask "which ids have no succeeded result yet?" and the answer is a simple set difference. This is the bookkeeping most sketches leave out, and it is what turns a failure into a small retry instead of a full rerun.

Step 3: Submit asynchronously, poll and respect the deadline

The pipeline submits each shard, then polls its status until it ends. Batches expire if processing does not complete within 24 hours, results are available for 29 days after creation, and most batches finish within 1 hour, all per the Anthropic documentation. The same page warns that processing might be slowed down by demand, so more requests may expire. OpenAI states a 24-hour turnaround and says batches often finish more quickly.

Treat 24 hours as the deadline, not the expected time. Validation of request parameters is also asynchronous, so errors come back only when the whole batch has ended. Anthropic advises testing the request shape with the ordinary Messages API first. A batch result is one file, so stream: true is not supported.

Step 4: Collect the results and sort them by outcome

When a batch ends, the pipeline downloads the results, a JavaScript Object Notation (JSON) Lines file (.jsonl) with one result per line, and sorts each line by its result type. Succeeded results are validated against your own schema, then stored under their ids. Errored, canceled and expired ids go to a retry list.

Write results to durable storage as soon as you download them, because the provider keeps them for 29 days only (Anthropic documentation). Validate content, not just status: a succeeded request can still hold an answer your downstream code rejects, such as text that is not valid JSON when you asked for it.

Keep job state in a manifest with one row per shard: the shard number, its batch id, its status and the ids it holds. Write accepted results as upserts keyed on custom_id, so a retry that succeeds late overwrites the same row instead of creating a duplicate.

Input: one results file per shard. Output: a store of accepted results keyed by id, and a list of ids still missing. You now have the state a retry needs.

Step 5: Retry the failures and cap the attempts

The pipeline resubmits only the ids on the retry list as a new, smaller batch. Fix invalid-request errors before retrying, because the same input will fail again; server errors can be retried directly. Expired requests are usually retried unchanged, since they did not fail on their own merits.

Cap the number of rounds, for example two, and send what still fails to a dead-letter list for a person to inspect. Because failed requests are not billed, retrying costs nothing extra for the failed attempts. The cost is time: each round can take up to the full window, so a job that allows two rounds should budget for twice the window. The worked example computes that.

1 · Shard2 · Tag3 · Submit and poll4 · Collect5 · RetryReplay

Step-by-step description

1. Cut the job to fit the batch limits.
2. Give every request an id.
3. Run asynchronously inside the window.
4. Sort results by outcome.
5. Resubmit only the failed ids.

• A batch is limited to 100,000 requests or 256 MB, whichever comes first.
• Each request carries a custom_id that joins its result to its input.
• Only succeeded requests are billed.

Worked example

Take an illustrative job of 100,000 requests of 200 input tokens and 150 output tokens each. That is 20 million input tokens and 15 million output tokens.¹ As of October 2026, Anthropic lists batch prices per million tokens of 1 dollar input and 5 dollars output for Sonnet 5.5, and 2 and 10 dollars for Opus 5.5, and states that batch use is charged at 50 percent of standard prices (Anthropic documentation).

The Sonnet job costs 20 × 1 + 15 × 5 = 95.00 dollars in a batch against 190.00 dollars standard.¹ The Opus job costs 190.00 dollars against 380.00 dollars.¹ A batch costs half the standard price.

Text description of the diagram

• A batch costs half the standard price.
• The Sonnet job costs 95.00 dollars in a batch against 190.00 dollars standard.

Sharding: assume each request is 4 KB, an illustrative size, and take 256 MB as 1,048,576 bytes per megabyte (an assumption, since the documentation does not say; a decimal megabyte gives 62,500 requests and still 4 shards). One batch then holds 65,536 requests by size, less than the 100,000 request cap, so the byte cap binds. A job of 250,000 requests needs 4 shards, not the 3 the request cap alone suggests.¹

Retries: suppose, illustratively, that 1.5% of requests error and 0.5% expire. That leaves 2,000 ids for the retry batch.¹ The first pass bills 98,000 succeeded requests for 93.10 dollars, the retry bills 1.90 dollars, and the total is 95.00 dollars, the same as a clean run.¹ Two rounds at the 24-hour limit could take 48 hours in the worst case.¹

Trade-offs and edge cases

Provider batch, synchronous call or self-hosted batch?

A provider batch API returns results within a window instead of while you wait. A self-hosted batch job moves the cost into GPU hours and operations work. Synchronous calls suit users who are waiting; batch suits bulk work with a deadline, such as evaluations, moderation, data analysis and bulk generation.

Text description of the diagram

• A provider batch API returns results within a window instead of while you wait.
• A self-hosted batch job moves the cost into GPU hours and operations work.

Self-hosting is plausible when volume is steady. As an illustration, take 20,000 requests per GPU-hour and 2.00 dollars per GPU-hour, both assumptions to replace with your own measurements. The 100,000-request job then needs 5 GPU-hours and costs 10.00 dollars, against 95.00 dollars for the Sonnet 5.5 batch. The comparison flips if the GPU sits idle: below about 11% utilisation of the hours you pay for, the provider batch is cheaper.¹ The comparison also assumes the self-hosted model matches the quality you need, which you must test.

Two layers of batching

An API batch is a job-level submission of independent requests. Engine batching is how the inference engine groups sequences in each forward pass; the continuous batching page covers it. They are different layers. The vLLM paper (Kwon et al., 2023) says high-throughput serving needs batching sufficiently many requests at a time, and that wasted KV cache memory limits the batch size. Paged attention is its fix, and the paper reports 2–4× higher throughput at the same latency than FasterTransformer and Orca (Kwon et al., 2023). An inference engine uses both layers at once.

Running the engine yourself

vLLM can run a file in the OpenAI batch file format, one JSON request per line, with this command (vLLM documentation):

vllm run-batch -i <requests-file> -o <results-file> --model <model>

The vLLM page says it is a guide to the file format, not the complete Batch API in its representational state transfer (REST) form, so it is not a hosted batch service. For a cluster, Ray Data LLM wraps vLLM engine replicas in map_batches. Its quickstart uses batch_size=32 and concurrency=1, meaning one engine replica; those are configuration illustrations, not recommendations. Its max_model_len is a hard cap on prompt plus output tokens. The Ray documentation also says a failed job resumes from the last successful block, so a self-hosted retry works by block rather than by id.

Other edge cases

Shared prompt prefixes interact with prompt caching: Anthropic suggests the 1-hour cache duration for batches, because they can take longer than the 5-minute default. Batches can slightly exceed a workspace spend limit. Tool use, vision, multi-turn conversations and extended thinking are supported in a batch per the same documentation.

How to answer it in an interview

The spoken structure: say why batch, then shard, then ids, then partial results and retries, then the deadline and cost trade.

Common mistakes

• "A batch either fully succeeds or fully fails." (wrong) Requests end as succeeded, errored, canceled or expired. (right) "I expect partial results and retry only the failed ids."
• "A batch always takes 24 hours." (wrong) That is the expiry, and most batches finish sooner. (right) "I treat 24 hours as the deadline and budget a retry round beyond it."
• "Batch API and continuous batching are the same thing." (wrong) One is job submission, the other is engine scheduling. (right) "The API batch is the job level, the engine batches sequences inside each forward pass."
• "vLLM run-batch is the OpenAI Batch API." (wrong) It reads the same file format only. (right) "It is a tool I run myself, with the same input format."
• "I will join results by line number." (wrong) That breaks when results are partial or reordered. (right) "I join on the request id."

Follow-up questions

How do you handle a job bigger than one batch allows?

Split it into shards that fit the request and size limits, record which ids belong to each shard, and submit them as separate batches. Check the byte size, since it can bind before the request count does.

What do you do when some requests in a batch fail?

Sort results by outcome, keep the succeeded ones, fix invalid-request errors, and resubmit the errored and expired ids as a smaller batch. Cap the rounds and send repeat failures to a dead-letter list.

Is a batch API the same as continuous batching?

No. A batch API accepts a job of independent requests and returns results later. Continuous batching is how an engine schedules sequences on the GPU.

When would you self-host batch inference instead of using a provider batch API?

When volume is steady enough to keep GPUs busy, when you need a model the provider does not offer, or when data must stay in your environment. Compare against the provider price using your own measured throughput and utilisation.

Can a batch job use tools or multi-turn conversations?

Per the Anthropic documentation, yes: tool use, vision, multi-turn conversations and extended thinking are allowed in a batch. Streaming is not, because results come back as one file.

Sources

1. Batch processing (Message Batches API), Claude Platform Docs — Anthropic, 26 Oct 2026
2. Batch API guide — OpenAI, 26 Oct 2026
3. Working with LLMs (ray.data.llm) — Ray project documentation, 26 Oct 2026
4. OpenAI batch file format, vLLM documentation — vLLM project documentation, 26 Oct 2026
5. Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv 2309.06180) — arXiv (Kwon et al.), 12 Sep 2023