In this article
Before you start
This page belongs to the large language model (LLM) inference optimization guide, and it builds on the transformer architecture.
Why interviewers ask this
Candidates report prompts of this kind in interviews at AI labs and large tech companies: run a language model over a billion documents as cheaply as possible, or build the scheduler behind a text-to-video service that accepts prompts, runs GPU jobs and reports status. The two prompts share one skeleton, so a single answer covers both.
The cost of getting it wrong is concrete: a design that treats a billion documents like a live request loses hours of paid GPU work every time one machine fails. A weak answer names a big model and a big cluster. A strong answer starts from the unit of work, the shard, and says what happens when it fails. A Staff-level answer also prices both engines in GPU-hours, names the lever that matters most for cost (batch shape and interruptible hardware), and says how a user learns that a job finished.
Billion-document batch inference from first principles
- Batch inference
- Batch inference is a way of running a trained model over a large, fixed set of inputs with no user waiting, so the system may delay, reorder and group requests to maximise work per GPU-hour instead of minimising the time of each answer.
- Idempotent job
- An idempotent job is a unit of work that produces exactly one stored result however many times it runs, usually because its output is written under a name derived from its job id.
In plain words
Picture a mailroom that must stamp a billion letters. Nobody is waiting for one letter. So the manager boxes the letters into trays and hands trays to clerks. The manager notes which trays are done. If a clerk goes home sick, only that tray is redone. The analogy breaks in one place. Here the clerks are expensive GPUs. Idle or wasted time is what you pay for.
In one precise sentence, the system maximises tokens processed per GPU-hour. Every shard stays independent. A failure costs one shard and never the run.
Step 1: Shard the corpus into small independent jobs
The corpus is cut into shards and each shard gets a stable shard id. Input: a billion documents. Output: a list of shard files, each with an id, a document count and a checksum.
JavaScript Object Notation (JSON) Lines is a text format that stores one JavaScript Object Notation (JSON) record per line, and it is called JSON Lines (JSONL) for short. Hosted batch interfaces take a JSON Lines (JSONL) file, one request per line, and two limits apply to each file. OpenAI's guide is restated as allowing up to 50,000 requests per batch and 200 MB per input file. A file over the request limit is rejected during validation and no request in it runs. Size each shard by whichever limit binds first, request count or bytes.
Take 500 input tokens per document, about 2,000 bytes at four bytes per token plus 500 bytes of prompt and JSON overhead (all illustrative). A 50,000-line file is then 125 MB, so the request cap binds, and a billion documents become 20,000 shards of 50,000 documents each.1 Documents of 5,000 tokens take about 20,500 bytes per line, so the 200 MB cap binds first: 9,756 documents per file and 102,502 shards.1 Inside a shard, sort documents by length, which Step 2 shows to be a real saving. You now know the unit of work: a shard that can be run, retried and counted alone.
Step 2: Pick the engine, hosted batch files or your own GPUs
Two engines can run a shard. A hosted batch interface takes the file, runs it within a completion window, and returns an output file and an error file. OpenAI's guide documents a 24-hour completion window and a 50% cost discount. The discount is a latency trade, not a model-quality one. Limits can change, so read the guide before you commit.
Hosted cost is priced per token, and input and output tokens have different prices: hosted cost = 0.5 × (input price × input tokens + output price × output tokens). No dollar figures appear here because prices change. Take 500 input tokens and 20 output tokens per document, as in a classification task (illustrative).
Submission has two costs. The restated guide documents a limit of 2,000 batches created per hour, so creating 20,000 batches takes at least 10 hours.1 The tighter limit is the enqueued-token cap. One secondary page says top-tier accounts can enqueue 200,000,000 or more tokens across pending batches, and lower tiers get less. A 50,000-document shard holds 25,000,000 input tokens, so eight shards fill the queue. The 500,000,000,000 input tokens need 2,500 full queue refills, and each refill can take up to the 24-hour window.1 One organisation cannot clear a billion documents in days this way. At this scale hosted batch needs a raised limit or several accounts, or the work moves to your own GPUs.
The alternative is your own GPUs running an open model. The two phases cost differently. Prefill, reading the input, is compute-bound and fast. Decode, writing the output one token at a time, is memory-bound and dominates GPU time when outputs are long. Take 10,000 input tokens per second and 1,000 output tokens per second per GPU (both illustrative). One document costs 50 milliseconds of prefill and 20 milliseconds of decode.
Decode is only 20 of 520 tokens but 28.57% of GPU time. A billion documents are 19,444.44 GPU-hours, or 19.44 hours on 1,000 GPUs.1 Multiply the GPU-hours by your hourly GPU price and compare the result with the hosted formula. That comparison, not the model name, decides the engine. The throughput depends on continuous batching, paged attention and quantization, which are covered in the inference engine overview.
Length sorting is the cheapest lever. Take four documents of 100, 900, 120 and 880 tokens (illustrative), run in two batches of two. In arrival order each batch pads to its longest document. Sorting by length raises useful tokens from 56.18% to 98.04% of padded slots.1 Engines with continuous batching waste less on padding, but similar lengths still help memory planning.
Step 3: Make every shard retryable and idempotent
Workers pull shard ids from a queue. A queue delivers a job at least once in most designs, so the same shard can run twice. Input: a shard id. Output: a result stored under a name built from that id.
Because the output name comes from the shard id, a second run replaces the first under the same name, so exactly one result per shard exists; that is what makes the job idempotent. The text may differ slightly, because sampling and batched GPU arithmetic are not bit-reproducible. So write once with an atomic rename (write to a temporary name, then rename in one step) or a conditional put (first writer wins), and never append.
The worker marks a shard done only after the output is written and its row count matches the input. A scheduler records state per shard (waiting, running, done, failed), re-queues a running shard whose worker stops reporting, and parks a shard that fails several times in a dead-letter list for a human to inspect.
Step 4: Run on cheap, interruptible GPUs
Cloud providers sell spare GPU capacity at a discount on the condition that they can reclaim it, usually with a short warning; check your provider's current terms for the notice and discount. Batch work suits this because a shard is independent. Input: an interruptible machine and a shard. Output: either a finished shard or a checkpoint.
A checkpoint is a saved record of how far a shard got, such as the last document written, so a restarted worker resumes there. Take shards of 50,000 documents at the illustrative 0.07 GPU-seconds per document: one shard takes 58.33 minutes. With no checkpoint, an interruption at a random time loses 29.17 minutes of work on average; checkpointing every 5,000 documents cuts that to 2.92 minutes.1 Keep the model weights on fast local storage so a replacement worker starts quickly. The engine also needs the KV cache and weights to fit in GPU memory, which caps the usable batch size, as the PagedAttention paper discusses.
Step 5: Merge, verify and report status
When every shard is done, a merge step concatenates outputs, checks that the document count equals the input count, and reports failures. Input: 20,000 shard outputs.1 Output: one verified dataset and a report.
Status is the async part: the caller gets a job id at once and learns the outcome by polling or by a webhook, a message your system sends to a web address the caller registered. The next section covers the details. A status of accepted or in progress only means the job exists; count a shard as done when its output is verified. Report progress as shards done out of shards total.
Async GPU jobs: ids, states, results and webhooks
A submit call returns a job id at once, and the client sends an idempotency key, a unique string it makes per request, so a client retry returns the same job instead of creating a duplicate. A job moves through five states: queued, running, succeeded, failed and cancelled. Only the scheduler changes state, and each change is recorded. Large results such as videos are written to object storage, a file store such as a cloud bucket, and the status endpoint returns a link, not the bytes.
Webhook payloads carry a signature the receiver checks, and failed deliveries are retried with growing delays. Cancelling marks a queued job cancelled at once and tells a running worker to stop at its next checkpoint. For a text-to-video service, give long video jobs and short ones separate GPU pools or priorities, so one long job cannot starve the queue.
Step-by-step description
- Cut the corpus into independent shards.
- Price hosted batch against your GPUs.
- Workers pull shards from a queue.
- Checkpoints limit interruption loss.
- Verify counts and report status.
- A billion documents become 20,000 shards of 50,000 documents each.
- Only the unfinished shard is re-queued, and its output name makes the rerun safe.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
A billion documents, shards of 50,000, 500 input and 20 output tokens per document, and 10,000 input and 1,000 output tokens per second per GPU (all but the document count illustrative) give the numbers below.
| Quantity | Value (computed or illustrative) |
|---|---|
| Documents | 1,000,000,000 |
| Documents per shard (the documented batch limit) | 50,000 |
| Shards needed | 20,000 |
| Hours to submit at 2,000 batches per hour | 10 |
| Input tokens per document (illustrative) | 500 |
| Output tokens per document (illustrative) | 20 |
| Total input tokens | 500,000,000,000 |
| Total output tokens | 20,000,000,000 |
| Enqueued-token cap, top tier (secondary source) | 200,000,000 |
| Full queue refills needed | 2,500 |
| Shards needed if documents are 5,000 tokens | 102,502 |
| GPU-hours at 10,000 input and 1,000 output tokens per second (illustrative) | 19,444.44 |
| Share of GPU time spent in decode | 28.57% |
| Wall-clock on 1,000 GPUs | 19.44 hours |
| Time to run one shard | 58.33 minutes |
| Expected loss per interruption, no checkpoint | 29.17 minutes |
| Expected loss, checkpoint every 5,000 documents | 2.92 minutes |
| Useful share of padded slots, arrival order (illustrative lengths) | 56.18% |
| Useful share of padded slots, sorted by length | 98.04% |
The expected loss assumes the interruption lands at a uniformly random moment, so on average half a unit of work is lost. Sorting by length raises useful tokens from 56.18% to 98.04% of padded slots, so the same hardware does about 1.75 times the useful work on that toy batch.1
Text description of the diagram
- Sorting by length raises useful tokens from 56.18% to 98.04% of padded slots.
- Each batch pads to its longest document.
Trade-offs and edge cases
Hosted batch removes the GPU work and keeps the provider's limits. Your own GPUs remove the limits and add engineering: you own retries, weights, drivers and the scheduler. The comparison below sums it up.
Text description of the diagram
- Hosted batch removes the GPU work and keeps the provider's limits.
- Your own GPUs remove the limits and add engineering: you own retries, weights, drivers and the scheduler.
Edge cases decide real runs. Duplicate documents waste money, so deduplicate by a content hash, a short fingerprint computed from a document's text so identical texts match, before sharding. A shared instruction at the start of every prompt can be reused through prompt caching, and a smaller model or a cheap first-pass model that sends only hard cases to a larger one lowers cost further. A poison document that crashes the worker must be isolated, not retried forever. Output order does not matter if each result carries its document id. A bigger batch does not always raise throughput, because KV-cache memory caps it. Finally, interruptible machines can vanish in bulk, so keep the queue state outside the workers.
How to answer it in an interview
Design a system to run LLM inference over a billion documents, as cheaply as possible.
I would treat it as a batch pipeline, not a service. First I shard the corpus into files under the request and byte limits, about fifty thousand documents each, sorted by length. Then I pick the engine by GPU-hours against the provider's token bill. Hosted batch has a queue cap that a billion documents will hit, so I would likely use an open model on my own GPUs with continuous batching. Workers pull shards from a queue and write output under a name built from the shard id, so retries are safe. They checkpoint every few thousand documents so interruptions of cheap GPUs cost minutes. A merge step checks counts, and a status endpoint reports shards done.
The spoken structure: name the unit of work, price both engines in GPU-hours, make retries safe, use interruptible hardware, then say how status is reported.
Common mistakes
- "I will send the billion documents as one batch." (wrong) A file over the documented 50,000-request or 200 MB limit is rejected whole. (right) "I shard into files under both limits and track each shard."
- "The batch discount means a weaker model." (wrong) The discount trades latency, not quality. (right) "Same model, results within a completion window, at a lower price."
- "If a worker dies I restart the run." (wrong) That repeats every finished shard. (right) "Only the unfinished shard is re-queued, and its output name makes the rerun safe."
- "More GPUs always finish sooner." (wrong) Queue limits, input reading and the merge step can bind first. (right) "I find the bottleneck, then add capacity there."
Follow-up questions
What if the documents average 5,000 tokens?
The byte limit binds before the request limit. At about 20,500 bytes per line a 200 MB file holds 9,756 documents, so the billion become 102,502 shards and submission alone takes about 51 hours.1 Prefill time per document also grows, so recompute the GPU-hours before choosing the engine.
How do you design a job scheduler for GPU video generation with status updates?
Accept the prompt, store a job record with a job id, and put the id on a queue. Workers with free GPUs pull jobs, update the state, and store the result. The caller polls the status or receives a webhook, and long jobs report progress.
How do you stop a retried job from writing duplicate output?
Name the output after the job id and write it atomically, with a temporary name and one rename, so a second run replaces the first and exactly one result exists. The text may differ between runs, so first writer wins. Mark the job done only after the write is verified.
How do you cut the cost of a batch run further?
Deduplicate inputs, reuse shared prompt prefixes through caching, quantize the model, and send easy documents to a smaller model first.
What happens when a worker dies halfway through a job?
The scheduler notices missing progress reports, re-queues the shard, and a new worker resumes from the last checkpoint. See GPU memory for why a restart must reload weights.
- The unit of work is the shard, sized by whichever binds first, request count or bytes.
- Compare hosted batch and your own GPUs in GPU-hours against the token bill, not by model name.
- At a billion documents the hosted queue cap, not submission speed, is the likely bottleneck.
- Idempotent output names give exactly one result per shard and make at-least-once queues safe.
- Interruptible GPUs are cheap, and checkpoints keep interruptions to minutes.
- Async status means a job id plus polling or a webhook, never "accepted" treated as done.
Sources
- OpenAI Batch API guide — OpenAI (vendor documentation; read 2026-10-26), 26 Oct 2026
- Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv 2309.06180) — arXiv (Kwon et al.), 12 Sep 2023
- OpenAI Batch API reference: how asynchronous jobs work — eesel.ai (secondary, undated; the date is the access date), 26 Oct 2026
- OpenAI Batch API Limits: Quotas, File Sizes, and Tokens — fast.io (secondary, undated; the date is the access date), 26 Oct 2026
