In this article
Before you start
This page is one design in the generative AI system design interview guide. If you also need the retrieval side of long documents, read how to design retrieval-augmented generation over real documents next.
Why interviewers ask this
A document pipeline is where a language model meets messy, high-stakes input, so one prompt shows whether a candidate designs for failure. Candidates report prompts such as processing 10,000 user uploads a month (payslips, identity documents, references) while detecting inconsistencies and surviving provider downtime, extracting parties, term and liability caps from a 200-page master service agreement (MSA), or building a generalised electricity bill extractor.
Retrieval over the same files is a different design, covered in how to decide whether you need retrieval at all, and uploads that carry access rules also raise the questions in permission-aware retrieval. A weak answer is "send the PDF to the model and ask for JSON." A strong answer separates reading from extracting, fixes the output with a schema, and says how a wrong value is caught. A Staff-level answer adds the numbers: how errors compound across fields, how many documents reach a human, and what happens when the provider is down.
An LLM document-processing pipeline from first principles
- Extraction schema
- An extraction schema is a written contract that lists every field to pull from a document, with its type, whether it is required, and the source quote that must back it, so the model's output can be checked by code rather than by reading it.
- Validation layer
- A validation layer is the deterministic code that runs after extraction and tests each value for format, range, agreement with other fields and presence in the source text, so a wrong value is caught without asking the model to grade itself.
In plain words
Picture a mail room. A clerk first throws out unreadable envelopes. Then the clerk types the letters into a form. A supervisor checks that the totals add up. Only puzzling forms go to a manager. The analogy breaks in one place. A clerk who mistypes usually knows it. A language model can write a confident, well-formed wrong value. That is why the supervisor is code and not the clerk. A model asked to grade its own output tends to accept it, whereas a written rule, such as "the totals must add up", gives the same verdict every time and can be tested.
In one precise sentence, the pipeline turns an untrusted file into data a downstream system can trust, using a model only where reading meaning is needed and code everywhere else.
Step 1: Triage the file, then read its pages
Triage rejects files the pipeline cannot process before any model is called, and the read step turns each accepted page into text. Check the file type, page count and readability first; a blurry or cut-off scan gets a clear re-upload message instead of a bad guess. A portable document format (PDF) with a real text layer is read directly. A scanned page goes through OCR, which should also return a per-word confidence.
Input: an uploaded file. Output: page text plus the page images and OCR confidences, kept for later steps. Fix the image (rescale, straighten, remove noise) before blaming the model; Tesseract's documentation on improving output quality covers rescaling, binarisation, noise removal and deskewing. You now know where the pipeline's text comes from; the next step turns it into fields.
Step 2: Run the document extraction pipeline one chunk at a time
Extraction asks the model to fill the extraction schema for one chunk of the document at a time. A chunk is a slice of the document, about 20 pages cut at section boundaries, small enough to read reliably. A definitions clause that explains terms used elsewhere may need to travel with every chunk. Liu et al. found that performance is often highest when relevant information is at the start or end of the input and degrades in the middle, even for long-context models, so a big window does not remove the need to chunk.
A schema feature helps with the format: OpenAI documents that Structured Outputs ensures responses adhere to your JSON Schema, so no required key is omitted. It says nothing about whether a value is right. Ask the model to return, for each field, the value and the exact quote it came from. Input: one chunk and the schema. Output: field values with quotes and page numbers.
Step 3: Validate every value with code, not with the model's confidence
The validation layer tests each extracted value with deterministic code, because a model can return a well-formed wrong answer and a self-reported confidence is not a checked fact. Four checks do most of the work. Format and range: dates parse, amounts are positive. Presence: the quoted text appears in the page text, which exposes invented values. Cross-field: on a payslip, gross pay minus deductions equals net pay; on a contract, the end date is after the start date. OCR confidence: low-confidence words under a quote lower trust in that field.
Input: field values, quotes and OCR confidences. Output: each field marked passed or failed, with a reason. This is the stage that candidates skip, and the worked example shows why it matters. Next comes what to do with failures.
Step 4: Route doubt to a person and survive provider downtime
Routing sends failed fields to the review queue and keeps everything else flowing. The reviewer sees the page image with the quote highlighted, fixes the value, and the correction is stored as a labelled test case. Passed documents move straight to the downstream system.
Model calls fail too, so each file is a job in a queue, keyed by a hash of its content so a retry never creates a duplicate. Retry with a growing delay, then fall back to a second provider through a gateway; the LLM gateway interview page covers that layer. Input: failed fields and failed calls. Output: corrected data, or a queued job that finishes later.
Step-by-step description
- Bad files stop here; good pages become text.
- A model fills the schema one section at a time.
- Code checks every value.
- Doubtful fields go to a person; passed ones move on.
- Triage rejects bad files and the read step turns pages into text.
- A model fills the schema one section at a time.
- Code validates every extracted value.
- Doubtful fields go to a review queue.
- Passed documents move to the downstream system.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Take 10,000 uploads a month, the volume candidates report for this prompt. Spread over 30 days that is 333.33 a day and 13.89 an hour, a rate one small queue of workers handles.1 The load is low, so the design problem is correctness, not throughput.
Next, a 200-page MSA. At an assumed 500 tokens a page (illustrative), the document is about 100,000 tokens; in chunks of 20 pages (also a design choice, illustrative) that is 10 chunks of about 10,000 tokens each.1
Now the compounding. A document is fully right only if every field is, assuming errors are independent across fields. At an illustrative 95% accuracy per field, a 5-field document is fully right 0.7738 of the time and a 20-field document only 0.3585 of the time; at 99% per field, 20 fields are fully right 0.8179 of the time.1
Per-field accuracy that sounds high still leaves most large documents with an error: at 95% per field, about 64% of 20-field documents carry at least one wrong value.1 Without validation, those 64% reach the downstream system unflagged. The 15% review share in the table below is the share of documents whose fields fail validation (illustrative). The errors left after validation depend on how many wrong values the checks catch, which you measure on the logged corrections.
| Quantity | Value (computed from illustrative inputs) |
|---|---|
| Uploads per hour, 10,000 a month | 13.89 |
| Tokens in a 200-page MSA at 500 a page | 100,000 |
| Chunks of 20 pages | 10 |
| Documents fully right, 20 fields at 95% each | 0.3585 |
| Documents fully right, 20 fields at 99% each | 0.8179 |
| Documents routed to review at an illustrative 15% | 1,500 |
| Reviewer time at 3 minutes each | 75 hours a month |
| Downtime, one provider at 99.5% availability | 3.6 hours a month |
| Downtime, two independent providers | 0.018 hours a month |
For review load, 15% of 10,000 is 1,500 documents, and at 3 minutes each that is 4,500 minutes, or 75 hours.1 For availability, a provider that is up 99.5% of a 720-hour month is down 3.6 hours; two providers failing independently are both down about 0.018 hours, roughly one minute, though real outages are often correlated.1
Text description of the diagram
- At 0.95 per-field accuracy, 5 fields are all right 0.7738 of the time and 20 fields 0.3585.
- At 0.99 per-field accuracy, 20 fields are all right 0.8179 of the time.
Trade-offs and edge cases
The main trade is cost and latency against accuracy. Extracting from OCR text is cheap and checkable, because you can test that a quote exists in the text. Sending page images to a vision-capable model can read layouts and stamps that OCR garbles, but you lose that text check unless you also run OCR. Many designs use text first and images only for pages that fail validation.
Other cases: a fixed template per document type is accurate but breaks on a new format, so a generalised extractor, such as one for electricity bills from many providers, relies on a flexible schema and validation instead of layout rules. A high review rate protects accuracy but costs reviewer hours, so tune thresholds on the logged corrections. A language model should never be the only check on identity documents or payslips; mismatches between fields are evidence, but the decision stays with a person.
Privacy needs its own design, because uploads such as payslips and identity documents hold personal data. Encrypt files at rest and in transit. Set a retention limit and delete page images once review is done. Redact personal fields from logs, and from test cases built from reviewer corrections. Before sending any document to a model provider, check that provider's data-use and retention terms, and restrict reviewer access by role so each person sees only the queues they need. Before launch, build a labelled set of documents, score each field against the labels, and re-run that set on every prompt or model change, so a regression shows up before users see it.
Text description of the diagram
- A blurry scan is caught by a quality check and the user is asked to re-upload.
- A wrong value is caught by a rule or cross-check and goes to review.
- A model outage is handled by retry, fallback and a queue.
A blurry scan is caught by a quality check and the user is asked to re-upload. A wrong value is caught by a rule or cross-check and goes to review. A missing field is caught by the required key in the schema and the section is retried. A model outage is handled by retry, fallback and a queue. If extraction must call a tool mid-run, such as looking up a supplier ID, see agentic retrieval-augmented generation.
How to answer it in an interview
Design a pipeline that extracts data from uploaded documents with an LLM.
I would build five stages. First, triage: reject unreadable or unsupported files. Second, read the pages from the text layer or with optical character recognition. Third, have a model fill a fixed schema one section at a time, each value with its source quote. Fourth, validate with code: formats, cross-field sums, and a check that each quote exists in the page text. Fifth, send failures to a human review queue. Model calls go through a queue with retries and a fallback provider, so an outage delays work instead of losing it. I validate in code because per-field accuracy compounds: at an illustrative 95% per field, a 20-field document is fully right only about a third of the time.
The spoken structure: name the stages, say why extraction is chunked and schema-bound, then say how a wrong value is caught and how outages are survived.
Common mistakes
- "A bigger context window means I can send the whole contract." (wrong) Accuracy still depends on where the information sits. (right) "I chunk by section and merge the results."
- "Structured Outputs means the values are correct." (wrong) A schema ensures the shape, not the truth. (right) "The schema fixes the format; code validates the content."
- "I ask the model how confident it is and review the low ones." (wrong) A self-report is not a checked fact. (right) "I gate on validation results and OCR confidence."
- "If OCR is poor, I try a better model." (wrong) The input is the problem. (right) "I improve the image first, or ask for a re-upload."
Follow-up questions
How would you improve OCR accuracy on low-resolution scans?
Fix the input before the model: rescale the image, straighten it, remove noise (as Tesseract's guidance on improving output quality describes), and reject or request a re-upload when it stays unreadable. Keep OCR word confidences so the validation layer can distrust fields built on weak words.
How do you extract structured data from a 200-page master service agreement?
Split it into chunks by section, extract each field with its quote and page number, then merge, preferring the section where a field normally lives (parties near the start, governing law near the end). Validate the merged record, and send conflicts to review.
How do you handle LLM provider downtime in a document pipeline?
Treat each file as a queued job with a content-hash key, retry with a growing delay, and fall back to a second provider through a gateway. Files wait in the queue, so an outage delays results rather than losing uploads.
How would you design a generalised electricity bill extractor?
Use one flexible schema (supplier, account, period, usage, total) instead of per-supplier templates, and rely on cross-field checks such as the line items adding up to the total. New suppliers then fail validation loudly, and those failures become test cases.
How do you detect inconsistencies in uploaded payslips and identity documents?
Run cross-field rules (gross minus deductions equals net, dates in order, names matching across documents) and flag mismatches for a person. The model extracts the values; the code finds the disagreement.
How would you design a redlining engine for two contract drafts?
Align the two drafts clause by clause, diff the aligned text, then ask a model to describe the meaning of each change in plain language, with each description tied to the changed quote. A lawyer reviews the output, so show the original text beside every summary.
- Separate reading from extracting, and extract section by section into a fixed schema.
- A schema ensures the shape of the output, not the truth of the values.
- Validate with code: format, presence of the quote, cross-field checks and OCR confidence.
- Per-field accuracy compounds, so large documents need review routing.
- Queue every job and add a fallback provider so an outage delays work instead of losing it.
Sources
- Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172) — arXiv (Liu et al.), 6 Jul 2023
- Structured model outputs - OpenAI API documentation — OpenAI (read 2026-10-26), 26 Oct 2026
- Improving the quality of the output - Tesseract OCR documentation — Tesseract OCR project (read 2026-10-26), 26 Oct 2026
