How do you design RAG over real documents with tables and charts?

In this article

1. Before you start
2. Why interviewers ask this
3. RAG over real documents from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is part of the GenAI system design guide, which covers the end-to-end designs interviewers ask for.

Why interviewers ask this

Tutorial RAG works on clean text, and real company data does not look like that. Invoices are scans, reports hold tables and stock charts, and a 200-page PDF mixes all of them. Interviewers ask this to see whether a candidate can name what breaks when the input is a real document: reading order, errors from optical character recognition (OCR), split tables and values that only exist inside a picture.

The question arrives as "describe a RAG pipeline; what do you do if the PDF has images", "is extracting with OCR enough", or "how would you use a vision-language model for chart-heavy reports". A weak answer says "chunk it and embed it". A strong answer describes the parse, the chunk shapes per element, the index and the citation, and sizes the cost.

RAG over real documents from first principles

In plain words

Layout analysis

Layout analysis is a step that detects the regions of a page, such as headings, paragraphs, tables and figures, and their reading order, so that text is read in the order a person would read it.

Structure-aware chunking

Structure-aware chunking is a way of splitting a document into retrievable pieces along its own structure, keeping a table whole with its header and a figure with its caption, instead of cutting at a fixed number of tokens.

Multi-vector page embedding

A multi-vector page embedding is a representation that stores one vector per image patch of a page and matches a query by comparing each query term with its best patch, a method called late interaction.

Picture an assistant who cannot see pictures. You hand over a book. A typed copy turns each table into a pile of numbers. It drops the charts. A careful assistant first sorts the pages into paragraphs, tables and figures. Each gets a label. Each chart gets a short description. The analogy breaks in one place. A person sees the layout at a glance. Software must be taught to find it, and its mistakes must be measured.

In one precise sentence, document RAG is a parse of each page into typed elements, a chunking and index that preserve them, and a retrieval and answer step that cites the page.

Step 1: Parse each page into typed elements, not a stream of text

The parse turns each page into a list of elements with a type, a position and an order, and everything later depends on it. The Docling technical report describes an open-source converter that turns PDFs into JavaScript Object Notation (JSON) or Markdown, understands page layout and reading order, locates figures, recovers table structure and optionally applies OCR to scanned pages, using models for layout analysis and table structure.

Scans are where quality drops, because a scan needs OCR and OCR errors reach the answer. The OCR Hinders RAG paper built a benchmark of 8,561 document images and found that none of the OCR solutions it evaluated was adequate for building high-quality RAG knowledge bases, and that both semantic noise and formatting noise reduce RAG performance, with more noise giving a larger drop. So the design measures parse quality, for example by sampling pages and comparing, instead of trusting it.

The input is a PDF or image pages. The output is typed elements, such as paragraph, table, figure and heading, in reading order, each tagged with its page number.

Step 2: Build chunks that keep the structure

Each element type gets its own chunk shape. Paragraphs are split to a size such as 512 tokens with a small overlap, which is an illustrative choice. A table is kept as one unit with its header, or split by rows with the header repeated, so that no row loses its column names. A figure becomes a chunk of its caption, its nearby text and, for a chart, a description written by a vision-language model (VLM) that includes the axis names and the key values.

Fixed-size splitting damages tables. In our illustration, a 40-row table with a 30-token header and 25 tokens per row does not fit in one 512-token chunk, so cutting at 512 leaves 21 rows in later chunks with no header¹. A note such as "all amounts in thousands" belongs in the header of every chunk of its table, because a row cut off from it is wrong by a factor of a thousand. The cost of captioning is real too: the ColPali paper notes that captioning-based approaches can take dozens of seconds per page.

The input is the typed elements. The output is chunks, each with a type, the page number and the document id.

Step 3: Index with metadata, or embed the page image

The index stores a vector per chunk plus metadata, and it pairs with keyword search. The page number, element type and document id let the answer cite its source and let a query filter by type, and a hybrid search index combines vector and keyword matches so that exact figures and names are found.

An alternative skips the parse. ColPali (Faysse et al., 2025) embeds page images directly with a vision-language model, producing one vector per image patch with late interaction matching, and reports outperforming modern document retrieval pipelines while being simpler and faster to index. The price is storage, because the paper stores a vector per patch: 1,024 patches plus 6 text tokens, each of 128 dimensions, which it reports as 257.5 KB per page¹.

The input is the chunks, or the page images. The output is a searchable index in which every entry points back to a page.

Step 4: Retrieve, rerank, and answer with the page cited

At query time the system retrieves candidate chunks, reranks them, puts the best few into the prompt and answers with a citation to the page. A cross-encoder reranker re-scores the candidates more carefully than the first-stage index. Order matters inside the prompt: Liu et al. (2023) found that model performance is often highest when relevant information is at the beginning or end of the input and drops when it is in the middle, which is related to the context rot problem.

For a chart question, pass the page image itself to a vision-language model along with the question, since the values live in pixels. Show the cited page next to the answer, so a person can verify it.

1 · Parse2 · Chunk3 · Index4 · AnswerReplay

Step-by-step description

1. Pages become typed elements.
2. Tables and figures stay whole.
3. Vectors, keywords and page metadata.
4. Retrieve, rerank, cite the page.

• The parse produces typed elements such as paragraphs, tables and figures.
• Answers cite the page that the retrieved chunk came from.

Worked example

The sizes below follow from one illustrative document: 200 pages with 650 text tokens a page, which is 130,000 tokens in all¹. With 512-token chunks and 64 tokens of overlap, the chunk stride is 448 tokens and the document yields 291 chunks¹.

The text-chunk dimension of 1,024 is an illustrative choice. The page-image row uses the ColPali figures: 1,030 vectors of 128 dimensions in two bytes each is 263,680 bytes, or 257.5 KB per page, which matches the paper¹. The two rows differ in precision as well as in design: the text vectors are fp32 (4 bytes a value) and the page vectors are fp16 (2 bytes a value), as ColPali reports. At equal precision the text index would be 0.60 MB, and the page-image index is about 88 times larger¹. For a corpus of 10,000 such documents the text index is about 11.9 GB and the page-image index is about 527 GB in total¹.

Text description of the diagram

• Text chunk vectors for one 200-page document need 1.19 MB in fp32 and 0.60 MB in fp16.
• Page-image vectors for the same document need 52.7 MB in fp16, about 44 times the fp32 text index and about 88 times the fp16 one.

The prompt sees only about two percent of the document, which is why retrieval quality, not model size, decides the answer. The ratio is also not a floor to design around. ColPali Appendix B.3 says centroid pooling or binary quantization can cut storage by about two orders of magnitude with minimal performance hits, and its Table 4 puts ColPali at 257.5 KB per page against 8.60 KB for a text-embedding model on the DocVQA test set, about 30 times, a measured figure on a different corpus than our illustration.

Trade-offs and edge cases

The central choice is parse-then-embed versus embed-the-page. Parsing gives compact, searchable text and exact numbers, and fails when the parse is wrong: the OCR Hinders RAG paper finds that OCR noise lowers RAG performance, and Docling is the open-source route to a layout-aware parse. Page-image embeddings avoid the brittle parse and see layout and charts. ColPali Table 5 measures indexing at 7.22 seconds a page for the Unstructured pipeline with layout detection, OCR and captioning, against 0.39 of a second for ColPali page encoding, and ColPali costs much more storage.

For the state of the field in 2025, the ViDoRe V2 paper (May 2025) reports that the first ViDoRe benchmark was approaching saturation, with top models above 90 percent nDCG@5, and adds harder queries and multilingual datasets, so visual retrieval is still being measured and improved.

Text description of the diagram

• A scan needs OCR, and OCR errors reach the answer.
• A table split across chunks loses its header.

A common design is hybrid: parse text and tables for the bulk of the index, and add page-image or chart descriptions for pages that the parse marks as figure-heavy. The results depend on the document type, so evaluate on your own questions. An evaluation set of real questions with the right page marked, scored with a tool such as Ragas, is the way to decide.

Edge cases to name: scanned documents with low-quality OCR need a quality check and a fallback; tables that span pages need stitching; a chart's exact values may not be readable from pixels, so ask for the source data when it exists; access rules must follow the document into the index; and multi-hop questions across documents may need a graph-based approach. A weak parse also means silent failure, so log the element counts per page.

How to answer it in an interview

The spoken structure is: parse, chunk by element, index with metadata, retrieve and cite, then measure.

Common mistakes

• "I extract text with a PDF library and embed it" (wrong). That loses reading order, table structure and charts (right: parse into typed elements first).
• "OCR output is ground truth" (wrong). The OCR Hinders RAG benchmark found none of the evaluated OCR solutions adequate for high-quality knowledge bases (right: sample and measure parse quality).
• "Split every document at 512 tokens" (wrong). A table cut across chunks loses its header (right: keep tables whole or repeat the header).
• "Put the retrieved chunks in any order" (wrong). Models do worse with relevant information in the middle of the input (right: put the best evidence first or last).
• "Embedding page images is always better" (wrong). It costs about 44 times the bytes of the text index as stored in our illustration, and about 88 times at equal precision (right: justify it by the document type, consider pooling or quantization, and measure).
• "The note says amounts in thousands, so the chunk can drop it" (wrong). A row without its scale note is off by a factor of a thousand (right: carry the unit line into every chunk of the table).

Follow-up questions

Design a RAG for PDFs with line charts and stock charts, about 200 pages each. How would you use a VLM?

Turn each chart into a small table at index time. A vision model reads the chart and returns rows of (date, series, value), plus the axis names and units, and these rows are stored as a chunk with the page number. At query time, retrieve by series name and date range. Take "the Apple (AAPL) share price on 11 January", when only a chart holds it: the answer is the value from the extracted row, stated as approximate because it was read from pixels, with the page cited. Where the source data exists, link it, and prefer it. Then send the cited page image and the question to the vision model as a check.

If you extract only with OCR, is there an issue, and what solutions are there?

Yes. The OCR Hinders RAG paper reports that OCR noise lowers RAG performance and that none of the evaluated solutions was adequate. Mitigate by using layout-aware parsing for digital PDFs, running OCR only on scans, checking quality on samples, and keeping the page image for a vision model to read.

A current option is olmOCR (Allen Institute for AI, February 2025), an open-source toolkit that uses a vision-language model trained for the task to turn PDFs into plain text in reading order, and whose abstract puts the cost at 176 USD per million pages. When OCR and the vision model disagree on a value, the page image is the tiebreaker, and the case is logged so the parse can be fixed.

What if the corpus grows from 10 documents of 100 pages to 100 documents of 100 pages?

The corpus grows tenfold and so does every index. With the illustrative settings, a 100-page document gives 145 chunks, so 10 documents give 1,450 chunks and 100 documents give 14,500¹. The text index grows from 5.9 MB to 59.4 MB in fp32, and the page-image index from 0.26 gigabytes to 2.64 gigabytes¹. Both fit in memory at this size, so the first thing that changes is not storage but retrieval quality: more near-duplicate pages compete, so add metadata filters, hybrid search and reranking, and re-measure on your question set.

Describe a RAG pipeline. You assumed a PDF, so what do you do if there are images in it?

Keep the figure as its own element, with its caption and nearby text. Describe it with a vision-language model for the text index, or embed the page image directly, and cite the page in the answer.

How do you evaluate a document RAG system?

Build a set of real questions, mark the page that answers each, and measure retrieval first, then answer quality. See Ragas for metrics, and sample the parse quality separately, since a bad parse caps everything after it.

When would you embed page images instead of extracted text?

When layout and figures carry the answer, such as chart-heavy or scanned reports, and the storage and indexing cost are acceptable. The ColPali paper reports better retrieval on visually rich documents, at 257.5 KB per page.

Sources

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401v4) — arXiv (Lewis et al.), 12 Apr 2021
2. Docling Technical Report (arXiv 2408.09869v5) — arXiv (Auer et al., IBM Research), 9 Dec 2024
3. ColPali: Efficient Document Retrieval with Vision Language Models (arXiv 2407.01449v6) — arXiv (Faysse et al.), ICLR 2025, 28 Feb 2025
4. OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation (arXiv 2412.02592v4) — arXiv (Zhang et al.), ICCV 2025, 30 Aug 2025
5. olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models (arXiv 2502.18443v3) — arXiv (Poznanski et al., Allen Institute for AI), 25 Feb 2025
6. ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval (arXiv 2505.17166) — arXiv, 22 May 2025
7. Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172v3) — arXiv (Liu et al.), TACL 2023, 20 Nov 2023