In this article
Before you start
This page sits inside the retrieval-augmented generation guide.
Why interviewers ask this
A RAG system is only as good as the chunks it can retrieve, so a wrong split caps quality before any model runs. Candidates report being asked what chunking is, which techniques exist, how chunking changes retrieval quality, how to handle large chunks, and how to chunk different kinds of documents.
A weak answer says "split the text into 500-word pieces." A strong answer names the trade-off between small chunks that match precisely and large chunks that keep context, and says how it would measure the choice. A Staff-level answer also knows that overlap has a cost, that context can be added to a chunk instead of cut around it, and that the right size depends on the data. The RAG interview questions page lists neighbouring questions.
Chunking strategies for RAG from first principles
- Chunk overlap
- Chunk overlap is a technique that repeats text at the edges of neighbouring chunks, so that a sentence cut at a boundary still appears whole in one chunk, at the price of embedding the repeated text twice.
- Late chunking
- Late chunking is a method that embeds a whole document first with a long-context embedding model and splits the result into chunk vectors afterwards, so each chunk vector carries context from the full text.
- Contextual retrieval
- Contextual retrieval is a method that prepends a short model-written description of where a chunk sits in its document to the chunk before it is embedded and indexed, so the chunk is no longer context-free.
In plain words
Think of an encyclopedia cut into index cards. Each card is easy to file and find. A card that says "it grew by a third" is useless alone, though. The analogy breaks in one place: a real reader sees the neighbouring cards, but a search engine scores each card by itself.
In one precise sentence, chunking decides the unit of retrieval, and every strategy balances a small unit that matches sharply against a large unit that keeps its context. Cards that are too small lose the subject of the sentence. The search then cannot tell which company or quarter a card is about. Cards that are too large mix several topics into one vector. The match turns vague, and the answer model must read extra text to find the useful part.
Step 1: Documents are split because embeddings blur long text
The late chunking authors note that dense retrieval often performs better with shorter segments, because their meaning is less likely to be over-compressed into one vector. So practitioners split documents and embed the pieces separately.
The price is lost context. The same paper warns that chunk embeddings can lose information from surrounding chunks; its example is a later sentence that says "its" or "the city" when only the first sentence names Berlin. A chunk reading "Revenue grew over the previous quarter" (illustrative) names no company and no quarter, so it matches poorly. You now know the tension: split to match sharply, and avoid cutting away the meaning. Each later step manages that tension.
Step 2: Fixed-size and recursive splitters decide where to cut
Fixed-size chunking cuts every N tokens or characters, wherever that falls. It is simple and fast, and it can cut mid-sentence.
A recursive splitter is a splitter that tries large separators such as paragraph breaks first, and falls back to smaller ones such as lines and words only when a piece is still too big. The LangChain splitter exposes chunk_size, the maximum size of a chunk, measured by a length_function. Check what your function counts, characters or tokens, because the unit changes the meaning of every number you tune.
Input: a document and a maximum size. Output: chunks that stay under the maximum and, where possible, end at natural breaks. A hard cut can still land mid-thought, which is what overlap tries to soften.
Step 3: Overlap repeats text, so more is not safer
Chunk overlap repeats text at the edges of neighbouring chunks. LangChain describes chunk_overlap as softening information loss when context is divided between chunks. It does not remove duplicates, so every overlapped token is embedded again and stored again.
Chroma's test shows the limits. For a plain recursive splitter at 400 tokens, adding 200 tokens of overlap moved recall from 89.5% to 88.1%. Precision and intersection over union (IoU), a score that penalises redundant text, fell in the same pair of rows, from 3.6% to 3.3%. At equal chunk size, then, the extra overlap bought no recall. Overlap also creates near-duplicate chunks, which crowd the context window at answer time.
The worked example below puts numbers on the cost. You now know that overlap is a trade, not a free safety margin.
Step 4: Structure-aware and semantic chunking cut at meaning
Structure-aware chunking cuts along the document's own structure: headings, list items, table rows, or functions in code. LangChain's code splitter builds a recursive splitter for a named programming language, so cuts follow code syntax.
Semantic chunking is chunking that places boundaries where the meaning shifts, found by comparing embeddings of neighbouring sentences or by clustering them, instead of by length. Chroma tested a cluster-based version: its ClusterSemanticChunker with a 400-token maximum reached recall of 91.3%, the second highest in that test. The same paper finds that a plain recursive splitter often performs well when its settings are chosen with care, so semantic chunking is not automatically better.
Large language model (LLM)-based chunking goes one step further: a language model is prompted to place the cuts itself. In Chroma's test an LLM-based chunker reached the highest recall of all methods, 91.9%, but it needs a model call over the whole corpus before anything is embedded. Rankings depend on data and embedder, so test on your own questions with the methods in how to evaluate a RAG system.
Step 5: Add context to the chunk instead of cutting around it
Two methods give a chunk the context that the cut removed. They change different things.
Anthropic's contextual retrieval prepends a short model-written context to each chunk before embedding and before indexing for Best Matching 25 (BM25), a keyword scoring method. Anthropic reports that contextual embeddings cut the top-20 retrieval failure rate by 35%, from 5.7% to 3.7%, adding contextual BM25 cut it by 49%, to 2.9%, and adding reranking brought the cut to 67%. The failure rate is one minus recall at 20.
Late chunking embeds all tokens of the long text first, then applies chunking just before mean pooling, and needs no additional training.
Step-by-step description
- Short pieces embed sharply but lose context.
- Recursive splitters cut at natural breaks first.
- Edge text repeats, so more tokens are embedded.
- Cuts follow headings, code or meaning shifts.
- Late chunking or contextual retrieval restores context.
- A chunk is a piece of a document that is embedded and retrieved on its own.
- Overlap repeats text at the edges of neighbouring chunks, so it adds tokens to embed.
- Contextual retrieval and late chunking give each chunk more context.
Parse before you chunk: PDFs and tables
Parsing is the step that turns a source file into clean text before any chunking, and a bad parse caps retrieval quality as surely as a bad split. A portable document format (PDF) file stores where each piece of text sits on a page, not where paragraphs, headings or table rows begin, so a naive extractor can merge columns or scatter a table across lines.
A scanned page holds only an image and needs optical character recognition (OCR) first. A strong pipeline extracts headings, paragraphs and tables as separate elements, keeps a table whole or splits it by row with its header repeated, and only then applies the chunking strategies above. Check a few parsed pages by eye before you tune any chunk size.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Take a document of 2,000 tokens (illustrative) split into chunks of at most 400 tokens. The number of chunks grows as overlap grows: 5 chunks with no overlap, 6 with 50 tokens of overlap and 9 with 200 tokens.1 The tokens you must embed grow faster: 2,000, then 2,250, then 3,600, which is 12.5% and 80% more than the document itself.1
| Overlap (tokens) | Chunks | Tokens embedded | Extra over no overlap |
|---|---|---|---|
| 0 | 5 | 2,000 | 0% |
| 50 | 6 | 2,250 | 12.5% |
| 200 | 9 | 3,600 | 80% |
Text description of the diagram
- Chunks of 400 tokens with no overlap embed 2,000 tokens from a document of 2,000 tokens.
- Chunks of 400 tokens with 200 tokens of overlap embed 3,600 tokens.
Now compare that cost with what it bought in a real test. Chroma reported, for a recursive splitter (Chroma Research, July 2024). The test used n = 5 retrieved chunks and OpenAI text-embedding-3-large; the recursive chunks averaged about 661, 312 and 276 tokens in the three rows.
| Size / overlap (tokens) | Recall (%) | Precision (%) | IoU (%) |
|---|---|---|---|
| 800 / 400 | 85.4 | 1.5 | 1.5 |
| 400 / 200 | 88.1 | 3.3 | 3.3 |
| 400 / 0 | 89.5 | 3.6 | 3.6 |
Chroma notes that 800 tokens with 400 overlap is the documented default for OpenAI file search; tested with a token splitter it gave slightly below-average recall (87.9%) and the lowest scores on every other metric (Chroma Research, July 2024). In the recursive rows, 800/400 had the lowest recall and the weakest precision and IoU.
Trade-offs and edge cases
No chunker wins everywhere, so choose by data and by what breaks.
- Large chunks. A chunk that exceeds the embedder's input limit is cut or rejected, and a chunk that fits can still blur several topics. Split it, or index small pieces and return the larger passage around the match (small-to-big, or parent-document, retrieval). Check your embedding model's maximum input before you tune sizes.
- Short documents. A one-page invoice becomes one chunk, so there is nothing to split and chunk size stops mattering. The hard part moves to metadata and to telling similar invoices apart.
- Different data. Medical reports have sections and long sentences; social posts are short and self-contained. Cut the first at headings, keep the second whole.
- Near-duplicate results. Overlap makes neighbouring chunks nearly identical, so retrieval can fill the context window with the same text. Lower the overlap or deduplicate before you add anything fancier.
- Changing strategy. A new splitter changes every chunk, so you must re-embed and re-index the corpus.
- Long documents and late chunking. Late chunking needs the whole document inside the embedder's context window; longer texts are processed in overlapping macro-windows ("long late chunking") (Günther et al., 2025).
Late chunking and contextual retrieval differ in cost and in what they change. This comparison is our synthesis of the two sources: late chunking needs a long-context embedding model but no extra model call, and contextual retrieval needs a language model call per chunk but works with any embedder plus keyword indexing. As of 2026, a 2025 comparison found late chunking more efficient but tending to give up relevance and completeness against contextual retrieval (Merola and Singh, 2025).
Text description of the diagram
- Late chunking changes the embedding step.
- Contextual retrieval changes the text that is indexed.
- Late chunking needs a long-context embedding model; contextual retrieval needs a model call per chunk.
For questions that depend on links between entities rather than on passages, see GraphRAG.
How to answer it in an interview
What chunking strategies do you know, and how do they affect retrieval?
Chunking splits documents into pieces that are embedded and retrieved separately, so small chunks match precisely but lose context. I start with a recursive splitter that respects paragraph and heading boundaries, measure size in tokens, and use little or no overlap, because overlap multiplies embedding cost and creates near-duplicate results. Then I test sizes on my own questions and compare recall. If chunks lose their subject, I add context: either prepend a model-written description to each chunk, which is contextual retrieval, or embed the whole document first and split afterwards, which is late chunking.
The spoken structure: define chunking, name the trade-off, give your default, then say how you would measure and fix it.
Common mistakes
- "Bigger overlap is always safer." (wrong) It repeats tokens you pay to embed and can lower precision. (right) "I keep overlap small and check it against recall on my own questions."
- "chunk_size is in tokens." (wrong) It is whatever the length function counts. (right) "I pass a token counter so the unit matches my embedder."
- "Semantic chunking is clearly best." (wrong) A simple recursive splitter did well in Chroma's test. (right) "I compare a recursive baseline with a semantic one on my data."
- "Late chunking and contextual retrieval are the same." (wrong) One changes the embedding step, the other the indexed text. (right) "Late chunking embeds first and splits after; contextual retrieval adds context before embedding."
Follow-up questions
How does chunking impact the quality of retrieval in RAG?
The strategies (fixed-size, recursive, structure-aware, semantic, late chunking and contextual retrieval) differ in where they cut and how much context each chunk keeps. Chunking sets the unit that the search can return. Chunks that are too small lose context and chunks that are too large blur topics, so recall and precision both move; in Chroma's test they changed with chunk size and overlap.
What are different parsing and chunking strategies?
Parse first: turn PDFs and tables into clean elements such as headings, paragraphs and table rows. Then chunk those elements with the strategies listed above.
What is the best chunk size for RAG?
There is no single best size. In Chroma's test, chunks of about 200–400 tokens with little or no overlap did best; measure on your own documents and questions.
How can we handle large chunks in a RAG pipeline?
Split them at structure boundaries, or index small pieces and return the larger surrounding passage at answer time. Make sure each piece fits the embedding model's input limit.
What is late chunking, and how does it differ from contextual retrieval?
Late chunking embeds the whole document and splits afterwards, with no extra model call. Contextual retrieval adds model-written context to each chunk before embedding, costing one call per chunk.
How would you chunk medical documents versus social-media posts?
Cut medical documents at section headings so each chunk keeps its context; keep short posts whole, since splitting them removes meaning. Then test both on real questions.
- Chunking sets the unit of retrieval, and every strategy trades sharp matching against lost context.
- Recursive splitting at paragraph boundaries is a strong, cheap baseline.
- Overlap multiplies embedding cost; in Chroma's test it did not raise recall and, at equal size, lowered it.
- Check whether chunk size counts characters or tokens.
- Late chunking and contextual retrieval add context in different places.
- Choose the strategy by measuring recall on your own questions.
Sources
- Contextual Retrieval — Anthropic Engineering, 19 Sep 2024
- Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models (arXiv 2409.04701v3) — arXiv (Günther et al., Jina AI), 7 Jul 2025
- Reconstructing Context: Evaluating Advanced Chunking Strategies for Retrieval-Augmented Generation (arXiv 2504.19754) — arXiv (Merola and Singh, ECIR 2025 workshop), 28 Apr 2025
- Evaluating Chunking Strategies for Retrieval — Chroma Research (Smith and Troynikov), 3 Jul 2024
- Splitting recursively (RecursiveCharacterTextSplitter) — LangChain documentation, 26 Oct 2026
- Split code (code splitter) — LangChain documentation, 26 Oct 2026
