How do you design a crawler for LLM training data?

In this article

1. Before you start
2. Why interviewers ask this
3. Crawler for LLM training data from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one design topic in the generative AI system design interview guide.

Why interviewers ask this

A candidate who answers "I would write a crawler with a queue and workers" has described the cheap half of the problem. This question separates people who see a training-data crawler as one stage of a data pipeline from people who stop at fetching. A weak answer lists fetchers, queues and storage. A strong answer first asks whether to crawl at all, because public archives exist, then covers politeness, robots.txt, deduplication, quality filtering and what happens when a site opts out.

A Staff-level answer adds numbers: pages per second, bandwidth, and how many pages survive cleaning. It also separates crawlers that collect training data from crawlers that serve search or a user's request. The wider design round is covered in the generative AI system design guide, and the ChatGPT and Claude.ai design walkthrough shows where training data sits in a whole product.

Crawler for LLM training data from first principles

Crawl snapshot

A crawl snapshot is a dated copy of the pages one crawling run fetched, stored as raw server responses, that later pipelines can read again and again without fetching the web a second time. Common Crawl, a nonprofit, publishes a new one about once a month.

Crawl politeness

Crawl politeness is the set of limits a crawler puts on itself, such as a pause between requests to one host and a cap on parallel connections, that keep it from overloading any single site. It is separate from robots.txt, which states what a site owner asks crawlers to leave alone.

Robots Exclusion Protocol

The Robots Exclusion Protocol is a standard that lets a site owner list, in a plain text file called robots.txt at the top of the site, which paths automated clients should not fetch. It is a request that well-behaved crawlers honor, not access control or a security measure.

In plain words

Picture a driver who collects every leaflet pinned to every noticeboard in a city, following the rules each board posts. Collecting is quick. The real work is sorting: throwing away copies, adverts and unreadable scraps, and keeping the leaflets worth printing in a textbook. The analogy breaks in one place. The rules on the noticeboards are requests that nobody enforces, so the driver's own behavior is the only safeguard.

In one precise sentence, a crawler for LLM training data fetches raw pages under stated rules, and a later pipeline of extraction, deduplication and filtering produces the stream of tokens the model trains on.

Step 1: Decide whether to run your own crawl

Many teams reuse a Common Crawl snapshot instead of crawling the web themselves, because the archive already exists and is free. Common Crawl is a nonprofit that publishes its crawls approximately once a month, each typically containing more than two billion web pages. Its crawler, CCBot, obeys robots.txt and rate-limits its requests to individual servers.

It also uses Harmonic Centrality, a graph-theoretic measure of how close a page is to the structural core of the web, to choose which uniform resource locators (URLs) to fetch first. The FineWeb paper shows reuse at scale: its dataset is sourced from 96 Common Crawl snapshots. Run your own crawl only when the archive cannot give you what you need, such as fresher pages or a narrow set of domains. Input: a data need. Output: a choice between an existing snapshot and a new crawl.

Step 2: Fetch politely and store raw responses

A polite crawler loops over a queue of URLs, called the frontier, and reads each host's robots.txt (cached, refreshed at least daily) before fetching from it. For each host it skips disallowed paths, waits between requests to that host, fetches the page, stores the full response, extracts the links and adds new ones to the frontier. The Robots Exclusion Protocol is specified in Request for Comments (RFC) 9309, and the rules live in a file named /robots.txt at the top-level path.

Politeness limits beyond that are your own design: Anthropic says it supports the non-standard Crawl-delay extension to limit crawling (Anthropic Help Center). WARC (Web ARChive) is a file format, standardized by the International Organization for Standardization (ISO) as 28500, that stores full HTTP responses as raw crawl data. Common Crawl also publishes a lighter metadata format extracted from those records, with HTTP headers and extracted links (Common Crawl).

Step 3: Clean and deduplicate the raw pages

Raw crawl data is not training data. Cleaning extracts the readable text from each response, identifies its language, removes personally identifiable information (PII) such as email addresses, drops repeats and filters by quality. Deduplication is the removal of pages or passages whose text repeats text already kept. MinHash is a method that estimates the Jaccard similarity of two pages' shingle sets (overlapping word sequences) from short signatures, so near-copies are found without comparing every pair; FineWeb applies MinHash deduplication to each snapshot (Penedo et al., 2024, arXiv 2406.17557).

The paper opens with the point that a large language model's performance "depends heavily on the quality and size of its pretraining dataset". Its result: 15 trillion tokens from 96 snapshots, which is about 156.25 billion tokens per snapshot on average.¹ A further filter, a custom classifier that rated pages as highly educational, kept a subset of 1.3 trillion tokens, about 8.67% of the whole.¹ To see near-duplicates, compare overlapping word triples. In an illustrative pair of seven-word sentences that differ in the last word, 4 of 6 distinct triples are shared, a similarity of 0.6667.¹ The stages after cleaning are covered in the LLM training pipeline.

Step 4: Honor opt-outs and check who is crawling

Sites choose what to allow per crawler name, and the vendors split their bots by purpose. OpenAI documents that a site can allow its search crawler to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training, and that each setting is independent (OpenAI). Anthropic says that when a site restricts ClaudeBot, it signals that the site's future materials should be excluded from training (Anthropic Help Center).

The word "future" matters: a robots.txt change does not by itself remove pages from snapshots already published or from a model already trained; removal from an archive is a separate request to its operator. Because a user-agent string is only a claim, operators also check a crawler's network addresses against the Internet Protocol (IP) ranges some vendors publish (OpenAI). The crawler's opt-outs are applied again at the start of every new crawl.

1 · Choose a source2 · Fetch politely3 · Clean4 · Honor opt-outsReplay

Step-by-step description

1. Reuse a snapshot or run your own crawl.
2. Read robots.txt and limit your request rate.
3. Extract text, deduplicate and filter.
4. A block applies to future crawls only.

• Many teams reuse a Common Crawl snapshot instead of crawling the web themselves.
• A polite crawler checks robots.txt and waits between requests to each host.
• Cleaning extracts text, removes duplicates and filters pages.
• An opt-out applies to future crawls only.

Worked example

Take the lower bound from Common Crawl: more than two billion pages in a monthly crawl. Suppose you want to fetch two billion pages in 30 days. That is 2,592,000 seconds, so you need about 772 pages per second.¹ Assume an average page of 100 kilobytes (illustrative; no source supplies it). The raw download is then 200 terabytes, and the sustained network rate is about 617 megabits per second.¹ Politeness sets how many hosts you must work on together: with an illustrative one-second gap between requests to a host, at least 772 hosts at a time; with a ten-second gap, at least 7,717.¹

Now the funnel. Assume (illustrative) that extraction keeps 90% of fetched pages, deduplication keeps half of those and the quality filter keeps 40% of what remains. In this illustrative funnel, 2,000 million fetched pages shrink to 360 million after filtering, an 18% yield, and deduplication halves the pages that remain after extraction.¹ The keep rates are assumptions; the FineWeb paper holds the real stage-by-stage figures.

Text description of the diagram

• In the illustrative funnel, 2,000 million fetched pages shrink to 360 million after filtering.
• Deduplication halves the pages that remain after extraction.

Trade-offs and edge cases

Reusing a snapshot saves fetching cost but gives you no say over freshness or coverage; your own crawl gives control and costs bandwidth, engineering and ongoing politeness work. For search-time freshness a model is often better served by fetching at query time, as in the search-with-answers design walkthrough, or by asking whether you need retrieval at all.

As of October 2026, OpenAI and Anthropic each run separate crawlers for training, search and user requests. Anthropic lists ClaudeBot for training, Claude-SearchBot for search and Claude-User for user-directed fetches (Anthropic Help Center). Google's Google-Extended is a robots.txt token, not a separate crawler: Google, as quoted by a secondary summary, says it has no separate user-agent string and that crawling uses existing Google user agents, with the token used in a control capacity. Reported practice on user-initiated fetches differs: OpenAI's crawler overview says robots.txt rules may not apply to ChatGPT-User, while Anthropic's help article lists Claude-User among the bots a site can block through robots.txt. Treat that as a vendor-policy difference.

Text description of the diagram

• OpenAI and Anthropic each run separate crawlers for training, search and user requests.
• Google-Extended is a robots.txt token, not a separate crawler.

Edge cases: a robots.txt entry covers only the names it lists, so a new crawler name is not blocked by old entries. RFC 9309 says "These rules are not a form of access authorization" (Koster et al., 2022), and listing paths in the file exposes them publicly. Enforcement against crawlers that ignore the rules needs firewall-level controls, because compliance is voluntary. Text extraction of tables and charts is its own problem; see retrieval-augmented generation (RAG) over real documents and the document-processing pipeline design.

How to answer it in an interview

The spoken structure: ask whether to crawl, describe the polite fetch loop, spend most of your time on cleaning, then cover opt-outs.

Common mistakes

• "I would write a crawler: a queue, workers and storage, done." (wrong) That is only the fetch. (right) "The fetch is the first stage; extraction, deduplication and filtering produce the training text."
• "Blocking the crawler deletes my content from models." (wrong) Anthropic describes a block as signaling that future materials should be excluded, and a robots.txt change does not by itself remove pages from snapshots already published or from a model already trained; removal from an archive is a separate request to its operator. (right) "An opt-out applies to future crawling."
• "Google-Extended is a crawler." (wrong) It is a control token with no user-agent string of its own. (right) "Googlebot fetches; the token controls use for Gemini training and grounding."
• "robots.txt keeps private pages private." (wrong) RFC 9309 says it is not a security measure. (right) "Private content needs real access control."
• "More crawled data always means a better model." (wrong) The FineWeb paper ties performance to quality as well as size. (right) "I measure what cleaning keeps, not only what the crawl fetches."

Follow-up questions

How do you avoid overloading a site when crawling?

Set a pause between requests to each host and a cap on parallel connections, and spread work across many hosts. At 772 pages per second, a ten-second gap per host needs at least 7,717 hosts in flight, so politeness shapes the whole scheduler.

How do you remove near-duplicate pages at scale?

Compare overlapping word sequences from each page rather than whole pages, since two near-copies share most of their sequences (0.6667 in the toy pair). At scale, MinHash estimates the overlap of those sequence sets from short signatures, so near-copies are found without comparing every pair.

Does blocking a training crawler remove my content from a model?

No: a block signals that a site's future materials should be excluded, and a model already trained keeps what it learned. A robots.txt change does not by itself remove pages from snapshots already published or from a model already trained; removal from an archive is a separate request to its operator.

Is robots.txt enough to keep private pages out of training data?

No: robots.txt is a request crawlers are asked to honor, not access authorization, and compliance is voluntary. Use authentication or firewall rules for anything private, and see permission-aware retrieval for the same idea inside a product.

How do you tell a real crawler from a fake one?

A user-agent string can be spoofed, so compare the requesting network address with the address lists some vendors publish for their crawlers. Treat a mismatch as untrusted traffic whatever the string says.

Sources

1. About Common Crawl — Common Crawl, 26 Oct 2026
2. RFC 9309: Robots Exclusion Protocol — IETF / RFC Editor (Koster et al.), 1 Sep 2022
3. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (arXiv 2406.17557) — arXiv (Penedo et al.), 25 Jun 2024
4. Overview of OpenAI Crawlers — OpenAI, 26 Oct 2026
5. Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic Help Center, 26 Oct 2026
6. The AI user agents landscape (quotes Google on Google-Extended) — nohacks.co (secondary), 1 Sep 2026