Design a web crawler

In this article

1. Before you start
2. Why interviewers ask this
3. A web crawler from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step on the system design learning path.

Why interviewers ask this

A crawler question tests whether you can design a system whose hardest constraint is being a good guest. Fetching pages is easy. Fetching a billion of them without overloading one site, without downloading the same content twice and without falling into an endless loop of generated links is the real problem. The prompt can add a twist, such as "ingest a billion documents" or "extract image URLs and expose an application programming interface (API) to start a job and check its status".

A weak answer draws one queue, a pool of workers and a database. A strong answer starts from the constraints, separates the frontier from the fetchers, explains how each host is protected, and says what the duplicate filters cost in memory. The same design now feeds search engines, training-data pipelines for language models and the retrieval index behind answer engines, so interviewers use it to see how you handle scale and responsibility together.

A web crawler from first principles

URL frontier

A URL frontier is the data structure that holds every URL still waiting to be downloaded and decides which one is handed to a fetcher next, including how often each host may be contacted.

Politeness

Politeness is the set of rules that limits how fast a crawler contacts any single host, such as one request at a time and a pause between requests, so the crawler does not slow or break the site.

Bloom filter

A Bloom filter is a compact bit-array structure that tests whether an item was added before; it can wrongly say yes, but it never wrongly says no.

In plain words

A crawler is a team of couriers collecting parcels from shops. Each parcel says where to find more parcels. The team keeps a to-do list of shops. Two rules keep it welcome. Only one courier visits a shop at a time, and no courier returns to the same shop straight away. The team also keeps a list of parcels already collected, so no parcel is fetched twice. The analogy breaks in one place. A shop's stock changes, so a visited shop must be visited again, and deciding when is its own problem.

In one precise sentence: a crawler is a loop of frontier, fetcher, parser and filters, with a store that remembers what was fetched and when.

Step 1: Seed the frontier with normalised URLs

The seed set is the list of URLs the crawl starts from, such as popular sites, sitemaps and the list a job request supplies. Each URL is normalised before it is queued: lower-case the host name, drop the fragment after the #, resolve relative links against the page they came from and remove default ports. The input is raw text; the output is one canonical string per page.

Normalisation matters because every later filter compares strings. Without it, Example.com/a and example.com/a#top are two pages. Heydon and Najork's Mercator crawler routes URLs by their canonical host name, so even its queueing depends on this step (Heydon and Najork, 1999). What you now know: every URL has one canonical form before it enters the system.

Step 2: Pick the next URL one host at a time

The frontier is not one queue. It holds one queue per host, and a scheduler hands out a URL only from a host whose last request has finished and whose pause has passed. Mercator's default frontier is a set of first-in-first-out subqueues, one per worker thread, chosen by canonical host name, so at most one thread downloads from a given server (Heydon and Najork, 1999).

A single queue fails because links cluster: many links on a page point to the same server (Heydon and Najork, 1999), so workers can hit one host together. In the toy test below, one host holds 12.2% of the URLs. With 100 workers pulling from one shared first-in-first-out queue, about 12 of them would be on that host at once, roughly 40 requests a second at 300 milliseconds a fetch, and all 100 if its links arrive together; a queue per host allows one. That host also sets the finish time of the whole crawl.¹ What you now know: politeness is a scheduling rule inside the frontier, not an afterthought.

Step 3: Fetch the page, after reading robots.txt

A robots.txt file is a plain-text file at the root of a site that tells crawlers which paths they may fetch. Before the first page of a host, the fetcher downloads it and caches it. Request for Comments (RFC) 9309 (September 2022) says crawlers should not use a cached copy for more than 24 hours, must parse at least 500 kibibytes (a kibibyte is 1,024 bytes), and must use the most specific matching rule, the one that matches the most characters. A 4xx response means the file is unavailable and the crawler may access any resource; a 5xx response means it is unreachable and the crawler must assume complete disallow.

The standard does not define a Crawl-delay line, so the delay is your own policy. A sound rule is to adapt: Google's documentation says server errors and HTTP 429 responses lower its crawl capacity limit, so it crawls less (Google Search Central, July 2026). Add a DNS (domain name system) cache and a timeout. What you now know: a fetch is permission, then a bounded download.

Step 4: Parse the page and drop duplicate content

The parser extracts links and a content fingerprint, a short number computed from the page body. If the fingerprint was seen before, the page is a duplicate: do not store it and do not follow its links again. Mercator's content-seen test used Rabin fingerprints, and in its May 1999 crawl 8.5% of the successful downloads were duplicates (Heydon and Najork, 1999).

Exact fingerprints miss pages that differ by a banner or a date. Simhash is a fingerprint built so that near-identical pages differ in only a few bit positions. Manku, Jain and Das Sarma report that for a repository of 8 billion web pages, 64-bit simhash fingerprints with a Hamming distance threshold of k = 3 (the Hamming distance is the number of bit positions where two fingerprints differ, so up to three differing bits still counts as a near-duplicate) are reasonable, and that 8 billion such fingerprints occupy 64 GB (Manku et al., 2007). What you now know: duplicates are caught by content, not by address.

Step 5: Admit only URLs you have not seen

Each extracted link is normalised and tested against the URL-seen filter: new ones enter the frontier, the rest are dropped. At billions of URLs an exact set is large, so many designs use a Bloom filter. Adding an element sets bits at several hash positions, so a false positive is possible and a false negative is not. Practice differs: Mercator describes the Internet Archive's Bloom filter but itself keeps an exact set of URL checksums on disk behind an in-memory cache, with no false positives and bounded memory (Heydon and Najork, 1999).

For n URLs and a false-positive rate p, the sizing is m/n = −ln p ÷ (ln 2)² bits per URL and k = (m/n) × ln 2 hash functions. For p = 0.01, m/n = 4.605 ÷ 0.4805 = 9.59 and k = 6.65, rounded to 7. For 10 billion URLs that is 10,000,000,000 × 9.59 ÷ 8 = 12.0 GB, against 80 GB for exact 8-byte checksums.¹ What you now know: a known error rate buys about a sixth of the memory.

Step 6: Store the page and schedule the recrawl

Fetched pages go to a blob store or a columnar file store, keyed by URL and fetch time, with the status code, the content fingerprint and the response headers beside them. A blob store holds large unstructured files addressed by key; a columnar file store keeps each field together, which suits scanning one field across billions of records. A crawl is never finished: pages change, so a scheduler picks which stored URLs return to the frontier, and when. Give each URL a priority from its observed change rate and its importance, and put it back with a due time.

The input is the stored page and its history; the output is a due time. Re-fetching everything on a fixed schedule wastes the budget on pages that never change. What you now know: freshness is a budget you spend where change is likely, as the worked example shows.¹

1 · Seed2 · Pick3 · Fetch4 · Parse5 · Filter6 · StoreReplay

Step-by-step description

1. Start the frontier from normalised seed URLs.
2. One queue per host gives politeness.
3. Check robots.txt, then download.
4. Extract links, drop duplicate content.
5. Only unseen URLs re-enter the frontier.
6. Save pages with their fetch time.

• The frontier keeps one queue per host so only one request per host is in flight.
• The fetcher reads robots.txt before the first page of a host.
• The parser drops pages whose content fingerprint was already seen.
• The URL-seen filter sends only URLs not seen before back into the frontier.

Worked example

Take the prompt "crawl 1 billion pages in 30 days". Every input is an illustrative assumption, and the script is saved with the page.¹ The rate is 1,000,000,000 pages divided by 2,592,000 seconds: 385.8 pages a second. At an average of 100 KB a page that is 38.6 MB a second, about 309 megabits, and 100 TB raw or 25 TB at 4-to-1 compression.

The two politeness rows are the surprise. A host with 5 million pages cannot be finished inside the 30 days with a pause after every request, so politeness, not machines, bounds the crawl. A toy simulation shows the same effect: 200,003 URLs on 2,000 hosts of Zipf-distributed size (a few huge hosts, many tiny ones), 100 workers, 300 milliseconds a fetch. With workers alone the crawl would finish in 600 seconds, 0.17 hours. One request at a time per host raises it to 7,543 seconds, 2.1 hours. Adding a one-second pause per host raises it to 31,901 seconds, 8.9 hours, and the largest host alone needs 31,792 of them.¹

Text description of the diagram

• With workers alone the toy crawl would finish in 0.17 hours.
• One request at a time per host raises it to 2.1 hours.
• A one-second delay per host raises it to 8.9 hours.

Freshness is the other budget. Assume pages change on average once a week, at random. Refreshing 1 billion pages every 7 days needs 1,653 fetches a second, more than four times the first-crawl rate. With a change rate λ of 1/7 per day, the share of stored copies that are current, averaged over a recrawl interval of T days, is (1 − e^(−λT)) ÷ (λT). For T = 7, λT = 1 and (1 − 0.368) ÷ 1 = 63.2%. The averages are 93.2% for a 1-day interval, 81.3% for 3 days, 63.2% for 7 days and 23.0% for 30 days. Just before the next refresh the share is lower, e^(−λT): 86.7%, 65.1%, 36.8% and 1.4%.¹

Text description of the diagram

• At a 1-day recrawl interval an average of 93.2% of copies are current.
• At 7 days it is 63.2%.
• At 30 days it is 23.0%.

Trade-offs and edge cases

The design has several decisions where a single answer is wrong. State the choice and the reason.

• Breadth against depth. First-in-first-out order inside each host queue crawls breadth-first. A priority frontier reaches important pages earlier but can starve the rest; mix both.
• Crawler traps. Calendars, session identifiers and faceted filters generate endless distinct URLs. Cap the depth and the number of URLs per host, and watch for very long or repeating paths. Google advises consolidating or blocking infinite spaces in robots.txt, not using noindex, because the page is still requested (Google Search Central, July 2026).
• Bloom filter false positives. A false positive silently drops a real page. Accept it for the wide web; use an exact store for a small, high-value set.
• Pages built by scripts. Rendering JavaScript costs far more than a plain fetch, so render only the hosts that need it.
• Distribution. Partition hosts across machines by a hash of the host name, so one machine owns each host's queue and its pause timer. Machines then exchange the links they find.
• Failure. Persist the frontier. After a crash the crawler resumes from the queues and does not restart from the seeds.
• Rules beyond robots.txt. The standard is a courtesy convention, not access control, and the law and each site's terms still apply.

For comparable shared-state problems see designing a distributed cache, the machine learning system design overview and a distributed rate limiter, which solves the same per-key pacing problem from the other side.

How to answer it in an interview

The spoken structure: numbers first, then frontier and politeness, then the two filters, then storage and recrawl, then the constraint that binds.

Common mistakes

• "One queue of URLs, many workers." (wrong) Links cluster by host, so workers hit one site together and overload it. (right) "One queue per host, and a scheduler enforces the pause."
• "Ignore robots.txt for speed." (wrong) The standard asks crawlers to follow it (it is not access authorization, so it does not replace other rules), and a 5xx response means assume disallow. (right) "Fetch and cache robots.txt for each host, then apply the most specific rule."
• "Keep every URL string in an in-memory hash set." (wrong) Full URL strings average far more than 8 bytes, so ten billion of them will not fit in memory. (right) "Store 8-byte checksums on disk with a cache, as Mercator did, or use a Bloom filter at 1% for about 12 GB and accept the dropped pages."
• "Dedup by URL only." (wrong) Different URLs serve the same content. (right) "Also fingerprint the content; simhash catches near-duplicates."
• "Recrawl everything weekly." (wrong) It spends the budget on pages that never change. (right) "Prioritise by change rate and importance."

Follow-up questions

How do you avoid crawling the same page twice at billions of URLs?

Normalise every URL, test it against a URL-seen filter (a Bloom filter sized for a 1% false-positive rate, or an exact checksum set), and fingerprint the content to catch duplicates under different addresses. At 10 billion URLs the filter takes about 12 GB, against 80 GB for exact checksums.

How do you decide what to recrawl?

Estimate each page's change rate from its fetch history and re-queue it with a due time and a priority. In the model above, a weekly-changing page is current 63.2% of the time on average at a weekly recrawl, so important or fast-changing pages earn shorter intervals.

How do you stay polite to a slow host?

Keep one request in flight per host and wait before the next. Lengthen the pause when responses slow or return 5xx or 429, which Google's documentation says lowers its own crawl rate (Google Search Central, July 2026).

How do you design the API for starting a crawl job and checking its status?

Expose an API call that creates a job from a list of URLs and returns a job identifier, a status call that returns the counts of queued, fetched and failed URLs, and a results call that pages through stored output. Make job creation idempotent (repeating the same request has the same effect as sending it once) with a client-supplied key, as in a payment system.

What changes when the crawl feeds a language model or a retrieval index?

Quality filters and content fingerprints matter more than raw page count, and the output is chunked (cut into passages) and embedded (turned into number vectors that capture meaning); see hybrid search. The politeness and robots.txt rules do not change, and a crawl that serves features also needs the freshness path of a feature store.

Sources

1. Mercator: A scalable, extensible Web crawler — World Wide Web 2 (Heydon and Najork), 1 Dec 1999
2. RFC 9309: Robots Exclusion Protocol — IETF (Koster et al.), 1 Sep 2022
3. Detecting near-duplicates for web crawling — WWW 2007 (Manku, Jain and Das Sarma), 8 May 2007
4. Managing your crawl budget (large sites) — Google Search Central, 22 Jul 2026