The web crawler coding question, step by step

In this article

1. Before you start
2. Why interviewers ask this
3. The web crawler from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the machine coding rounds guide.

Why interviewers ask this

A crawler looks like a plain graph walk, and that is why it works as a screening problem: the first version fits in a few lines, and each follow-up adds one real-world constraint. Candidates report this question from Anthropic's coding rounds, both as an onsite multithreaded crawler and as a levelled industry-coding assessment that moves from fetch and parse to rate limiting and then distributed crawling. One report, at a staff level, asked to strip query-string noise, print only unique links, then make the crawler multithreaded. These are candidate reports, not Anthropic's published practice.

A weak answer walks the links and forgets to mark pages seen, so the program loops forever on a site whose pages link to each other. A strong answer states its assumptions first, keeps one clear owner for the seen set, and explains what changes when threads and then machines arrive. The neighbouring rounds in the machine coding guide test the same habit of growing a design in levels.

The web crawler from first principles

Web crawler

A web crawler is a program that starts from one or more seed URLs, downloads each page, extracts the links on it, and follows them to new pages, while remembering which pages it has already handled so that no page is processed twice.

URL normalization

URL normalization is the rewriting of a URL into one canonical form, for example by resolving a relative link against its page, lower-casing the host and dropping the fragment, so that different spellings of the same page compare as equal (see Request for Comments (RFC) 3986, section 6).

In plain words

Picture exploring a town. Every street sign points to other streets. You keep a to-do list of streets whose signs you have seen. You also keep a tick list of streets you have walked. Take the next street from the to-do list and walk it. Add every new street its signs point to, then tick it off. The analogy breaks in one place. A crawler can make many walkers share the same lists. Then two walkers can reach for the same street together.

In one precise sentence, a crawler is a graph traversal in which pages are nodes, links are edges, and a seen set stops the traversal from revisiting a node.

Step 1: Fetch a page and extract its links

The input is one URL; the output is the list of links on that page. In the interview this is usually a function you are given, such as a get_links(url) call, because downloading and parsing hypertext transfer protocol (HTTP) pages is not the point. Ask whether the function can fail or be slow, and say you will treat a failure as "no links" unless told otherwise.

Everything else is bookkeeping around that call. The toy site used below has 8 pages that carry 16 raw links between them.¹ You now know the unit of work: one fetch in, a list of links out. The next step cleans that list.

Step 2: Normalize and filter every link

A raw link is not yet a page. It can be relative, carry a query string such as ?utm=1, or end in a fragment such as #top. Normalization turns each link into the form you compare and store: resolve it against the page it came from, lower-case the host, and drop the fragment. A typical version of the question also asks you to stay on the start URL's host, so links to other hosts are discarded; state that as an assumption.

Dropping the query string is a choice, not a law. A query can name different content, so say you strip it here because the question asks for it, and that you would keep it for a site that serves distinct pages by query. In the toy site, 1 of the 16 raw links points to another host and is dropped.¹ Real normalization can go further, for example by unifying default ports and the http and https variants of the same page. In Python, the undefined documentation describes the functions that split and rebuild URLs. The output of this step is a clean, same-site URL or nothing.

Step 3: Mark a URL as seen before you queue it

The seen set is a hash set of normalized URLs. The rule is to test and add in one step, before the URL goes into the queue: if it is already in the set, skip it; otherwise add it, then enqueue it. Marking a URL only when you fetch it lets the same page enter the queue many times, and with threads it can be fetched twice.

On the toy site, 8 of the 16 raw links are duplicates that the seen set removes, 1 is external, and 7 links lead to new pages, which together with the start page makes 8 unique pages.¹ Marking at enqueue time also keeps the queue from growing without bound on a site with loops. You now have the single-threaded crawler; the code below is all of it.

from collections import deque
from urllib.parse import urljoin, urlsplit, urlunsplit

def normalize(base, href):
    # Resolve against the page, keep scheme, host and path only.
    # Possible extension: unify default ports and http/https.
    p = urlsplit(urljoin(base, href))
    return urlunsplit((p.scheme, p.netloc.lower(), p.path or "/", "", ""))

def crawl(start, get_links):
    host = urlsplit(start).netloc.lower()
    start = normalize(start, start)
    seen, queue, order = {start}, deque([start]), []
    while queue:
        url = queue.popleft()
        order.append(url)                      # "print" the unique link
        for href in get_links(url):
            nxt = normalize(url, href)
            if urlsplit(nxt).netloc != host or nxt in seen:
                continue
            seen.add(nxt)                      # mark seen BEFORE queuing
            queue.append(nxt)
    return order

Step 4: Add workers that fetch in parallel

A multithreaded crawler fetches several pages in parallel, because the work is waiting for the network, not computing. Python threads overlap network waits despite the global interpreter lock (GIL), so this claim holds. The simplest correct design fetches one level of the site at a time with a pool of worker threads, then collects the results in the main thread.

from concurrent.futures import ThreadPoolExecutor

def crawl_threaded(start, get_links, workers=4):
    host = urlsplit(start).netloc.lower()
    start = normalize(start, start)
    seen, frontier, order = {start}, [start], []
    with ThreadPoolExecutor(workers) as pool:
        while frontier:
            order.extend(frontier)
            nxt_level = []
            for url, links in zip(frontier, pool.map(get_links, frontier)):
                for href in links:
                    nxt = normalize(url, href)
                    if urlsplit(nxt).netloc == host and nxt not in seen:
                        seen.add(nxt)
                        nxt_level.append(nxt)
            frontier = nxt_level
    return order

Only the main thread touches the seen set, so it needs no lock. That is the trade: you wait for the slowest fetch of each level. The alternative, workers that pull from a shared queue and update a shared set, needs a lock or a thread-safe queue around every check-and-add.

That design has a hard part: knowing when to stop. An empty queue does not mean the crawl is done, because another worker may be mid-fetch and about to add links. A shared-queue crawler stops only when the queue is empty and no fetch is in flight. Track that with a pending counter under the lock, with queue.join() and task_done(), or by waiting on outstanding futures; a worker that sees an empty queue must not exit while work is pending.

1 · Fetch2 · Normalize3 · Seen, then queue4 · WorkersReplay

Step-by-step description

1. A page gives its links.
2. Links become canonical URLs.
3. Mark seen before queuing.
4. Workers fetch in parallel.

• The unit of work is one fetch in, a list of links out.
• Normalization turns each link into the form you compare and store.
• Test and add the URL to the seen set before the URL goes into the queue.
• Workers fetch pages in parallel.

Worked example

The toy site is illustrative: 8 pages on one host, with relative links, query strings, a fragment, repeated links and one link to another host. Starting from the home page, the crawl visits them in the order /, /a, /b, /c, /d, /e, /f, /g.¹ The pages fall into 4 levels of 1, 3, 3 and 1 pages by distance from the start.¹

With one worker, each fetch taking one time unit (illustrative), the crawl takes 8 units. With two workers fetching a level together it takes 6, and with four workers it takes 4, because a level cannot be finished faster than one fetch.¹

Text description of the diagram

• One worker needs 8 time units on the toy site.
• Four workers need 4 time units, one per level.

Trade-offs and edge cases

Threads help only while there is waiting to overlap, and a real site limits how fast you may hit it. Politeness is the rule that a crawler spaces its requests to one host so that it does not overload it. Request for Comments (RFC) 9309 (September 2022) specifies the Robots Exclusion Protocol that Martijn Koster originally defined in 1994, a file through which a site's owner states how crawlers may access it. A crawler that honours it reads that file before fetching a host.

Distributing the crawler across machines adds one decision: who owns a URL. A common design hashes the host or the URL to pick an owning machine, and sends every discovered link to its owner, which alone checks its own seen set. On six illustrative hostnames, a toy hash assigns two to each of three machines.¹ A single-host crawl gains no parallelism from host hashing, so hash the full URL there. That spreads one host over several machines, so per-host politeness then needs a shared or coordinated rate limiter.

Text description of the diagram

• Only the main thread touches the seen set, so it needs no lock.
• A threaded crawler keeps the seen set in the main thread.
• Across machines, each URL has one owning machine.

Other edge cases: pages that link to endless generated URLs need a depth or page cap; a failed fetch needs a retry limit; and a seen set that grows without bound on a huge crawl needs a compact structure. A related habit, bounding memory with a clear eviction rule, appears in the least recently used (LRU) cache write-up, and the same grow-by-levels shape appears in the in-memory database assessment.

How to answer it in an interview

The spoken structure: state assumptions, normalize, mark seen before queuing, then threads, then ownership across machines. Say aloud what you would measure next, such as the fetch time per host.

Common mistakes

• "I mark a page seen when I fetch it." (wrong) The same URL can sit in the queue many times, and two threads can fetch it. (right) "I add it to the seen set at the moment I enqueue it."
• "I compare raw URLs." (wrong) /a?utm=1 and /a count as two pages. (right) "I normalize first, and I say which parts I drop and why."
• "I strip every query string, always." (wrong) Some sites use the query to pick the content. (right) "I strip it because this question asks; otherwise I keep it."
• "More threads always means faster." (wrong) The toy crawl cannot beat its 4 levels,¹ and a host limits requests. (right) "Threads help until the level width or the per-host limit binds."
• "robots.txt blocks bad crawlers." (wrong) It is a protocol a crawler chooses to follow. (right) "My crawler reads it and obeys it before fetching."

Follow-up questions

How would you make the web crawler multithreaded?

Use a worker pool that fetches a level at a time, or workers that pull from a shared thread-safe queue. Keep one owner for the seen set, or guard the check-and-add with a lock, so a URL is never fetched twice. With a shared queue, stop only when the queue is empty and no fetch is in flight, because an empty queue alone does not mean the crawl is done.

How do you strip query-string noise from URLs safely?

Parse the URL into parts, then rebuild it without the query and fragment instead of cutting the text by hand. Say that this merges pages that differ only by query, which is wrong on sites that serve different content that way.

How do you stop a crawler from hammering one website?

Keep a per-host delay and a small cap on concurrent requests per host, and read the site's robots file first. The frontier then picks the next URL from a host that is not on cooldown.

How do you distribute a web crawler across multiple machines?

Hash each URL or host to an owning machine, send discovered links to their owner, and let the owner dedupe against its own seen set. Add retries and a way to hand over work if a machine fails. Related bookkeeping appears in event and log processing problems and low level design questions.

Should a crawler use breadth-first or depth-first search?

Breadth-first search with a queue is the usual choice, because it visits pages near the start first and maps neatly to levels of workers. Depth-first search with a stack also finds every page, but it makes the level-by-level thread design harder.

Sources

1. RFC 9309: Robots Exclusion Protocol — IETF / RFC Editor (Koster et al.), 1 Sep 2022
2. RFC 3986: Uniform Resource Identifier (URI) Generic Syntax — IETF / RFC Editor (Berners-Lee, Fielding, Masinter), 1 Jan 2005
3. urllib.parse: Parse URLs into components — Python Software Foundation, 26 Oct 2026