Design a multi-threaded web crawler.

What a strong answer does

I would bound the crawler before drawing boxes, because the scale decides whether this is a thread-pool exercise or a distributed system. Ask how many pages we need to crawl, over what period, what page size and latency to expect, and how fresh the index must be. A billion pages per month is about 400 pages per second on average, before retries, robots fetches, parsing, and spikes. That one number drives worker count, queue throughput, storage write rate, network bandwidth, and whether the frontier and seen-set fit on one machine or must be sharded.

The core design is a frontier queue of URLs, a pool of fetch workers, a parser that extracts links from fetched pages, and a seen-set that prevents duplicate URLs from entering the frontier. Workers take URLs from the frontier, fetch them, hand content to parsing and storage, and newly discovered links are canonicalized and checked before enqueueing. If those four pieces are explicit, the rest of the system becomes a set of engineering choices around partitioning, backpressure, retries, and storage durability rather than a vague “many threads crawl pages” answer.

Deduplication happens at two different levels. First, normalize URLs before the seen check: lowercase hostnames, remove fragments, sort or filter query parameters when safe, resolve relative paths, and standardize schemes and ports. That catches the same address written multiple ways. Second, hash fetched content or near-content so the system can notice the same page served from different URLs, mirrors, tracking links, or duplicate hosts. URL dedupe saves fetching work; content dedupe saves indexing and storage work, and it catches cases URL rules can never infer.

Politeness is central, not an add-on. The frontier should be partitioned by host or domain, with per-host rate limits and next-fetch times, so one domain is not hammered and one slow host cannot block the whole worker pool. The crawler must fetch and respect robots.txt, apply crawl-delay or internal policy limits, identify itself with a user agent, and avoid concurrent bursts to the same site. Globally we want high throughput, but locally each host should see a controlled, respectful request pattern; otherwise the crawler gets blocked or causes harm.

Failure cases are where the design becomes credible. If a worker dies after taking a URL, the queue item should be leased, not deleted, and retried after the lease expires, with retry limits and dead-letter handling. Crawler traps can generate infinite calendars or faceted-search URLs, so impose depth limits, URL-pattern filters, and per-host caps. If the seen-set is too large for memory, put a Bloom filter in front of durable storage to reduce lookups, while stating the tradeoff: false positives mean some pages are skipped, usually acceptable for broad crawling.