01.02 · Concept · Free
The Crawler: Frontier, Budget and Politeness
Describe how a crawler chooses what to fetch next, why crawl budget exists at all, and what Googlebot actually requests when it visits a page.
Curated for this lesson1/3
How Search Actually Works
How Googlebot Crawls the Web
Directly targets Googlebot crawling at concept depth from Google’s own search education channel.
A crawler is a scheduler for fetching URLs, not a simple queue. It prioritises known and newly discovered URLs against freshness, value, duplication, failures, host limits and site behaviour. Crawl budget exists because search engines and sites both have finite capacity, so reliable, useful, changing pages are fetched more readily than wasteful or harmful URL spaces.
What this lesson answers
- how does a search crawler choose URLs
- why does crawl budget exist for Googlebot
- what resources does Googlebot fetch from pages
Notes
A web crawler is the fetching part of a search engine. It starts with known URLs, keeps a frontier of URLs it may fetch later, chooses the next ones according to priority, downloads resources, extracts more links, and repeats. The frontier is not just a queue; it is a scheduling system that balances freshness, importance, duplicates, failures, host limits, and what the search engine already knows about a site.
Crawl budget exists because both sides have limits. Google cannot fetch the entire web constantly, and sites cannot safely receive unlimited automated traffic. A crawler therefore decides how much attention a site is worth and how fast it can be visited without causing harm. Popular, changing, reliable pages may be revisited often; low-value, duplicate, slow, erroring, or infinite URL spaces may be crawled less.
Politeness means the crawler behaves like a careful distributed client, not like a load test. It spaces requests, respects site-level constraints such as robots.txt where applicable, backs off when servers fail or slow down, and avoids overwhelming one host just because many URLs are available. Googlebot may request the HTML page, but also resources needed to understand it, such as JavaScript, CSS, images, and other fetched URLs discovered during rendering or link extraction.
A common misconception is that submitting a URL or having a sitemap means Google will crawl and index everything immediately. In reality, discovery, crawling, rendering, indexing, and ranking are separate stages, each with its own limits and decisions. After this lesson, you should be able to explain why a crawler skips or delays pages, why infinite URL combinations are dangerous, and why server performance and site structure affect crawl behavior.
References
Common questions
- What is a crawler frontier?
- A crawler frontier is the set of URLs a search engine knows about but has not necessarily fetched yet. It is better understood as a scheduling system than a queue. The crawler ranks candidates using signals such as importance, freshness, duplication, failures, host constraints and prior knowledge of the site.
- Why might Googlebot not crawl every URL on a site?
- Googlebot has to decide where fetching is worthwhile and safe. Duplicate pages, low-value paths, slow responses, repeated errors and endless parameter combinations can reduce crawl attention. A sitemap or submitted URL can help discovery, but it does not force immediate crawling, indexing or ranking.
- Does Googlebot only request the HTML document?
- No. Googlebot can fetch the HTML and other resources needed to interpret the page, including JavaScript, CSS, images and URLs found through rendering or link extraction. What it fetches depends on what the crawler needs, what the site allows and how the page is discovered and processed.
Short definition: what is Crawler: Frontier, Budget and Politeness?