Crawler: Frontier, Budget and Politeness
Crawler frontier, budget and politeness are the scheduling rules that decide which discovered URLs a search crawler fetches, how much crawling a site receives, and how gently requests are sent. The crawler maintains candidate URLs, prioritises them, limits work per site, and adapts to server behaviour, page value and duplication.
A crawler needs these controls because the web is effectively unbounded from its point of view. One page can lead to many more, URL parameters can generate endless variants, and some sites change faster than others. At the same time, the crawler has finite machines, bandwidth and time, while the target site has finite capacity. Without scheduling, the crawler would waste effort on duplicates, stale low-value pages or traps, and could accidentally behave like an attack against a host.
The frontier is the crawler’s working set of known but not yet fetched URLs. New candidates arrive from links, sitemaps, redirects and previous crawl history. The scheduler scores them using signals such as expected importance, freshness, prior failures, duplication risk, host limits and whether similar URLs have already been seen. Fetching a URL can add more candidates, update existing priorities, or reduce confidence in a part of the site if it returns errors or produces low-value variants.
Crawl budget is the practical result of that scheduling, not a fixed allowance that a site can simply spend. It reflects both crawl demand and crawl capacity: how worthwhile the crawler thinks more fetching is, and how safely the server appears able to handle it. Politeness trades completeness and speed for safety. Backing off on slow or erroring servers can delay discovery, but ignoring those signals would harm sites and waste crawler resources.
Engineers meet this in server logs, robots.txt, sitemap design, faceted navigation, rendering pipelines and Search Console crawl reports. Googlebot may request the document itself and, when needed to understand the page, supporting resources such as scripts, stylesheets and images. A common misunderstanding is that discovery means indexing. A URL can be known, queued, fetched, rendered, indexed or ranked, and each stage can decline or delay it for different reasons.
Common questions
- Does submitting a sitemap guarantee crawling?
- No. A sitemap is a discovery and hinting mechanism, not a command. It helps a crawler find canonical-looking URLs and understand update signals, but the scheduler still decides whether and when to fetch them. Site quality, duplication, server health, internal linking and prior crawl outcomes all affect whether those URLs receive attention.
- What makes a crawler reduce attention to a site?
- Common causes include repeated server errors, slow responses, large numbers of duplicate URLs, infinite parameter spaces, weak internal linking and pages that rarely change or appear low value. The crawler learns from what previous fetches produced. If more crawling mostly finds waste or risk, the frontier will usually prioritise other work.
- Does Googlebot only download HTML?
- No. The initial request may be for the HTML document, but understanding a modern page can require other resources. A crawler may fetch JavaScript, CSS, images, redirects and URLs discovered during parsing or rendering. Whether those resources are fetched depends on the crawler’s rendering process, permissions, prioritisation and the value of doing so.
- Is crawl budget something engineers can directly set?
- Not directly. Engineers influence it by making important URLs easy to discover, reducing duplicate and infinite URL spaces, returning correct status codes, improving server reliability and avoiding blocked resources needed for understanding. The honest answer is that it depends on both site demand and safe fetch capacity, as observed by the crawler.