03.01 · Walkthrough
robots.txt, Status Codes, and Blocked vs Noindex
Choose correctly between disallow, noindex, a 404, a 410 and a canonical for a given page, and explain why a blocked page can still rank while a noindexed one cannot.
Crawling and indexing are separate decisions, so robots.txt, noindex, missing-page status codes and canonicals solve different problems. Disallow stops fetching, not necessarily search visibility. Noindex requires access to the page. Missing content should return the right status, while duplicate content usually needs consolidation rather than removal.
What this lesson answers
- robots.txt vs noindex which should I use
- why can a blocked page still rank
- when to use canonical instead of noindex
Notes
A search engine has two separate jobs: crawling and indexing. Crawling is fetching a URL; indexing is deciding whether that fetched content should be stored and eligible to appear in results. robots.txt controls crawling before the page is fetched. A noindex directive controls indexing after the page is fetched. That ordering is the heart of the lesson: if Google is blocked from fetching a page, it may never see the noindex instruction on that page.
Use disallow in robots.txt when you want to save crawl budget or keep crawlers out of areas whose content they do not need to fetch, such as…
References
Common questions
- Does robots.txt remove a URL from search results?
- No. robots.txt tells crawlers not to fetch matching URLs, but it is not an index removal instruction. If the URL is already known through links, sitemaps or earlier discovery, it can still appear with limited information because the crawler is blocked from reading any on-page directive.
- Why does noindex work differently from robots.txt disallow?
- Noindex is an indexing instruction found after a crawler fetches the page. That means the crawler must be allowed to access the URL before the directive can be seen and processed. A disallow rule happens earlier and can prevent the crawler from ever seeing the noindex instruction.
- Should duplicate pages be noindexed or canonicalised?
- Use a canonical when the duplicate or near-duplicate page should remain reachable but its signals should point to a preferred URL. Use noindex when the page itself should not be eligible to appear in search. Canonicalisation is for consolidation; noindex is for exclusion.