Module 03
Being Retrievable At All
Everything in module 1 and 2 is unreachable if the page is never fetched, never indexed, or never rendered. This is the floor.
6 lessons · 15 videos · 6h 13m
- 03.01
robots.txt, Status Codes, and Blocked vs Noindex
Choose correctly between disallow, noindex, a 404, a 410 and a canonical for a given page, and explain why a blocked page can still rank while a noindexed one cannot.
- 03.02
The AI Crawlers: GPTBot, OAI-SearchBot, Google-Extended
Write a robots.txt that allows the crawlers which make you citable while refusing the ones that only take training data, and explain what each decision costs.
- 03.03
Rendering: How an SPA Goes Invisible
Diagnose whether content exists in the HTML, after rendering, or nowhere a crawler will see it, and pick between SSR, prerendering and hydration for a given page.
- 03.04
Canonicals, Duplicates, and the URL as a Permanent Key
Resolve a duplicate-content situation with the right combination of canonical, redirect and parameter handling, and explain why a URL is a contract you cannot casually break.
- 03.05
Site Architecture: The Graph You Actually Control
Design an internal link structure that puts your most important pages within reach of a crawler and a reader, and explain why this is the only part of the link graph you own.
- 03.06
Sitemaps, IndexNow, and the Path Into Bing
Publish a change and get it into an index the same day, and explain why Bing's index being the backing store for ChatGPT Search and Copilot makes Bing Webmaster Tools an AI-search surface.