03.02 · Concept
The AI Crawlers: GPTBot, OAI-SearchBot, Google-Extended
Write a robots.txt that allows the crawlers which make you citable while refusing the ones that only take training data, and explain what each decision costs.
No video curated for this lesson yet
This lesson is written, ordered and part of the path - the video slot is the only thing still open. We are working through SEO lesson by lesson; 39 of 52 have their video so far.
The written notes below cover this idea in full - you lose nothing by reading instead of watching.
AI crawler access should be split by use: allow crawlers that can make pages retrievable and citable, and block crawlers mainly used for model training or product improvement. A good robots.txt names agents explicitly, avoids damaging search indexing, and treats the file as policy, not security.
What this lesson answers
- which AI crawlers should robots.txt allow
- block GPTBot without blocking OpenAI search
- Google-Extended versus Googlebot for SEO
Notes
Robots.txt is a public policy file at the root of your site that tells well-behaved crawlers which paths they may fetch. For AI visibility, the important distinction is not “AI versus not AI,” but “what will this crawler do with my pages?” Some crawlers make your content retrievable in search-like answers with links or citations; others primarily collect text for model training or product improvement. Your job is to decide which outcomes you want, then express that as user-agent rules.
A practical mental model is to split crawlers into buckets. OAI-SearchBot is meant for OpenAI search and…
Common questions
- Should I block all AI crawlers in robots.txt?
- No. A blanket block can remove the crawlers that help your pages appear in search-like AI answers with links or citations. The better approach is to decide by crawler purpose: allow retrieval and citation agents if you want visibility, and refuse agents mainly used for training or model improvement.
- What is the difference between GPTBot and OAI-SearchBot?
- OAI-SearchBot is associated with search and retrieval experiences, where allowing access can help your pages become cited sources. GPTBot is associated with collecting content for improving or training models. Blocking GPTBot may reduce training use, but it can also reduce how familiar future systems are with your site.
- Does Google-Extended affect normal Google Search indexing?
- Google-Extended is separate from ordinary Googlebot. It is used to express whether your content may be used in Gemini and related AI systems. Blocking Google-Extended should not be treated as blocking Google Search, but your robots.txt must keep the rules explicit so you do not accidentally restrict indexing.