The 2026 robots.txt playbook for AI crawlers
Which AI bots to allow, which to block, and how to write a robots.txt that keeps you visible in AI answers without giving away everything. A practical playbook.
robots.txt used to be a quiet technical file. In 2026 it is a strategic one, because a growing list of AI agents check it before deciding whether your content can appear in their answers. Every line is now a commercial choice about which surfaces you show up in.
Know the three categories
- Answer and retrieval bots: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Bingbot. These feed live AI answers — the channels you sell into.
- Training crawlers: CCBot and similar. They feed the models of tomorrow rather than today's answers.
- Social preview bots: facebookexternalhit, Twitterbot, LinkedInBot, Slackbot. Block these and your links render as bare URLs.
A sane default
For most brands whose goal is AI visibility, the right default is permissive: allow the answer bots, allow social preview bots, and disallow only internal and API paths. Reserve blocking for genuine reasons — proprietary data, paywalled content, or bots that ignore the rules.
User-agent: GPTBot
Allow: /
Disallow: /api/
Disallow: /dashboard/
User-agent: PerplexityBot
Allow: /
Disallow: /api/
User-agent: *
Allow: /
Disallow: /api/
Sitemap: https://example.com/sitemap.xmlWhen to block on purpose
Blocking is legitimate when the content is a moat you do not want ingested, when licensing is unresolved, or when a crawler is hammering your servers. Just be clear-eyed about the trade: disallowing an answer bot removes you from that engine's citations entirely. That is sometimes right — but it should be a decision, not a default.
Common pitfalls
- Blocking Google-Extended and wondering why you vanished from AI Overviews.
- A stray global Disallow: / left over from a staging config.
- Forgetting to list your sitemap, so crawlers never get the URL inventory.
- Treating robots.txt as access control — it is a request, not a lock. Protect real secrets at the server.
Audit it regularly
New AI agents appear constantly. Review your robots.txt on a schedule, cross-reference it against the bots actually hitting your logs, and confirm your intended allow/block posture still holds. A free crawler check will flag answer bots you are unintentionally blocking.
Frequently asked questions
- Should I block AI bots to protect my content?
- Only with intent. Blocking answer bots removes you from their citations. Block for licensing, proprietary data, or abusive crawling — not by reflex.
- What is the difference between GPTBot and OAI-SearchBot?
- GPTBot is OpenAI's general crawler; OAI-SearchBot is specific to ChatGPT Search retrieval. Allow both if you want to appear in ChatGPT answers.
- Does robots.txt guarantee a bot obeys?
- No. Reputable, named bots comply, but the file is advisory. Enforce hard limits at the server or WAF for anything sensitive.