Robots.txt for AI Crawlers: What to Allow and Block

By Om Usturge · Updated October 31, 2026 · 3 min read

Short answer: If you want AI search tools to cite your pages, don't block their search crawlers. Many companies run separate bots for training data and for search or user requests, so you can allow one and block another. Robots.txt is a polite request, not a lock, and it can't hide anything private.

First, the honest part: bot names and behaviours change. Check each company's documentation before you copy a rule. The names below are the common ones as of October 2026.

Training bots vs search bots vs user bots

  • Training crawlers collect content to train models.
  • Search crawlers index pages so an AI search product can show and cite them.
  • User-triggered fetchers open a page because someone asked the assistant about it.

Blocking a training bot doesn't necessarily stop an assistant from citing you through its search bot. Blocking the search bot can make you invisible in that product.

Common user agents (verified October 2026)

CompanyUser agentRole
OpenAIGPTBottraining
OpenAIOAI-SearchBotsearch index
OpenAIChatGPT-Useruser-triggered
AnthropicClaudeBottraining
AnthropicClaude-SearchBotsearch quality
AnthropicClaude-Useruser-triggered
PerplexityPerplexityBotsearch index
GoogleGoogle-Extendedcontrol token for AI training and grounding use
Common CrawlCCBotopen web dataset

Google-Extended is a control token, not a separate crawler. Googlebot still crawls your site for Search, and Google has said this token doesn't affect Search ranking.

What wizetool.com actually does

This is wizetool.com's real robots.txt, as served on 31 October 2026:

User-agent: *
Allow: /
Disallow: /api/

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://wizetool.com/sitemap.xml

The decision behind it: the site is written to be quotable, so search and user fetchers are explicitly allowed. The only thing blocked is /api/, which serves tool endpoints rather than citable pages. Training bots are left allowed by the User-agent: * rule — that can be changed later without touching the search rules.

A sample file for a site that wants citations

User-agent: *
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://wizetool.com/sitemap.xml

If you'd rather opt out of training only, add separate groups:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

This is a business decision. Free tool sites that want reach often allow more. Publishers selling content often block training.

How to test

  1. Open https://yourdomain.com/robots.txt and confirm it loads with a 200 status.
  2. Check for syntax slips: one blank line between groups, correct spelling of bot names.
  3. Use Search Console's robots.txt report to confirm Google reads it.
  4. Check your server logs after a few weeks to see which bots visit. Bots that ignore robots.txt will still appear.

What robots.txt can't do

  • It doesn't remove pages already indexed (use noindex or removal tools).
  • It isn't security. Anything public can be fetched.
  • It doesn't guarantee a bot obeys. Well-behaved ones do.
  • It doesn't create citations. Quality and clarity decide that. See how to get cited by AI search.

Mistakes to avoid

  • Disallow: / under User-agent: * left over from staging.
  • Blocking CSS or JS files the page needs.
  • Blocking a bot and then wondering why that product never cites you.
  • Copying a long block list from a random site without understanding it.

Quick checklist

  • robots.txt loads at the root
  • No leftover staging block
  • Decision made: allow or block training bots
  • Search and user bots allowed if you want citations
  • Sitemap line present
  • Bot names verified in vendor docs this month

FAQ

Will allowing AI bots get me cited? No promise. It removes one barrier. Content quality and structure still matter, as covered in what is GEO.

Should I block everything to protect my content? It's your call. Understand that blocking search bots reduces visibility in those products.

Where does llms.txt fit? It's a separate idea. Read llms.txt: does it help?.

Sources

Related guides