robots.txt and AI Crawlers: Who to Let In, Who to Block

Who visits your website? Google? Yes. OpenAI, Anthropic, Perplexity, and Microsoft? Probably. But do you want them to have access? robots.txt is the answer — the file that tells every bot: "you can go here, you cannot." In this article, we'll show you which AI crawlers exist, what each one does, and how to manage them from a GEO perspective.

Who's crawling your site? (GPTBot, ClaudeBot, PerplexityBot, Google-Extended)

Four major AI crawlers are currently active on the web:

1. GPTBot (OpenAI): This powers ChatGPT's training and browsing functions. GPTBot performs live web crawls to find fresh content for ChatGPT's browsing feature. It's active and performant.
2. Claude-Web (Anthropic): This powers Anthropic's Claude AI. Like GPTBot, Anthropic performs web crawls to gather fresh information for Claude.
3. PerplexityBot (Perplexity AI): Perplexity is a search-engine-like AI platform that pulls from live web pages. Your site is nearly guaranteed to be visited if you allow PerplexityBot.
4. Google-Extended (Google): OpenAI, Anthropic, and other AI companies share training data with Google or vice versa. Google-Extended is needed for Google AI Overviews, a separate token Google uses when indexing your site.

Block or allow? The GEO perspective

If GEO is the goal — getting AI engines to cite you — the recommendation is: allow all of the above bots. Blocking GPTBot, Claude-Web, and PerplexityBot means your new content never reaches those AI engines. This breaks your entire GEO strategy.

Blocking Google-Extended affects AI Overviews but doesn't impact traditional Google search. However, if you want to appear in AI Overviews, you need to allow Google-Extended too.

Of course, if your site's content (like patented research or paywalled material) shouldn't be used by AI, you can block everything. But nothing in a GEO strategy requires denying permission.

How to write your robots.txt rules

A basic robots.txt structure looks like this:

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

This is the default: allow all bots. If you want to block a bot, write:

User-agent: GPTBot
Disallow: /

This tells GPTBot not to crawl any part of your site. If you want to block certain sections (like `/admin/` or `/proprietary-data/`), write:

User-agent: GPTBot
Disallow: /admin/
Disallow: /proprietary-data/
Allow: /

The `Allow: /` at the end says: "everything else is OK".

What robots.txt does NOT fix

Important: robots.txt only directs future crawls. If an AI engine has already trained its model on your content (visited it before), changing robots.txt won't "delete" that. This means even if you block GPTBot today, GPT already has information about your site it can retain.

Second, robots.txt doesn't prevent indirect data collection. Information the AI finds about you elsewhere — on Twitter, Wikipedia, or third-party sites — can't be controlled from robots.txt. AI engines draw from multiple sources, and robots.txt only affects crawls directed at your site.

The practice: recommended robots.txt for GEO

If GEO is your goal, a recommended robots.txt looks like this:

User-agent: *
Allow: /

# AI crawlers — allowed for GEO
User-agent: GPTBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

# Disallow specific parts if needed
User-agent: *
Disallow: /admin/
Disallow: /private/

This says: "For all AI bots, all content is OK. Admin and private sections are not."