AI Crawlers Explained: Should You Allow or Block GPTBot?

You heard you can 'block AI' from your website, and it sounds prudent. Why let some bot hoover up your words to train a model that competes with you? But that single phrase, 'block AI,' hides a decision that can quietly backfire. Some AI crawlers train models. Others are how ChatGPT and Perplexity find you and cite you when a customer asks for a business like yours. Block the wrong ones and you don't protect your site, you erase it from the answers people now trust. Here's how to tell them apart and make a deliberate choice.

What is an AI crawler, and why should I care?

An AI crawler is an automated program that visits your web pages and reads them, the same way Google's crawler has done for years. The difference is what happens next. Some AI crawlers copy your content to help train a model. Others visit your pages so an AI assistant can quote you in a live answer, with a link, the moment someone asks a question you can help with. Both knock on the same door: a small file called robots.txt that tells bots what they may read. Understanding the two jobs is the whole game, because 'allow' and 'block' mean very different things depending on which crawler you're talking to.

What's the difference between training crawlers and AI search crawlers?

This is the split that matters, and almost no 'block AI' tutorial explains it. Line the major crawlers up and they fall into two families with opposite consequences for your visibility.

Crawlers that train models

These collect web content to improve the next generation of AI models. OpenAI is blunt about its training crawler: 'GPTBot is used to crawl content that may be used in training our generative AI foundation models,' and disallowing it 'indicates a site's content should not be used in training' (OpenAI). Blocking these keeps your words out of future model training. It does not remove you from the answers those models give today, because that's a different crawler's job.

• GPTBot (OpenAI) collects content that may be used to train OpenAI's foundation models.
• ClaudeBot (Anthropic) collects web content to train Claude's models.
• Google-Extended controls whether Google may use your content to train future Gemini models and for grounding. It's a standalone robots.txt token, and Google states it 'does not impact a site's inclusion in Google Search nor is it used as a ranking signal' (Google).

Crawlers that power live AI answers

These fetch and cite your pages when someone asks an AI assistant a question right now. OpenAI describes its search crawler plainly: 'OAI-SearchBot is used to surface websites in search results in ChatGPT's search features' (OpenAI). Perplexity says the same of its bot, which is 'designed to surface and link websites in search results' and 'is not used to crawl content for AI foundation models' (Perplexity). Block these and you can drop out of the AI search results your customers are already reading.

• OAI-SearchBot (OpenAI) surfaces and links websites in ChatGPT's search.
• PerplexityBot (Perplexity) indexes pages to surface and link them in Perplexity answers, and is not used for model training.
• Claude-SearchBot (Anthropic) crawls the web to improve the quality of Claude's search results (Anthropic).
• Googlebot is still the crawler behind Google Search and its AI Overviews. Blocking it is how you disappear from Google itself.
• User-triggered fetchers (ChatGPT-User, Perplexity-User, Claude-User) visit a single page only when a person asks the assistant to. They're actions on your behalf, not bulk crawling.

Will blocking AI crawlers make me invisible in AI answers?

If you block the search crawlers, yes. That's the trap in the 'block AI' advice. When a customer asks ChatGPT or Perplexity to recommend a plumber, a bakery, or a bookkeeper near them, those tools lean on their search crawlers to find current pages and cite them. If OAI-SearchBot and PerplexityBot can't read your site, you're not in the running. Perplexity notes that if you disallow its bot, it may still show your domain and a brief factual summary, but it won't use the full content of the page, so you show up thinner, if at all.

The reason this happens by accident is that most 'block AI' snippets floating around online paste in a long list that disallows everything with 'AI' in the name, search crawlers included. People run it to keep their content out of training and quietly switch off their AI-search visibility at the same time. That's the opposite of what a business trying to get found actually wants.

Should I block GPTBot and the training crawlers?

This one is a genuine judgment call, not a safety rule, so ignore anyone who says you 'must.' Blocking GPTBot, ClaudeBot, and Google-Extended keeps your content out of future model training. What you give up is small for most small businesses, because training crawlers aren't what earns you citations in live answers. What you gain is also modest: your public pages are already public, and blocking training doesn't claw back anything a model has already learned.

Blocking the training crawlers makes more sense if your pages hold genuinely original, hard-won material you'd rather not feed a model for free: in-depth guides, proprietary research, a large body of writing that is itself the product. Google-Extended is the low-risk one to block if that's your stance, since Google confirms it 'does not impact a site's inclusion in Google Search nor is it used as a ranking signal.' For a typical local business that mostly wants more customers, allowing the training crawlers is a perfectly reasonable default, and not the thing to lose sleep over.

How do I control AI crawlers with robots.txt?

All of this lives in one plain text file at yoursite.com/robots.txt, the same file that has guided search engines for decades. You write one short rule per crawler, naming the user-agent and whether it's allowed. If you can already open yoursite.com/robots.txt in a browser, you have one; if not, your site platform or a plugin can add it. This is different from an llms.txt file, which suggests your best pages to AI but can't allow or block anything. robots.txt is the one with the actual on/off switches.

The user-agent names are the important part, because a rule only applies to the exact name you write. Group them by the two jobs:

• Training crawlers you might choose to block: GPTBot, ClaudeBot, Google-Extended.
• Search crawlers you'll usually want to allow: OAI-SearchBot, PerplexityBot, Claude-SearchBot.
• Googlebot, which you should almost never block, because it powers Google Search and Google's AI Overviews.

Two honest caveats. robots.txt is an honor system: reputable crawlers from OpenAI, Anthropic, and Google respect it, but it's a request, not a lock, and a few crawlers have been caught ignoring it. And be careful with a stray 'Disallow: /' under the wrong heading, which can wave off far more than you meant, Google included. When in doubt, change one rule at a time and re-check the file in your browser afterward.

What's a sensible default for a small business?

Keep it simple. Unless you have a specific reason to hoard your content, let the AI world read the site you already publish, and pay real attention only to the training decision. A default that serves most small businesses looks like this:

• Allow the search crawlers, OAI-SearchBot, PerplexityBot, and Claude-SearchBot, so AI assistants can find and cite you.
• Never block Googlebot; that switches off Google Search and AI Overviews in one move.
• Block GPTBot, ClaudeBot, and Google-Extended only if keeping your content out of model training genuinely matters to you.
• Open yoursite.com/robots.txt and check you're not already blocking the search crawlers by accident.

And remember the file is the smaller lever. Being cited by AI depends far more on whether you have clear pages that plainly answer what customers ask, which is the real reason AI can't find some businesses. It's worth learning how to show up in AI search and get recommended by ChatGPT before you spend an afternoon tuning a text file. If keeping those pages current is the part you never get to, that's the gap Laspi fills: its WordPress plugin writes SEO and AI-search articles grounded in your real business, so the site those crawlers read stays clear and up to date.

So: allow or block GPTBot? Allow it unless you have a real reason not to, and whatever you decide, keep the search crawlers welcome. 'Block AI' was never one decision. It's two, and only one of them is about protecting your content. The other is about staying in the answers your customers already trust.