How to Block AI Training Without Disappearing From AI Search

I get asked a version of this question a lot at the moment: "can I stop AI companies training on my website without disappearing from ChatGPT?" The honest answer is yes, mostly, but only if you understand that "AI training" and "AI search" are handled by completely different crawlers, often from the same company. Block the wrong one and you don't protect anything. Block the right one and you can still show up when someone asks ChatGPT or Perplexity a question that your site would have answered.

Two jobs, two crawlers

Most AI companies run at least two separate bots, and they behave nothing alike.

• Training crawlers hoover up content to build or fine-tune a model. OpenAI's is called GPTBot. Anthropic's is ClaudeBot. Common Crawl, which many labs use as a base dataset, runs CCBot. Google has Google-Extended, which controls training for Gemini and other Google AI products, separate from normal Googlebot indexing.
• Retrieval or search crawlers fetch a page in the moment, to answer a specific question or show it in a result. OpenAI's is OAI-SearchBot. Perplexity runs PerplexityBot for this. These aren't building a model, they're answering someone right now, closer to how a search engine works than how a training run works.

If you block GPTBot but leave OAI-SearchBot alone, you can opt out of training while still turning up when someone asks ChatGPT a question your site answers. Block both by mistake, using one blanket "AI" rule, and you've quietly removed yourself from a channel that's growing fast.

What actually controls this

It's all done through robots.txt, the same plain text file that's controlled search engine crawling for decades. You name the user agent and say whether it can fetch your pages:

• User-agent: GPTBot / Disallow: / stops OpenAI training on your content.
• User-agent: OAI-SearchBot / Allow: / keeps you visible in ChatGPT's live search results.
• User-agent: Google-Extended / Disallow: / opts out of Gemini training without touching your normal Google Search ranking, which is handled separately.

Most of the major AI companies publish which user agent does which job. It's worth checking the current list rather than trusting a plugin or a WordPress setting that claims to "block AI" with one toggle, because that toggle usually blocks everything indiscriminately, or nothing at all.

Where this gets messier

A few things worth knowing before you assume the job is done:

• robots.txt is a request, not a lock. Well-behaved crawlers respect it. Nothing forces a scraper to. If your content genuinely matters to you, robots.txt reduces exposure, it doesn't guarantee anything. I've written before about how content gets scraped without any system being breached, and the same logic applies here.
• AI Overviews aren't the same as Google-Extended. Google's AI Overviews in normal search results are built from the standard Google Search index, controlled by Googlebot, not by Google-Extended. Blocking Google-Extended stops Gemini-the-product training on you, it does nothing to remove you from AI Overviews. If you want out of both, you need to disallow both user agents, and understand you'll likely also affect your regular search ranking, since they overlap in what they crawl.
• llms.txt is not standard yet. You'll see people recommending an llms.txt file to describe your site to AI tools directly. It's a genuinely useful idea and some tools do read it, but it isn't an agreed web standard the way robots.txt is, so treat it as a bonus, not a substitute.
• Bot lists change. New user agents appear as new AI products launch. A rule you set six months ago might be missing the crawler a newer tool uses. Worth a five-minute check every quarter rather than a "set and forget" file.

A sensible default for most businesses

If you run a site with content you'd like to keep out of training data, but you'd still like to be findable when someone asks an AI assistant a question, a reasonable starting position is:

• Disallow the known training bots: GPTBot, ClaudeBot, CCBot, Google-Extended, and any others relevant to models you specifically want to opt out of.
• Allow the known retrieval/search bots: OAI-SearchBot, PerplexityBot, and standard search engine crawlers like Googlebot and Bingbot.
• Check your current robots.txt against a live list of AI user agents rather than a copy-pasted template from a blog post, since these change.

This isn't a decision you make once and forget. If your business relies on being found, whether that's through Google, an AI assistant, or a client Googling your name before a call, it's worth understanding which crawler does what before you block anything. I covered the related question of what to actually check before opting your business data out of AI training in more depth, which is worth reading alongside this if the two topics have been getting mixed up for you, because they usually are.