AI crawlers: who's reading your website, and who should you let in?

Search engines and AI companies send programs to read websites. They're called crawlers (Google). Some read your pages so they can show them in answers. Some read them to train AI. Some come only because a person asked an AI to open one of your pages. OpenAI, for one, runs a separate visitor for each of the three (OpenAI).

You can decide which of them get in. Start with what you want.

What do you want?

I want my business to show up in AI answers

We suggest you let in the search crawlers: OAI-SearchBot for ChatGPT, Claude-SearchBot for Claude and PerplexityBot for Perplexity, along with Googlebot and Bingbot.

Here is why. Sites that block OpenAI's search crawler "will not be shown in ChatGPT search answers" (OpenAI). Blocking Anthropic's "may reduce your site's visibility" in Claude's search (Anthropic). And Perplexity recommends allowing its crawler if you want to appear in its results (Perplexity).

I don't want my pages used to train AI

You can say no to training and still be found. Each company has a name for its training use: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Applebot-Extended (Apple) and Meta-ExternalAgent (Meta). We suggest you block the ones you mind.

OpenAI, Google and Apple each say that blocking their training name doesn't take you out of their search (OpenAI, Google, Apple). One thing to know: Anthropic's page talks about leaving out your "future materials"; it doesn't say what happens to pages it has already collected (Anthropic).

I want to stay in Google, but not in its AI answers

Don't block Googlebot for this: its rules cover all of Google Search, not only the AI parts (Google).

Use the switch Google made for it instead. In Search Console, under Settings, the Search generative AI control decides whether your site appears in AI Overviews and AI Mode. It has been open to every website since 31 August 2026. Google says a change "generally takes a few days" to work, and some content may take longer (Google). Sites that switch it off "will not receive traffic or impressions" from those features (Google).

For Microsoft's Copilot, the tool is a tag on the page called NOARCHIVE, not robots.txt. It keeps the page out of Copilot's answers and its grounding, the look-up behind an AI answer (Microsoft).

My site runs on Cloudflare

Look at Cloudflare first, because it has its own settings for AI visitors. Since 15 September 2026, choosing "Block" there stops Googlebot, Bingbot and Applebot too, "search included". To stop training and keep search, Cloudflare says to choose "Disallow AI Training" (Cloudflare). You'll find these under Security Settings for your domain (Cloudflare). Cloudflare's help page, updated 2 November 2026, lists that choice as "Disallow", for training only (Cloudflare docs).

See what your site says today

Type yoursite.com/robots.txt into your browser. You'll see a short text file, your robots.txt. It tells crawlers which pages they may visit (Google).

Each crawler is named on a User-agent: line. Anthropic's own example for keeping its crawler out looks like this (Anthropic):

User-agent: ClaudeBot
Disallow: /

Compare the names in your file with the list further down. For more tests you can run yourself, see Check your site.

Good to know

It's a request, not a lock. The standard behind robots.txt says crawlers are "requested to honor" it, and that it is "not a form of access authorization" (IETF, RFC 9309).
Some visitors skip it when a person asks. OpenAI says robots.txt "may not apply" to ChatGPT-User (OpenAI); Perplexity says Perplexity-User "generally ignores" it (Perplexity); Meta says one of its fetchers "may bypass" it (Meta).
Names can be faked. A visitor's user agent, the name it gives your site, is just text: Google warns it "can be spoofed" (Google). Google, Microsoft, OpenAI, Anthropic, Perplexity, Apple and Common Crawl each publish a way to check whether a visitor really is theirs, on the pages linked below.
Apple copies your Google rules. If your file doesn't name Applebot but does name Googlebot, Applebot follows the Googlebot rules (Apple).
Changes take about a day. OpenAI says about 24 hours for search (OpenAI); Perplexity and Meta say up to 24 hours (Perplexity, Meta).
One open question. We found no published Google statement on whether blocking Google-Extended changes whether your site appears in AI Overviews or AI Mode (what we searched, 29 October 2026). For those, use the Search Console switch above.

Who's who

Google

Googlebot reads pages for Google Search, "including Discover and all Google Search features". Its rules apply to all of them (Google).
Google-Extended isn't a visitor. It's a name you can use in robots.txt to keep your pages out of training Google's Gemini models, and out of the look-ups behind Gemini's answers. Google says it "does not impact a site's inclusion in Google Search" (Google).

Microsoft

Bingbot is "our standard crawler", Microsoft says (Microsoft). Its guidelines cover Bing search and Copilot; for Copilot, they point to the NOARCHIVE tag (Microsoft).

OpenAI (ChatGPT)

OAI-SearchBot finds pages for ChatGPT search. Blocked sites won't be shown in ChatGPT search answers, "though can still appear as navigational links".
GPTBot collects pages that may be used to train OpenAI's models. Blocking it is a no to training, and has no effect on OAI-SearchBot: "Each setting is independent of the others".
ChatGPT-User may open a page when someone asks ChatGPT a question; robots.txt "may not apply".

All three: OpenAI.

Anthropic (Claude)

Claude-SearchBot finds pages for Claude's web search.
ClaudeBot collects pages that may be used to train Anthropic's models.
Claude-User may open a page when someone asks Claude a question.

Anthropic says blocking the search crawler or Claude-User may make your site less visible in Claude's answers. All three: Anthropic.

Perplexity

PerplexityBot finds pages for Perplexity's answers, and is "not used to crawl content for AI foundation models".
Perplexity-User may open a page when someone asks Perplexity a question, and "generally ignores robots.txt rules".

Both: Perplexity.

Apple

Applebot reads pages for search in Spotlight, Siri and Safari, and its data may also be used to train Apple's models.
Applebot-Extended isn't a visitor. It's a name you use in robots.txt to say no to training; pages that use it "can still be included in search results".

Both: Apple.

Common Crawl

CCBot belongs to Common Crawl, a non-profit that publishes "an open repository of web crawl data" that anyone can use (Common Crawl).

These pages were dated: Google 30 October 2026, Anthropic 7 April 2026, Apple 4 September 2026, Meta 21 May 2026. OpenAI's, Microsoft's, Perplexity's and Common Crawl's carry no date.

Questions

If I block GPTBot, will I disappear from ChatGPT? Not from ChatGPT search. OpenAI gives this exact case: a site "can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot" (OpenAI).

Will blocking Google-Extended keep me out of AI Overviews? We found no published Google statement either way (what we searched). Google's switch for AI Overviews is the one in Search Console (Google).

Can I stop an AI from opening my page when someone asks it to? Not reliably with robots.txt: OpenAI and Perplexity both say their visitors that open a page for a person may not follow it (see "Good to know" above).

Checked on 3 November 2026.