Block AI Crawlers in robots.txt: Google-Extended, GPTBot

If you want to block AI crawlers in robots.txt, the first thing to settle is which crawlers you actually mean. The phrase covers bots that build a search index, bots that gather text to train models and fetchers that visit a page because a person asked an assistant about it. A single rule rarely suits all three, and mixing them up is the most common way publishers lose visibility they wanted to keep or leave the door open on content they meant to protect.

This guide gives publishers and site owners a decision frame rather than a copy-paste blocklist. It separates crawlers by purpose, explains the tokens that the major providers document, shows allow and disallow examples, covers the limits of robots.txt and describes how to verify who is really visiting. Crawler names and behaviors change quickly, so every statement about a provider here comes from its own documentation as read on 2026-10-21. Reopen the linked pages before you edit a live file.

Search Crawlers Versus AI Training Crawlers

Think of automated visitors as three families. The first family crawls so that a search product can find your pages and show them in results. The second family collects content that may be used when a provider trains or improves its generative models. The third family fetches individual pages on demand, for instance when a user pastes a link into an assistant or asks it to look something up.

The families matter because the consequences differ. Blocking a search-oriented crawler can mean your pages are less likely to appear in that provider's search answers. Blocking a training-oriented crawler expresses a preference about model training and, depending on the provider, may have no direct effect on search visibility. User-triggered fetchers sit in a gray zone: some providers say their robots.txt handling differs for these requests.

A short mental model helps:

• Search family: trade visibility for access. Open doors bring you potential exposure.
• Training family: trade content use for control. Closed doors state a preference about how your text is used.
• On-demand family: one request at a time, initiated by a person, often outside robots.txt logic.

If you have not yet checked your file for accidental blocks, start with the robots.txt audit guide, which covers the general hygiene of the file. This article assumes the file works and zooms into one question: what should each named AI-related token be allowed to do on your site?

What Google-Extended Is and What It Does Not Affect

Google's common crawlers documentation (Opens in a new tab) describes Google-Extended as a standalone product token that publishers can use to manage whether content Google crawls may be used to train future Gemini models. The same page says it also covers grounding in Gemini Apps and Grounding with Google Search on Vertex AI.

Two statements in that documentation deserve to be remembered. First, Google-Extended does not impact a site's inclusion in Google Search, and it is not used as a ranking signal in Google Search. Second, it has no separate HTTP request user agent string: crawling happens with Google's existing user agents, and the token only acts as a control in robots.txt.

That has practical implications:

1. Disallowing Google-Extended is not described as a way to leave or damage your Google Search presence.
2. You will not see a separate Google-Extended user agent in your logs, so you cannot measure the token's effect there.
3. The scope of the token belongs to Google and can be revised. Always confirm in the current documentation which products it governs.

What this section deliberately does not claim is how Google-Extended relates to specific AI features inside Search. Unless Google's documentation says so in plain words, any statement about that relationship is speculation. If you need a firm answer about a particular Search feature, read Google's Search documentation about that feature and its controls instead of inferring from the token name.

Other Known Crawler Tokens and Their Documentation

Other providers publish their own crawler pages. What follows summarizes what those pages said on the access date above; treat it as a pointer to the documents, not a replacement for them.

OpenAI's bots documentation (Opens in a new tab) lists several agents. OAI-SearchBot is used to surface websites in ChatGPT's search features; the page recommends allowing it in robots.txt and notes that sites opted out will not be shown in ChatGPT search answers, with roughly a day needed for a robots.txt update to take effect. GPTBot is used for content that may help train generative foundation models, and the page says disallowing it indicates that a site's content should not be used in training. ChatGPT-User handles user-initiated actions, is described as not used for automatic crawling, and the page states that robots.txt rules may not apply to such requests.

Anthropic's support article describes ClaudeBot for training data, Claude-User for fetching content when a user asks Claude a question and Claude-SearchBot for improving search results, and says these bots honor standard robots.txt directives. Perplexity's documentation describes PerplexityBot as a search crawler that is not used to train foundation models, and Perplexity-User as a fetcher for user actions that generally ignores robots.txt rules. Both companies say they publish IP ranges.

The lesson is a habit, not a list. Before you write a rule for any token, open the provider's page and note three facts: what the bot is for, whether it follows robots.txt and how its traffic can be verified. Put the answers next to your decision, and re-read them when you review the policy.

How to Block AI Crawlers in robots.txt: Allow and Disallow Examples

The robots.txt file lives at the root of a host. Each group begins with a user agent or product token and is followed by rules. The snippets below only show syntax; decide the policy first.

Block a training token across the whole site:

User-agent: GPTBot
Disallow: /

Block one directory for a token while leaving the rest open:

User-agent: Google-Extended
Disallow: /research/
Allow: /

Keep a search-oriented crawler welcome while turning away a training crawler, using two separate groups:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

A few details prevent common mistakes. Start every group with its own user agent line and separate groups with a blank line, otherwise rules attach to the wrong token. Paths may be case sensitive, so match them to your real URLs. If you also have a general group that applies to every crawler, remember that a named group is normally treated as more specific; verify how each provider and Google process overlapping groups instead of assuming.

Also remember what robots.txt is not. It manages crawling; it is not the right tool to keep a page out of an index. To control indexing you use directives described in the noindex tag and X-Robots-Tag guide. A crawler that cannot fetch a page also cannot see a noindex tag on it, which is why the two controls should be planned together rather than stacked blindly.

The Scope and Limits of Blocking

Google's introduction to robots.txt (Opens in a new tab) frames the file as a way to manage crawler traffic. It says plainly that robots.txt is not a mechanism for keeping a page out of Google, and that obeying the rules is up to the crawler. Reputable crawlers follow them, others may not.

Apply that to AI crawlers and four limits appear:

• Compliance is voluntary. A provider may state that it honors robots.txt, but you cannot enforce it from the file. Real access restrictions need server-level controls such as authentication.
• Scope is per host. Anthropic's article says to add rules for each subdomain. If you run several subdomains, each needs its own file and review.
• It does not rewrite history. The file is a directive about future crawling. What happens to material collected earlier is governed by each provider's policies, and no provider documentation reviewed here promises deletion. Ask the provider through its official channels if that matters to you.
• User-initiated fetches may behave differently. When a person asks an assistant to open your page, some providers state that robots.txt may not apply, so plan for that in your policy.

There is one more limit that people forget: copies. Your text may be quoted, syndicated, archived or scraped elsewhere, and none of those copies read your robots.txt. If something must stay confidential, do not publish it. Treat robots.txt as a polite and useful signal, not as a lock, no matter how firm the wording of your rules feels.

Verifying Bots With Server Logs

A user agent string is just text, and anyone can send it. Google's documentation notes that the HTTP user agent string can be spoofed. Verification therefore has to rest on something harder to fake, such as network identity.

Google's documented method is a two-way DNS check: run a reverse lookup on the IP address in your log, confirm that the hostname belongs to googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that hostname and confirm it returns the original IP. For volume work, compare addresses against the IP range files Google publishes. OpenAI, Anthropic and Perplexity each say they publish IP ranges for their bots, so the same comparison approach applies.

A repeatable routine looks like this:

1. Filter access logs for lines whose user agent contains the bot name you care about.
2. Reduce the matches to unique IP addresses.
3. Check each address against the provider's published ranges or the DNS method.
4. Set unverified requests aside as possible impersonation.
5. Review verified requests: which URLs they hit, which status codes they received and how often they returned.

For the broader technique of reading logs for crawl behavior, server log analysis for SEO explains how to set up the data. Here you are applying that technique to a narrow list of names. One more caution: CDN and firewall products may classify and block bots on their own. If your edge rules disagree with your robots.txt, the bot may see an error rather than your policy. Keep both in one document so they do not drift apart.

The Business Decision: Visibility Versus Content Protection

With the facts in hand, the real question is commercial. What creates value on your site, and what do you gain from appearing inside AI products? Sort your content into groups. Open guides and promotional pages usually benefit from being discoverable. Original datasets, paywalled analysis and licensed material are the assets you may want to protect.

Most sites end up in one of three postures:

• Open: allow search and on-demand crawlers and allow training crawlers too. The goal is maximum reach and wide reuse is acceptable.
• Selective: allow search and on-demand crawlers, disallow training crawlers. Visibility stays while you state a preference on model training.
• Closed: disallow most known AI crawlers. Protection comes first, though some providers tie their search surface to the same rules, so you may lose a channel.

Whatever you choose, align it with your content strategy. If getting cited in AI answers is a goal, read the generative engine optimization guide for how content is shaped for those surfaces, and look at sponsored articles and brand mentions in AI search for the role of third-party mentions. Those pieces cover the content side. This article covers a different decision: who is allowed through the gate.

Take three questions to your next planning meeting. Is this content public by design? Does being cited in AI products bring us customers? Is our objection to training a matter of principle or of strategy? Write the answers down and let them drive the file, not the other way around.

Risks of Blocking the Wrong Bot

Most mistakes here are small typos with large effects. Some only reduce AI visibility, but a few can hurt ordinary search traffic.

1. A careless catch-all rule. Disallowing everything in a group that targets all crawlers can shut out search engines as well. Always retest search crawler access after an edit.
2. Confusing training with search. If you disallow a provider's search token when you only meant to refuse training, you may lose presence in its search answers.
3. Assuming one token behaves like another. We know from Google's page that Google-Extended does not affect Google Search. That tells you nothing about any other provider's tokens.
4. Using crawl rules for indexing control. To keep a page out of an index, use the right indexing directive.
5. Letting the firewall and the file disagree. A bot blocked at the edge but allowed in robots.txt gets errors and may retry repeatedly.

Reversals are easy but not instant. OpenAI says a robots.txt update can take about a day to be reflected, and Perplexity mentions a similar delay. Make changes when you have time to watch the effects, keep a dated copy of the previous file, and write down what you expected so you can compare it with what happens.

Testing and Monitoring

Publishing the file is the start of the work. Check that it is served from the root of the host with a successful status and as plain text. Then run sample URLs through a robots.txt parser to confirm which tokens are allowed or blocked for them, especially for the directories you protect.

A monthly monitoring routine is enough for most sites:

• Check whether the bots you restricted are still making requests, and verify them as described earlier.
• Report unverified requests that carry a bot name separately.
• Look for new crawl errors in your search console reports after each change.
• Sample how your brand and key pages are described in AI products and note changes.

The last item connects to how well systems understand who you are. If your organization, authors and topics are described consistently, mentions are easier to attribute correctly. The entity SEO guide on entities and topics explains how to build that clarity. Use what you find to interpret the effects of your crawler rules: look less at how often you are mentioned and more at whether the mention is accurate and in context.

Keep short dated notes: what changed, what you expected, what you observed. After a few cycles those notes show which rules earned their place. No outcome or timeline can be promised, because each provider decides how and when it reads your file.

A Policy Note and Maintenance

Keep your crawler decisions in a written policy instead of in one person's memory. It can be a single page: which bots are allowed, which are refused, the reason behind each decision, the date and the next review. Anyone who inherits the site, including an agency, can then see the logic behind the file rather than reverse engineering it.

Maintenance has three parts. First, review the list of bots regularly, because providers can add, rename or repurpose them. Pick an interval your team can sustain; every quarter is a reasonable starting point, but that is a suggestion, not a rule. Second, link each entry to its source document so you can recheck wording quickly. Third, make sure the policy fits your wider content principles.

That last point matters if you publish AI-assisted material yourself. How you use machine help in your own writing and how you treat other companies' crawlers should read as one consistent stance. Document your production steps, editorial responsibility and fact-checking alongside the crawler policy, using the principles in AI-written articles and human editing.

The policy page should always include a backup of the current robots.txt, the name of the decision owner, the last review date and a change log. An unmaintained policy is worse than none, because everyone assumes it is current.

Sample Template: AI Crawler Decision Table

The table below is filled in for a hypothetical publisher: a small news site with free guides and a paid research section. The decisions are illustrative values, so adapt them to your goals and recheck each row against the provider's current page.

In this hypothetical case the publisher keeps its free guides visible and keeps the paid research out of training pipelines it can influence. Adding a "last checked" column helps the next reviewer see which rows are stale at a glance.

Frequently Asked Questions

Does blocking Google-Extended affect Google Search?

According to Google's common crawlers documentation, no. Google-Extended does not impact a site's inclusion in Google Search and is not a ranking signal there. It is a control for whether crawled content may be used for Gemini training and grounding. Recheck the page, since scope can change.

Does blocking GPTBot remove you from ChatGPT?

Not necessarily. OpenAI's documentation ties GPTBot to model training and OAI-SearchBot to ChatGPT's search features, so they are controlled separately. Disallowing GPTBot states a training preference. Whether you appear in search answers depends on how you treat OAI-SearchBot.

Does a robots.txt rule delete data collected earlier?

The file is a directive about future crawling, and no reviewed provider documentation presents it as a deletion tool. What happens to earlier material is governed by each provider's own policies. If this matters, use the provider's official request channels rather than relying on robots.txt.

How do you verify a bot is really that bot?

Do not trust the user agent alone, because it can be spoofed. For Google, use the reverse and forward DNS check or the published IP range files. OpenAI, Anthropic and Perplexity say they publish IP ranges, so compare the addresses in your logs against those lists.

Is blocking every AI crawler recommended?

There is no blanket recommendation, since crawlers serve different purposes. Blocking search-oriented bots can reduce visibility in that provider's results. Decide by content type and business goal, separate training from search, and review the policy on a regular schedule.

How often should the bot list be reviewed?

Often enough to catch provider changes. A quarterly review is a reasonable starting point for many teams, though the right interval depends on your capacity and how much depends on the file. At each review, reopen the source pages, date your decisions and keep a backup of the previous file.

Conclusion and Next Step

Handling AI crawlers well is less about editing a text file and more about deciding what your content is for. Split bots into search, training and on-demand families, read each token's documentation, turn your choices into a short written policy and confirm behavior with logs rather than names alone.

A sensible first step this week: back up the current file, fill in the first three rows of the decision table and look for those bots in your logs. If you want help drafting the policy page or an explainer for your readers, you can look at the content writing listings. You pick a service, add it to the cart and pay, the seller approves, a delivery notice follows and an objection route exists if the result is not as described. The decision itself stays with you and your team.