Verify Googlebot: Fix WAF and CDN Crawl Blocks

Googlebot access means that your server, CDN and firewall let Google's crawlers fetch your pages, so that nothing between the crawler and your content turns a legitimate visit into an error. When access breaks, the symptoms are easy to see in Search Console and hard to explain: server errors, pages that cannot be fetched, or a crawl rate that drops without a deploy to blame. Teams often re-read robots.txt several times and find nothing, because the blocker sits in front of it, in a WAF rule, a bot-management setting or a rate limit.

This guide is written for technical SEOs, developers, and hosting or security administrators who share the problem and rarely share a vocabulary. It separates the kinds of Google requests you will see in logs, shows how to verify that a request is genuinely from Google, explains why protective layers misjudge legitimate bots, untangles what 403, 429 and 503 mean to Google, and ends with a repeatable way to investigate and document an access incident. It stays product-neutral: no vendor screen, price or setting name is assumed.

Google Crawlers and Googlebot Types

Not every request labelled "Google" in a log is the same kind of client. Google's overview of its crawlers (Opens in a new tab) sorts them into three families, and the family decides which rules apply and what you should permit at the edge.

Three families, three behaviors

• Common crawlers serve Google's products, and Googlebot belongs here. Google says they always respect robots.txt rules for automatic crawls.
• Special-case crawlers work for specific products, usually under an agreement with the site being crawled. They may or may not respect robots.txt; the documentation notes, for example, that AdsBot ignores the global user-agent wildcard with the ad publisher's permission.
• User-triggered fetchers run because a person pressed a button in a tool, such as Google Site Verifier. Google says these fetchers ignore robots.txt rules.

The practical point is that a rule written as "allow Google" is too vague to be safe. A user-triggered fetcher ignoring robots.txt is by design, since a human asked for that fetch; it is not evidence that your robots.txt is broken.

Googlebot is two crawlers under one name

Per Google's Googlebot page (Opens in a new tab), the name covers a smartphone crawler that imitates a mobile user and a desktop crawler that imitates a desktop user. They share one robots.txt product token, so you cannot address only one of them in robots.txt. Because Google mainly indexes the mobile version of most sites, the smartphone crawler produces most of the requests. If an edge rule treats anything with a mobile user agent and no cookies as suspicious, it will land on exactly the traffic that matters most.

Size limits and transport details that look like blocks

A few technical limits can imitate an access problem. For Search, Googlebot fetches the first 2 MB of a supported file type, including HTML, and 64 MB for PDFs; referenced CSS and JavaScript files are fetched separately under the same limit, which applies to uncompressed data. The general crawler documentation states a default of the first 15 MB per file and adds that individual projects can set different limits. A bloated HTML document can therefore push important links or content beyond the point the crawler reads.

The same overview says crawlers support HTTP/1.1 and HTTP/2, with HTTP/1.1 as the default, and accept gzip, deflate and Brotli encodings, advertised through the Accept-Encoding header. Responding with a 421 over HTTP/2 is the documented way to opt out of HTTP/2, which the page describes as a temporary workaround. If an origin or CDN mishandles one of these, the result can resemble a security block even though the root cause is protocol configuration.

Verifying Real Googlebot: Reverse DNS and Official IP Lists

A user agent string is a claim, not an identity. Google's Googlebot page says the header is often spoofed by other crawlers and should not be relied on alone. The verification documentation (Opens in a new tab) offers two methods, one for individual lookups and one for automation.

Method one: reverse then forward DNS

Google describes this as sufficient for most use cases. Work through it in order:

1. Take the accessing IP from your log and run a reverse DNS lookup with the host command.
2. Check that the returned domain is googlebot.com, google.com or googleusercontent.com.
3. Run a forward DNS lookup on that returned domain.
4. Confirm the result matches the original IP from the log.

The forward step is what makes the check meaningful. Whoever controls an IP block can often set its reverse record to anything; only the forward lookup proves that the named Google domain really points back to that address. The documentation illustrates the procedure with the addresses 66.249.66.1, 35.247.243.240 and 66.249.90.77.

Hostname patterns also tell you which family you are looking at. Common crawlers resolve to names like crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. Special-case crawlers show rate-limited-proxy-***-***-***-***.google.com. Google-controlled user-triggered fetchers use google-proxy-***-***-***-***.google.com, while the other user-triggered fetchers resolve under gae.googleusercontent.com.

Method two: match published IP ranges

For large-scale lookups Google recommends comparing the requesting address with its published JSON files, which list ranges in CIDR notation. There is a separate file per family: common-crawlers.json for crawlers like Googlebot, special-crawlers.json for special-case crawlers, user-triggered-fetchers.json and user-triggered-fetchers-google.json for user-triggered fetchers, and user-triggered-agents.json for user-triggered agents. For other Google addresses, such as those Apps Script uses, the general list is goog.json on gstatic.com.

The page does not say how often these files change. Do not paste a snapshot into a firewall and forget it; build a scheduled refresh from the official source, and review the documentation periodically for changes.

Choosing between the two

Think of method one as the investigator's tool and method two as the automation's tool. A person reading a suspicious log excerpt can resolve a handful of addresses in minutes. A firewall exception or a log filter that must classify millions of requests cannot afford a DNS round trip per request, so it should compare against ranges, with periodic manual spot checks to catch drift.

What verification does and does not tell you

A verified address proves Google infrastructure sent the request. It does not tell you the request is a ranking-relevant crawl, and it does not oblige you to serve unlimited volume. If a verified crawler is overwhelming a weak origin, the answer is a controlled slowdown through the status codes described later, not a block that Google will read as content being unreachable.

Why WAF, CDN and Bot Protection Block Legitimate Bots

Protective layers decide, request by request, whether the client is a person, a tolerable automation or an attack. Googlebot is an automation. Defaults tuned to stop scrapers and credential stuffing can therefore catch it, and so can emergency rules added during an incident and never revisited. Below are the patterns worth checking; which one applies to you is something only your own logs can say.

Rate rules and bursts

A rule such as "block any IP exceeding N requests per window" counts requests, not intent. Googlebot can crawl a section of a site densely for a while, particularly after a sitemap update or a template change that touches many URLs. The symptom is a cluster of 429 or 403 responses at the same hours each day rather than a steady trickle.

Header and fingerprint checks

Some rules reject clients whose request shape does not look like a mainstream browser. A crawler that declares itself openly as a bot may fail a check designed to expose headless browsers. The opposite mistake is just as costly: a rule that waves through anything whose user agent contains the word Googlebot gives every scraper a free pass.

Reputation lists and geographic rules

Country allowlists and IP reputation feeds can sweep up crawler ranges. Avoid assumptions about where Google's crawlers originate; the Googlebot page only notes that its time zone is Pacific Time when it crawls from US addresses. A rule that admits traffic only from a few chosen countries should be tested against verified crawler ranges before it goes live.

Challenges that return 200

The hardest case to spot is a challenge page. When a bot-management layer serves a JavaScript test or CAPTCHA, the response code may be 200 while the body contains nothing from your site. Server logs report a success. Only fetching the page the way Google does, for instance with the URL Inspection tool in Search Console, and comparing the rendered result against the real page, reveals it.

What a CDN means by "verified bot"

Some CDNs keep a curated list of well-behaved bots. Cloudflare's verified bots documentation (Opens in a new tab) defines a verified bot as one Cloudflare has confirmed is transparent about who it is and what it does. To qualify, a bot has to identify itself deterministically, through a Web Bot Auth signature or a published IP list with a stable user agent, and must not behave abusively. The page also states that verified bots are excluded from default bot configurations on all plans. That says nothing about custom rules you wrote yourself, which may still match them, so audit your own rules separately. This section reflects Cloudflare's documentation only; other providers name and default these categories differently, so read your own provider's pages.

Fictional example: the Tuesday night spike

Fictional example, with no real site or figures behind it. A news publisher moves its article templates to a new layout on a Monday. Overnight, Googlebot revisits thousands of URLs whose internal links changed. A rate-limit rule added months earlier for a scraper incident starts answering with 429 for part of that burst. Search Console later reports a rise in server errors on the article directory, while category pages look normal. The server log shows the 429 responses clustered in a few hours and tied to addresses that verify as Googlebot. The fix is not to remove rate limiting, but to exempt verified crawler ranges from that one rule and let Google's own slowdown behavior regulate the pace. The example exists to show that errors can be confined to a time window and a URL pattern, which is why grouping log data by both is essential.

robots.txt, 403, 429 and 503 Responses: Where Access Rules Collide

Two different mechanisms can stop a fetch: robots.txt tells a crawler not to request a URL, while the HTTP status code tells it what the server answered when it did. When both are involved, diagnosis gets muddy. To test robots.txt rules themselves, see the robots.txt audit guide; for how each status code affects indexing, see the HTTP status code guide. This section covers only the collisions that matter for access.

What Google does with each class of error

The documentation on HTTP status codes and network errors (Opens in a new tab) is explicit. Any content received from a URL with a 4xx status, 429 excepted, is ignored. Crawlers signal to the next processing stage that the content does not exist; indexed URLs are removed, newly found 404 pages are not processed, and crawl frequency gradually decreases. The page adds a direct instruction: do not use 401 and 403 to limit crawl rate, because 4xx codes other than 429 do not affect it.

Google treats 429 as a sign the server is overloaded, so it is handled like a server error. Both 5xx and 429 make Google temporarily slow its crawling. URLs already indexed stay at first but are eventually dropped. For 500 the slowdown is proportional to the number of URLs returning errors, while persistently failing 502 and 503 URLs are removed from the index. Once the server returns 2xx again, Google gradually raises the crawl rate. The page leaves network and DNS errors, and the robots.txt fallback, to other documents.

Using 503 and 429 on purpose

If the origin truly cannot cope, Google's guide to reducing crawl rate (Opens in a new tab) offers an emergency measure: return 500, 503 or 429 instead of 200 to crawl requests for a short time, such as a couple of hours or one to two days. The reduction applies to the whole hostname, including URLs that would have returned content, and recovers automatically when errors drop. With 503 or 429 you may add a Retry-After header holding either a number of seconds or an absolute UTC date and time.

The same guide states the cost of overdoing it. If Googlebot sees these codes on the same URL for multiple days, the URL may be dropped from the index. Googlebot also discovers fewer new pages, refreshes existing ones less often, and removed pages may linger longer in the index; Google Ads campaigns may be paused or canceled. And the page is blunt in the other direction: you cannot request an increase in crawl rate, though a Search Console form exists for reporting an unusually high rate and stating your optimal one.

A 5xx on robots.txt itself

The least visible collision involves the robots.txt file. Google's robots.txt documentation (Opens in a new tab) says 4xx responses other than 429 are treated as if no valid robots.txt existed, so no restrictions apply. For 5xx the behavior is staged: Google stops crawling the site for 12 hours while it retries; if it still cannot obtain a new copy it uses the last good cached copy for up to 30 days; with no cached copy it assumes no restrictions. After 30 days of unresolved errors, a generally reachable site is treated as having no robots.txt while Google keeps checking, whereas a site with general availability problems stops being crawled and gets periodic robots.txt requests. Google generally caches the file for up to 24 hours and processes up to 500 KiB of it.

The security implication is direct. If a WAF rate-limits or challenges only the /robots.txt path, the file may become unreachable at the very moment you rely on it to fence off sections. Monitor that the file always answers 200 as plain text, from the same layers that serve the rest of the site.

Quick reference by symptom

A Diagnostic Flow and Evidence Log for Access Problems

Fixing access by trial and error usually means toggling rules on and off until something changes, with no record of why. The sequence below is designed so that each step produces evidence that someone else can review. For how to read the raw files efficiently, the server log analysis guide covers technique; here it is applied to a specific incident.

Step 1: Write down the symptom

Record which URL group is affected (whole site, one directory, one parameter pattern), when it began, and what Search Console reports and for which dates. Keep screenshots and dates. A vague "Google cannot crawl us" cannot be debugged; "the /articles/ directory returns server errors since the 14th" can.

Step 2: Pull the Google-labelled requests

From the origin, CDN or WAF logs, extract requests claiming to be Google for the affected URLs. Keep timestamp, IP, user agent, URL, status code and, if available, which layer produced the response. Three outcomes are common and point in different directions: no requests at all, requests answered with 4xx or 5xx, and requests answered 200 with the wrong body.

Step 3: Verify the addresses

Sample the IPs and verify them with reverse and forward DNS, or compare them in bulk with the JSON ranges. This separates real Googlebot from impostors. Blocking unverified clients is working as intended. Blocking verified ones is the defect to fix.

Step 4: Locate the responding layer

Compare what the same URL returns through each layer: via the CDN as an ordinary client, directly at the origin where you are authorized to test it, and in the WAF's own event log. If the error appears only through the CDN, the edge rule is the cause; if it also appears at the origin, look at the application or server configuration. Test only systems you are authorized to access, and be careful when bypassing a layer on production.

Step 5: Narrow the fix

The least invasive remedy is to exempt verified crawler ranges from the specific rate or bot rules that caught them, rather than whitelisting by user agent text. Write down which paths and which rules the exemption covers. Try it on a limited URL group first and watch the logs.

Step 6: Confirm recovery

After the change, inspect a few affected URLs with the URL Inspection tool, confirm in the logs that Google requests for the affected group now return 200 with real content, and watch for several days. Search Console reports lag, so logs are the faster proof. Re-check that robots.txt returns 200 and that the body is the real file.

The evidence log

Keep one row per incident with the following fields, so the next investigation starts from precedent instead of memory:

• Date detected and how (alert, log review, report from another team).
• Affected URL group plus three sample URLs.
• Sample log entries: time, IP, user agent, status code.
• Verification result and method (reverse and forward DNS, or JSON range match).
• Responding layer and the rule name or identifier.
• Change made, who made it, when, and how to roll it back.
• Post-fix check result and the date of the next review.

A log like this gives security and SEO teams one shared language. "Rule 17 returned 429 to verified crawler addresses on this directory between these hours" is something a security engineer can act on in minutes; "the bot is blocked" is not.

Bot Policy Beyond Search: AI Crawlers and Blanket Rules

When teams tighten bot controls, they often target AI training and answer-engine crawlers in the same pass. That is a legitimate policy decision, but it is a different decision from keeping search crawling healthy. Managing AI crawlers through robots.txt is covered in the guide to AI crawlers and robots.txt. What matters here is the edge: a single "block all bots" rule can easily include Googlebot unless it is built around verified identity.

Before deploying any broad bot rule, ask whether it works on user agent text or on IP ranges, whether verified Google ranges are excluded, whether it covers the robots.txt and sitemap URLs, and how the exemption list is maintained over time. Writing the policy down in plain language helps: which bots are allowed, for what purpose, and who reviews the list.

Assets, subdomains and rendering

Googlebot needs more than the HTML document. CSS, JavaScript and images are fetched separately according to the Googlebot documentation, so a static-asset subdomain or CDN hostname with stricter bot rules can leave pages looking broken to the crawler even when the main hostname is healthy. Check bot settings on every hostname that serves resources your pages need.

Common Mistakes

• Trusting the user agent: allowing anything that says Googlebot invites impostors, while mishandling the mobile user agent can lock out the real crawler.
• Stopping at the reverse lookup: without the forward lookup, the check is incomplete.
• Using 403 as throttling: Google's documentation says not to; 403 closes content, it does not set speed.
• Leaving 503 in place too long: beyond a day or two the risk of URLs dropping from the index grows.
• Covering the robots.txt path with a rate or challenge rule: an unreachable file changes how restrictions are applied.
• Freezing an IP list: the documentation does not state how often ranges change, so read them from the official files on a schedule.
• Relying on Search Console alone: reports lag; logs and verified IPs are quicker evidence.
• Treating every bot the same: search crawlers, AI crawlers and monitoring tools arrive for different reasons.

Pre-launch Checklist for Edge Rules

Before turning on a new WAF, CDN or bot-protection rule, working through a short list costs less than discovering the damage in a Search Console report:

1. Is the rule's scope written down: which paths, which IPs, which request traits?
2. Is there a separate exemption for verified Google crawler ranges?
3. Are robots.txt and sitemap URLs handled separately or exempt?
4. Did the rule run first in monitor-only mode, and were the logs reviewed?
5. Were rate thresholds compared against Googlebot's historical request density?
6. If a challenge page can be shown, was its status code and body tested?
7. After release, were a few critical URLs inspected in Search Console?
8. Does someone own the rollback?

The list does not replace a security review. Its only job is to protect crawl access; security decisions remain with whoever owns the site's risk.

Frequently Asked Questions

How can I tell whether a request is real Googlebot?

Do not trust the user agent string alone, because other crawlers often spoof it. Run a reverse DNS lookup on the IP from your logs, confirm the returned domain is googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that domain and confirm it returns the same IP. For bulk checks, match the IP against the JSON range files Google publishes.

Why would a WAF or CDN block Googlebot?

Bot protection usually scores requests by rate, IP reputation, headers and behavior. Googlebot is an automated client that can request many URLs in a short time, so it can trip the same rules aimed at scrapers. Rules written in a hurry against fake Googlebot traffic can also catch the real one. The fix is a rule that tells verified traffic apart, not a blanket allow for anything that claims to be Googlebot.

Does returning 403 slow Googlebot down?

No. Google's documentation says 401 and 403 should not be used to limit crawl rate, and that 4xx codes other than 429 have no effect on it. Content from 4xx URLs is ignored, and URLs already indexed are removed over time. A 403 therefore closes access to the content instead of adjusting speed.

Which response should I send when my server is overloaded?

For an emergency, Google suggests temporarily returning 500, 503 or 429 for a short period, from a couple of hours up to one or two days. With 503 and 429 you can add a Retry-After header. If the codes persist on the same URL for several days, the URL may be dropped from the index, so the lasting fix has to be capacity or caching.

What happens when robots.txt returns a 5xx error?

According to Google's robots.txt documentation, Google first stops crawling the site for 12 hours while it retries the file. If it cannot get a fresh copy, it uses the last good cached copy for up to 30 days, and with no cached copy it assumes there are no crawl restrictions. After 30 days of errors the behavior depends on the general availability of the site.

Is a user-agent allowlist enough to let Googlebot through?

No. A user agent is trivial to fake, so an allowlist built only on it also admits impostors. Verify with reverse and forward DNS or the official IP ranges, and if your CDN offers a verified bot category, factor that in as well.

Conclusion and Next Step

Three habits solve most Googlebot access problems. Verify identity with reverse and forward DNS or the official ranges rather than the user agent. Remember that robots.txt is not the only gatekeeper, because WAFs, CDNs and rate limits can close the door earlier. And use 403, 429 and 503 deliberately, knowing that Google reads each differently and that error codes held for days put indexed URLs at risk. When every incident leaves behind log extracts, a verification result and a rule record, fixes rest on evidence instead of guesses.

Once crawlers can reach your pages, what they find there decides the outcome. For pages worth crawling, Akua Adforce offers SEO content writing; scope and terms are described on the product page. The advice in this guide is general technical guidance and does not guarantee any ranking, crawl frequency or indexing result.