Managing Bad Bot Traffic: WAF, Rate Limits, Googlebot

Managing bad bot traffic means telling requests apart and putting limits on the ones that cause harm, not blocking everything automated. Three tools work together: a web application firewall (WAF) that judges requests against rules, rate limits that cap how many requests one source can send, and a check that confirms search bots are genuine before you let them through. A robots.txt file is not one of these tools. If scrapers copy your pages, a form is flooded with junk or automated requests are straining your server, work through the steps in this order and confirm at every stage that Googlebot still gets in.

Why robots.txt is not a lock

RFC 9309, the specification of the Robots Exclusion Protocol, is explicit. It states that the rules in a robots.txt file are not a form of access authorization, and that the protocol is not a substitute for valid content security measures. Because the listed paths are public, sensitive areas need real protection such as authentication. The same document says crawlers must obey the rules of the group that matches their product token, which describes what a well-behaved crawler does and not what a hostile one is forced to do. Read the full text in RFC 9309 (Opens in a new tab).

The consequences are practical. Writing a private path into robots.txt does not protect it and may advertise it. For friendly crawlers, though, robots.txt remains the right instrument. The RFC adds that crawlers should not use a cached copy for more than 24 hours unless the file is unreachable, that a crawler must assume complete disallow when the file cannot be reached because of server or network errors, and that parsing limits must be at least 500 kibibytes. The middle point matters for security work: if your server answers 5xx, robots.txt can become unreachable and a crawler may treat the whole site as off limits. To confirm your file behaves as intended, see the robots.txt audit guide.

Three kinds of automated visitor

Not every bot is bad. Search engines, uptime monitors and link preview tools all send automated requests, and some serve your interests. The problem is automation with unclear identity or harmful intent.

The Cloudflare bots documentation (Opens in a new tab) describes three solutions that detect automated traffic and let you choose the response: Bot Fight Mode, Super Bot Fight Mode and Bot Management for Enterprise. The principles carry over to other providers, but product names and features differ.

AI crawlers are a separate question

Crawlers operated by AI companies raise their own questions about robots.txt directives and opt-outs; see AI crawlers and robots.txt. This topic is about protecting your own site.

Measure first: symptoms and where to find them

Do not assume a bot problem; show it from the logs. Every intervention can hit real users, so the basis should be data.

Cleaning bot traffic out of reports is a measurement task, handled in filtering internal and bot traffic in GA4. A report filter does not reduce load on the server, so protection and measurement need separate work.

Fields worth logging

• Timestamp, path and query string of each request.
• Response status code and response time.
• Source IP address and, where available, its network owner.
• User agent string, treated as a claim and not as proof.
• The rule or setting that handled the request, if a firewall was involved.

WAF rules and rate limits

A rule has three parts: an expression that selects requests, an action, and an exception list. Cloudflare's bots page adds detail on the available products. Bot Fight Mode can challenge detected bot traffic across a whole domain with one toggle. Super Bot Fight Mode, available on Pro, Business and Enterprise without an add-on, adds configurable actions per bot category and supports WAF custom rule exceptions, but lacks per-request bot scoring. Bot Management offers per-request bot scores, custom rules, per-endpoint handling and detailed analytics; the page recommends it for ecommerce, banking and security use cases and says to contact your account team to enable it. The page does not define what a bot score is.

For rate limiting, Cloudflare's WAF documentation describes defining limits for incoming requests that match an expression, together with the action to take when a limit is reached. The action list sits on the linked page, so check your own provider's current documentation.

Setting a rate limit from data

Practical suggestion: choose limits from the log, not from instinct. Work out how frequently a genuine user might request the same path, the pace of a search crawler, and the pace of the unwanted source. Set the limit higher than a genuine user reaches and lower than the harmful pace. For requests over the limit, consider a challenge first and a temporary block second, and avoid one blanket block.

Order of work

1. Measure which path, which source and how fast.
2. Start with cheap fixes: server-side form validation, login attempt limits.
3. Add rate limits on the harmed paths only.
4. Use a WAF rule against clearly harmful patterns.
5. Add a bot management product if the need persists.
6. After each step, check the effect on Googlebot and on real users.

Form spam and scrapers need different answers

Cloudflare describes Turnstile as a smart CAPTCHA alternative that runs less intrusive challenges. Other vendors sell comparable products. Check accessibility and privacy terms before choosing one.

Verifying Googlebot so you never block it

Do not accept a request as Googlebot because its user agent says so. Google's verifying Googlebot page (Opens in a new tab) describes two methods. In the manual method you run a reverse DNS lookup on the IP address in your logs, check that the domain name ends in googlebot.com, google.com or googleusercontent.com, run a forward DNS lookup on that name and confirm that it resolves to the original IP. In the automatic method you match the requesting IP against Google's published IP range files in CIDR notation, which are split by crawler type: crawlers, special-case crawlers, user-triggered fetchers and user-triggered agents. A general goog.json list covers other Google IP addresses.

The page notes that spammers may claim to be from Google, which is exactly why verification matters. If your firewall admits anything labelled Googlebot, a hostile bot only has to take that name. If it ignores the label entirely, the real Googlebot may be shut out. The remedy is to verify per rule. For blocks caused by a WAF or CDN, see Googlebot access verification and WAF or CDN blocks.

Not every Google request is Googlebot

A request from a Google address is therefore not necessarily Googlebot, and a robots.txt rule does not bind all of them. Decide which client to admit rule by rule.

Monitoring and undoing false positives

After a rule goes live, measure two things: did the unwanted traffic fall, and did genuine users or search bots suffer? Search Console's crawl statistics help with the second question: a rising share of 403 or 429 responses, or a host status problem, is a warning. Crawl budget and server load are discussed in the crawl budget optimization guide.

Undoing a false positive

1. Find the blocked request in the log: IP, path, user agent, rule.
2. Switch the rule to monitor mode and measure again.
3. Verify the legitimate client and define a rule exception.
4. Write the change, date and reason in a change log.
5. If 5xx errors also rose, review the server using hosting outage and 5xx crawl impact.

Example scenario: a pet shop

Example scenario: a pet shop runs a grooming appointment form and a product price list on its site. After the shop announces a new-season food offer, the form starts receiving gibberish submissions and the price list pages are requested over and over in quick succession. The freelance developer who built the site opens the access log. The submissions come from a handful of address ranges. The price list requests present a browser-style user agent, yet the gaps between them are far too even for a person.

The developer adds server-side validation and a low-friction challenge to the form, then caps submissions. Next comes a rate limit on the price list path only. Then the developer confirms that Googlebot is untouched: the response code split in crawl stats is unchanged, and the log still shows verified search bot requests answered with success codes. A week later the form is clean, but the shop's pet-food supplier reports that its stock-checking tool is being refused. Its address range is verified and a rule exception is recorded. The developer's closing note says that listing the price list path in robots.txt would not have stopped any of the unwanted requests.

Server-side readiness: caching, limits and a request budget

The damage a bot can do is measured by how well the server absorbs it. The same request volume is invisible to a site with solid caching and fatal to one that sends every request to the database. Before writing rules, review three things on the server side.

Google's page on HTTP errors says 5xx and 429 responses make its crawlers slow down temporarily, so on any path where you apply rate limits, remember that a search bot may receive a 429 as well. Keep limits to the paths that really suffer, and leave out the content pages that search bots visit.

A request budget per path

Practical suggestion: write down the request rate you accept for each path. The home page and article pages get a wide budget; search and login endpoints get a narrow one. When a client exceeds its budget, slow it first and challenge it second. A written budget also makes the rule easier to defend later, because you can explain the number by pointing at a table.

Rule design: expression, action and exception

Whatever vendor you use, a rule has three parts: an expression that selects requests, an action applied to them, and an exception list of clients exempt from the action. Sketch the three before you write the rule.

Action names differ between providers. Practical suggestion: order actions from mild to severe. A logging action stops nothing but shows whom the rule would catch. A challenge gives a genuine visitor a chance. A temporary block expires by itself and repairs its own mistakes. A permanent block should be the last resort. The narrower the expression, the lower the chance of false positives. Combining path, method, network and request rate catches the target and little else, while a rule that rests on the user agent alone is simple to evade and can sweep in legitimate tools.

The cost to real users

Bot defences sometimes cost real people. A challenge screen that cannot be operated with a keyboard or screen reader shuts out disabled visitors. A per-address rate limit can throttle many genuine customers who share an office network. Country blocks cut off travellers and customers abroad.

An incident log template

Questions for a provider

• Which plan includes bot protection and which features need an extra license?
• What actions can a rule take, and is there a monitor mode?
• How long are logs of blocked requests kept?
• Is there a separate setting for verified search bots?
• If a false positive happens, which channel does support use and how quickly does it answer?

These questions are not a brand comparison. Their purpose is to put the provider's answers in writing.

Decision table

Decision rule: when the harm from a pattern is proven, apply a targeted limit to that path; when it is not, run the rule in monitor mode first. For the wider security context, read this alongside the website security and hosting checklist.

Twelve-point checklist

• The problem was shown from logs: path, source, speed.
• robots.txt is not treated as a security measure; sensitive areas use authentication.
• Form spam is handled by server-side validation and a challenge.
• Rate limits were chosen from log data and applied to the harmed path only.
• Each WAF rule was tried in monitor mode first.
• Googlebot is verified by DNS or IP range, never by user agent alone.
• Crawl statistics and response codes were checked after each rule change.
• Legitimate tools are on an exception list and their source was verified.
• Every rule change is logged with date and reason.
• Support tickets about being unable to open the site are watched.
• Current rule names and actions were checked in the provider's documentation.
• A review date for the rules is on the calendar.

Reviewing rules on a schedule

Rules age. A path that was under attack last spring may be harmless now, and a new form may need a limit it does not have. Put the review on the calendar, name an owner, and keep the change log current so the next person can see why each rule exists.

Once the checklist is complete, summarize the result in one sentence: which path is protected, which rule runs until which date, and who watches it. Keep that sentence next to the change log so both are read together. It restores context when the team changes or when you write to the provider's support, and it stops rules from outliving their reasons.

Frequently Asked Questions

Does robots.txt block bad bots?

No. RFC 9309 says robots.txt rules are not a form of access authorization and not a substitute for valid content security measures. Protect sensitive areas with authentication or similar controls.

How do I verify Googlebot?

Per Google's documentation, run a reverse DNS lookup on the IP in your logs, confirm the name ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup and confirm it returns the same IP. Alternatively, match the IP against Google's published IP range files.

Can a rate limit affect Googlebot?

It can if set badly. Choose the value from log data, try it in monitor mode, and check crawl response codes after the rule goes live.

Do these principles apply if I do not use Cloudflare?

Yes. The principles apply with any provider, although feature and menu names differ.

What separates Bot Management from Super Bot Fight Mode?

According to Cloudflare's documentation, Super Bot Fight Mode offers configurable actions per bot category without per-request bot scoring, while Bot Management provides per-request bot scores and custom rules.

Is a challenge enough against form spam?

It may not be enough alone. Combine server-side validation, a submission rate limit and, where useful, a smart challenge, and make sure accessibility is not harmed.

Conclusion

Managing bad bots comes down to showing the problem in your logs, never treating robots.txt as a security measure, and using targeted WAF rules and rate limits. Try each rule in monitor mode, verify Googlebot through DNS or published IP ranges and not through the user agent, and check crawl responses after every change. When you see a false positive, switch the rule to monitor mode and add a verified exception.

To build the authority of a protected, fast site with quality links, take a look at our backlink packages.