Cloudflare, a leading web performance and security company, protects our blogs from all bot-traffic and DDoS attacks. Its Super Bot Fight Mode allows website owners to customize their bot management strategy and create whitelists for specific bots or user agents.
Bots are automated programs that perform various tasks on the web, such as crawling, scraping, spamming, hacking, etc. Some bots are good, such as search engine crawlers or site monitoring bots, which help website’s visibility, ranking, and functionality. However, some bots are bad, such as scrapers, spammers, credential stuffers, etc., which can harm our website performance, security, and reputation.
According to Cloudflare, bots account for nearly 40% of all Internet traffic, and more than half of them are malicious. To protect websites from malicious bots, Cloudflare offers a feature called Bot Fight Mode, which identifies traffic matching patterns of known bots and issues Manage challenges in response to these bots. However, this feature may also affect some good bots, such as Google and Bing.
To avoid blocking good bots, Cloudflare recommends using their Super Bot Fight Mode, which is included in their Pro and Business plans. This feature allows website owners to create custom rules for different types of bots, such as verified bots, likely bots, and definitely automated. Website owners can also create allowlists for specific bots or user agents, such as Googlebot, Bingbot, or Claude bot, an AI assistant created by Anthropic, a company that focuses on AI safety research.
Cloudflare claims that Super Bot Fight Mode can help website owners improve their bot management strategy and reduce their bandwidth costs, server load, and security risks. Cloudflare also provides a dashboard that shows a breakdown of bot traffic and the impact of bot rules.
Some of Cloudflare’s partners, such as Quora, Notion, and DuckDuckGo, have praised Super Bot Fight Mode for its effectiveness and reliability. Users also report that Super Bot Fight Mode is easy to use and configure, and that it helps them block bad bots while allowing good bots.
Website owners who are interested in using Super Bot Fight Mode can upgrade to the Pro or Business plan and enable the feature in their Cloudflare dashboard. They can also learn more about how to manage good bots and bad bots in Cloudflare’s blog.
For website owners who are using the free plan and cannot use the Super Bot Fight Mode, there are some other methods to manage bots, such as:
- WAF rules:
Using Firewall Rules, create custom expressions that match the characteristics of the bots you want to allow or block. For example, you can use the `cf.client.bot` attribute to identify requests from known bots, or the `http.user_agent` attribute to match requests from specific user agents.
- Page rules:
Using Page Rules, apply different settings for different URLs on your website. For example, you can use the Security Level setting to adjust the sensitivity of the browser integrity check, which can help block bots that try to spoof legitimate browsers.
Using WAF Firewall Rules to create custom, user-defined logic by blocking or allowing traffic that leverages all the components of the HTTP requests and dynamic fields computed by Cloudflare, such as Bot score. This is an alternative method for the free plan users who cannot use the Super Bot Fight Mode.
These methods may not be as effective or convenient as the Super Bot Fight Mode, but they can still help you improve your bot management strategy with the free plan.
Cloudflare WAF rule to allow Googlebot and Bingbot
To create a Cloudflare WAF rule to allow all major search engine bots like Googlebot and Bingbot, you can use a combination of user agent strings commonly associated with these bots. The sample expression that you can use:
(http.user_agent contains “Googlebot” or
http.user_agent contains “Google Search Console” or
http.user_agent contains “Googlebot-Image” or
http.user_agent contains “Googlebot-Video” or
http.user_agent contains “Mediapartners-Google” or
http.user_agent contains “bingbot” or
http.user_agent contains “BingPreview” or
http.user_agent contains “msnbot”)
This expression allows requests where the user agent string contains any of the specified substrings associated with Googlebot, Bingbot, and related crawlers.
