General availability of Azure WAF Bot Manager1.1 Ruleset

Today, we are launching the general availability of Bot Manager1.1 ruleset in Azure WAF integrated with Azure Front Door.

Bot Manager1.1 extends all the rules in the existing Bot Manager1.0 ruleset and adds multiple new rules to provide comprehensive bot management capabilities to web applications. The new capabilities introduced in this ruleset include new Goodbots rules and a new Badbots rule.

 

The main value prop of the new ruleset is to reduce false positives in good bot detections and increase true positives in malicious bot detections.

 

Benefits of the new rules in the Goodbots rule group:

 

1. Improving SEO rankings due to good bots crawling websites and reducing FP (false positive) seen by customers.

Customer websites are crawled by good bots which results in increased SEO (search engine optimization) rankings. With Bot Manager 1.1 ruleset, a comprehensive set of rules are added to the Goodbots rule group which allows a larger set of legitimate published bots. Examples of such Goodbots include Googlebot, Bingbot etc.

As a real-life scenario, we encountered an issue with the Bot Manager1.0 ruleset where certain Goodbots were absent, leading to blocked requests to web applications. For example, a valid Google crawler bot was getting blocked by the Bot Manager1.0 100200 rule, which resulted in lower SEO rankings for the customer and eventually disappearing from the SEO rankings. As a workaround, the customer disabled rule 100200 which brought their SEO rankings up but resulted in lowered protection from true malicious bots that have falsified their identities. Prior to implementing the Bot Manager1.1 ruleset, the only other alternative to allow legitimate crawlers was to add custom rules to allowlist their IP addresses. However, this approach posed challenges due to the dynamic nature of crawler IPs, which change frequently.

With the new updates to Bot Manager1.1, a comprehensive list of good bot IPs is added to the existing rule 200100 which results in lower false positive detections by the Bot Manager ruleset. The 200100 rule from Bot Manager1.0 ruleset is now revamped to only include good bots in the search engine crawler category.

2. Bringing clarity to the Goodbots rule group

With Bot Manager 1.1 ruleset, many new verified good bot rules have been added that target different categories of good bots. These new rules include the link checker, social media, content fetchers, feed fetcher and advertising bots. Additional bots that don’t fit into any particular category are added to 200200 as verified miscellaneous bots.  This empowers customers to have granular control over their WAF policy. For example, if a customer does not wish to have social media bots crawling their sites, they can achieve this by changing the action associated with the social media rule.

 

 

Benefits of the new rule in the Badbots rule group:

Today customers see malicious bots perpetuating many malicious attacks. Examples includes:

• Scraping websites and spreading dis-information, executing targeted phishing attacks and social engineering attacks.
• Spamming customer websites with form submission pages.
• Manipulating rankings of content tooling websites’ analytics pages.
• Launching denial-of-inventory attacks.
• and many others.

 

The new Bot Manager1.1 ruleset incorporates a novel rule, Bot100300, complemented by the existing rules in the Badbots rule group rules, effectively mitigates malicious bot attacks.

 

 

Let’s take a closer look at the Bot Manager1.1 ruleset:

Goodbots rule group

The following screenshot describes the new good bot rules added to the new ruleset

 

The details of all the good bot rules are given below:

Good bot rule ID	Status	Description	Explanation
200100	Updated	Verified search engine crawlers	Search engine bots- Google, Yahoo, Bing etc.
200200	Updated	Verified misc bots	All verified good bots that do not fit into any specific good bot category
200300	New	Verified link checkers	Link checker bots give information about a link or a domain name. It returns the screenshot or metadata about the link that the user trying to get to.
200400	New	Verified social media bots	Social media bots – Facebookbot, LinkedInbot etc.
200500	New	Verified content fetchers	Content fetcher bots retrieve content for websites on desktop, in-app browsers, mobile apps etc.
200600	New	Verified feed fetchers	Feed fetcher bots periodically refresh feeds like the RSS feeds requested by users.
200700	New	Verified Advertising bots	Advertising bots – GoogleAds, BingAds etc.

 

The default action for all the new good bot rules is ‘allow’ by default but it is possible to change them to any of the supported actions.

 

 

Badbots rule group

The following screenshot describes the new bad bot rule, rule Bot100300.

 

The bots detected by the Bot100300 rule includes risky IPs that are based on their high-risk score detected by threat intelligence. These IPs differ from the Bot100100 rule, which identifies verified malicious IPs detected by Microsoft Threat Intelligence and are subject to a different set of criteria, including their Tactics, Techniques, and Procedures (TTPs), any related lateral threat activity seen by the IP, and other indicators of compromise.

 

The default action for Bot100300 is ‘block’ by default but it is possible to change it to any of the supported actions.

 

 

JavaScript(JS) challenge mitigation in Bot Manager1.1 ruleset

 

The ruleset the newly released JS challenge on AFD WAF as an action to any of the Bot Manager rules. The JS challenge is an addition to existing actions and provides all the feature benefits of the invisible web challenge to protect web applications.

 

JS challenge action is available in Bot Manager 1.0 as well for backward compatibility.

 

 

How to enable Bot Manager1.1 ruleset 

The Bot Manager1.1 ruleset can be assigned through the drop-down “Assign” option as part of the managed rulesets.

 

The new Bot Manager1.1 ruleset expands the bot management capabilities to provide comprehensive protection against malicious bots while allowing verified good bots to go through Azure WAF.

 

You can obtain more details about this feature on MS Learn at What is Azure Web Application Firewall on Azure Front Door? | Microsoft Learn and Azure Web Application Firewall DRS rule groups and rules | Microsoft Learn

 

Sowmya Mahadevaiah

Principal Product Manager, Azure Networking