The bot management module of Web Application Firewall (WAF) provides the scenario-specific configuration feature. This feature allows you to configure anti-crawler rules based on your business requirements and helps protect your business from malicious crawlers. This topic describes how to configure anti-crawler rules for websites.
Background information
The scenario-specific configuration feature allows you to configure anti-crawler rules based on your business requirements. You can use this feature together with intelligent algorithms to identify crawler traffic in a more precise manner. The feature can also automatically handle the crawler traffic that matches the configured anti-crawler rules. After you configure anti-crawler rules, you can verify the rules in a test environment. This prevents adverse effects on your websites or apps caused by inappropriate rule configurations or compatibility issues. The adverse effects include false positives and undesired protection results.
Prerequisites
- Subscription WAF instance: If your WAF instance runs the Pro, Business, or Enterprise edition, the Bot Management module is enabled. For more information, see Purchase a WAF instance.
- Your website is added to WAF.
For more information, see Tutorial.
Procedure
FAQ
Error | Cause | Solution |
---|---|---|
No valid test requests are detected. See WAF documentation or contact us to analyze the possible causes. | The test request fails to be sent or is not sent to WAF. | Make sure that the test request is sent to the IP address that maps the CNAME provided by WAF. |
The header fields in the test request do not match the header fields that you specify for Traffic Characteristics in the anti-crawler rule. | Modify the settings of Traffic Characteristics in the anti-crawler rule. | |
The source IP address of the test request is different from the public IP address that you specify in the anti-crawler rule. | Use the correct public IP address. We recommend that you click Alibaba Network Diagnose Tool to obtain your public IP address. | |
The test requests failed the verification. See WAF documentation or contact us to analyze the possible causes. | No real user access is simulated. For example, the debugging mode or automation tools are used. | Simulate real user access during the test. |
An incorrect service type is selected. For example, Websites is selected when you configure an anti-crawler rule for apps. | Change the value of the Service Type parameter. | |
An intermediate domain name is used, but an incorrect intermediate domain name is selected in the anti-crawler rule. | Select Use Intermediate Domain Name. Then, select the correct intermediate domain name from the drop-down list. | |
Compatibility issues occur in the frontend. | Contact customer service in the DingTalk group or submit a ticket. | |
No verification is triggered. See WAF documentation or contact us to analyze the possible causes. | No test rules are generated. | Perform the test several times until the test rule is generated. |
No valid test requests are detected or blocked. See WAF documentation or contact us to analyze the possible causes. | The test request fails to be sent or is not sent to WAF. | Make sure that the test request is sent to the IP address that maps the CNAME provided by WAF. |
The header fields in the test request do not match the header fields that you configure for Traffic Characteristics in the anti-crawler rule. | Modify the settings of Traffic Characteristics in the anti-crawler rule. | |
The source IP address of the test request is different from the public IP address that you specify in the anti-crawler rule. | Use the correct public IP address. We recommend that you click Alibaba Network Diagnose Tool to obtain your public IP address. |