The bot management module of WAF is upgraded to provide the scenario-specific configuration feature. You can configure anti-crawler rules based on your business requirements. This feature protects your business from malicious crawlers. This topic describes how to configure anti-crawler rules for apps.

Prerequisites

  • If you use a subscription WAF instance that runs the Pro, Business, or Enterprise edition, the bot protection module and the apps protection feature are enabled.
  • The Anti-Bot SDK is integrated into the apps that need protection. For more information, see Incorporate Anti-Bot SDK into apps.

Background information

The scenario-specific configuration feature allows you to configure anti-crawler rules based on your business requirements. This feature can be used in combination with intelligent algorithms to identify crawler traffic. In addition, this feature can automatically handle the crawler traffic that matches the configured anti-crawler rules. After you configure anti-crawler rules, you can verify the anti-crawler rules in the test environment. This prevents adverse effects, such as false positives and undesired protection results, on your websites or apps due to inappropriate rule configurations or incompatibility issues.

Configure anti-crawler rules for apps

  1. Log on to the Web Application Firewall console.
  2. In the top navigation bar, select the resource group to which the instance belongs and the region, Mainland China or International, in which the instance is deployed.
  3. In the left-side navigation pane, choose Protection Settings > Website Protection.
  4. In the upper part of the Website Protection page, select the domain name for which you want to configure the whitelist. Switch Domain Name
  5. Click the Bot Management tab. In the Scenario-specific Configuration section, click Start to create your first anti-crawler rule. Enable the scenario-specific configuration feature for the first time
    If you have created an anti-crawler rule, you can skip this step and click Add in the upper-right corner to create a rule.
  6. In the Configure Scenarios step, configure the basic information about the scenario in which you want to protect apps and click Next.
    Parameter description:
    • Scenario: Enter the type of scenario in which you want to protect apps. Examples: logon, registration, and order placement.
    • Service Type: Select App to protect native iOS and Android apps.
      Note HTML5 apps are not native iOS or Android apps. If you want to protect HTML5 apps, set the Service Type parameter to Websites.
    • Traffic Characteristics: Add match conditions for requests to the apps that you want to protect by using the anti-crawler rule. For more information about header fields in the match conditions, see Fields in match conditions. You can add up to five match conditions.
      Notice After you enter an IP address, you must press Enter.
  7. In the Configure Protection Rules step, configure the details about the anti-crawler rule and click Next.
    Parameter description:
    • Check Invalid App Signature: After you enable this feature, the anti-crawler rule detects and controls the requests to apps, which have invalid signatures or do not have signatures. You cannot disable this feature. You can configure Action to handle these requests to apps. If you set the Action parameter to Monitor, the anti-crawler rule allows the traffic that matches the rule and records the traffic in security reports and logs. If you set the Action parameter to Block, the anti-crawler rule blocks the traffic that matches the rule.
    • Check Abnormal Device Behaviors: After you enable this feature, the anti-crawler rule detects and controls the requests from the devices that have abnormal characteristics.

      Abnormal characteristics include Use Simulators, Use Proxies, and Use Root Tools.

      You can set the Action parameter to Monitor or Block based on your business requirements. If you set the Action parameter to Monitor, the anti-crawler rule allows the traffic that matches the rule and records the traffic in security reports and logs. If you set the Action parameter to Block, the anti-crawler rule blocks the traffic that matches the rule..

    • Action: You can set this parameter to Monitor or Block. This setting takes effect for Check Invalid App Signature and Check Abnormal Device Behaviors.
    • IP Address Throttling, Device Throttling, and Custom Session-based Throttling: After you enable these features, you can configure throttling conditions to filter abnormal requests. This way, HTTP flood attacks are mitigated.
      • IP Address Throttling: You can configure throttling conditions for IP addresses. If the number of requests from the same IP address within the specified time period exceeds the threshold, WAF applies a specified action to subsequent requests. You can also configure the period during which the specified action is performed. The action can be monitor, block, or slider CAPTCHA. You can add a maximum of three conditions. For more information, see Create a custom protection policy.
      • Device Throttling: You can configure throttling conditions for devices. If the number of requests from the same device within the specified time period exceeds the threshold, WAF applies a specified action to subsequent requests. You can also configure the period during which the specified action is performed. The action can be monitor or block. You can add a maximum of three conditions.
      • Custom Session-based Throttling: You can configure throttling conditions for sessions. If the number of requests from the same session within the specified time period exceeds the threshold, WAF applies a specified action to subsequent requests. You can also configure the period during which the specified action is performed. The action can be monitor or block. You can add a maximum of three conditions. For more information, see Create a custom protection policy.
  8. Optional:In the Verify Actions step, test the effectiveness of the anti-crawler rule.
    This step is optional. To skip this step, you can click Skip in the lower-left corner. Before you publish the rule, we recommend that you complete this step.
    Parameter description:
    • Public IP Address Test: Enter the public IP address of your test device, for example, a mobile phone. The test of the anti-crawler rule takes effect only for the public IP address. The test does not affect your business.
      Notice If you want to obtain the public IP address of your test device, you can click Alibaba Network Diagnose Tool. Then, on the displayed page, search for local IP. You can also search for the IP address by using your browser.
    • SDK Signature Verification: Click Start Test to verify that the SDK signature of apps is normal.
      Note Make sure that the Anti-Bot SDK for apps is integrated into the test device. If the Anti-Bot SDK is not integrated into the device, the signature verification fails, normal requests are blocked, and the test cannot be completed.
    • Action Test: Test the effectiveness of the Block action. After you click Start Test for an action, WAF immediately sends the anti-crawler rule to the test device. WAF also provides the demonstration diagram and description of the test results. We recommend that you read the demonstration diagram and description of the test results.

      After the test is complete, you can click I Have Completed Test to go to the next step. If the test result is abnormal, you can click Go Back to optimize the anti-crawler rule. Then, implement the test again.

      For more information about the exceptions that may occur during a test and about the solutions to these exceptions, see FAQ.

  9. In the Preview and Publish Protection Rules step, confirm the content of the anti-crawler rule and click Publish.
    After the anti-crawler rule is published, the rule immediately takes effect.
    Note If this is the first time to create an anti-crawler rule, you cannot view the rule ID until you publish the rule. The rule ID is displayed on the Bot Management tab of the Security report page. You can use the ID of an anti-crawler rule to check for requests that match the rule in Log Service for WAF.

FAQ

Error Cause Solution
No valid test requests are detected. See WAF documentation or contact us to analyze the possible causes. The test request fails to be sent or is not sent to WAF. Verify that the test request is sent to the IP address that maps the CNAME provided by WAF.
The header field in the test request does not match the header fields that you configure in Traffic Characteristics in the anti-crawler rule. Modify the settings of Traffic Characteristics in the anti-crawler rule.
The source IP address of the test request is inconsistent with the public IP address that you specify in the anti-crawler rule. Use the correct public IP address. We recommend that you click Alibaba Network Diagnose Tool to query your public IP address.
The test requests failed the verification. See WAF documentation or contact us to analyze the possible causes. You do not simulate real user access. For example, you use the debugging mode or automation tools, but do not simulate real user access. Simulate real user access during the test.
An incorrect service type is selected. For example, you select apps for Websites protection. Change the value of the Service Type parameter.
An intermediate domain name is used but is not correctly configured in the anti-crawler rule. Modify the anti-crawler rule, select Use an Intermediate Domain Name, and select the source domain name from the drop-down list.
Frontend incompatibility issues occur. Contact customer service in the DingTalk group or submit a ticket.
No verification is triggered. See WAF documentation or contact us to analyze the possible causes. The anti-crawler test rule is not sent. Perform the test several times until the anti-crawler test rule is sent.
No valid test requests are detected or blocked. See WAF documentation or contact us to analyze the possible causes. The test request fails to be sent or is not sent to WAF. Verify that the test request is sent to the IP address that maps the CNAME provided by WAF.
The header field in the test request does not match the header fields that you configure in Traffic Characteristics in the anti-crawler rule. Modify the settings of Traffic Characteristics in the anti-crawler rule.
The source IP address of the test request is inconsistent with the public IP address that you specify in the anti-crawler rule. Use the correct public IP address. We recommend that you click Alibaba Network Diagnose Tool to query your public IP address.
The test request is blocked. See WAF documentation or contact us to analyze the possible causes. Some code may have logic issues when the Anti-Bot SDK is integrated. As a result, requests have invalid signatures. For example, the content of a signature does not match the actual request, or the request does not have a signature. Check whether a signature issue occurs and fix the issue if possible. For more information, see App protection.
Proxies are used, or you do not use a real device. Perform the test again by using a real device.