The bot management module of WAF is updated to provide the scenario-specific configuration feature. This feature allows you to configure anti-crawler rules based on your business requirements and helps protect your business from malicious crawlers. This topic describes how to configure anti-crawler rules for apps.

Prerequisites

  • If you use a subscription WAF instance that runs the Pro, Business, or Enterprise edition, the bot protection module and the apps protection feature are enabled.
  • Anti-Bot SDK is integrated into the apps that you want to protect. For more information, see Incorporate Anti-Bot SDK into apps.

Background information

The scenario-specific configuration feature allows you to configure anti-crawler rules based on your business requirements. This feature can be used in combination with the intelligent algorithm feature to precisely identify crawler traffic. In addition, this feature can automatically handle the crawler traffic that matches the configured anti-crawler rules. After you configure anti-crawler rules, you can verify the rules in a test environment. This prevents adverse effects on your websites or apps caused by inappropriate rule configurations or compatibility issues. The adverse effects include false positives and undesired protection results.

Configure anti-crawler rules for apps

  1. Log on to the Web Application Firewall console.
  2. In the top navigation bar, select the resource group and region to which the WAF instance belongs. The region can be Mainland China or International.
  3. In the left-side navigation pane, choose Protection Settings > Website Protection.
  4. In the upper part of the Website Protection page, select the domain name for which you want to configure the whitelist. Switch Domain Name
  5. Click the Bot Management tab. In the Scenario-specific Configuration section, click Start to create your first anti-crawler rule. Enable the scenario-specific configuration feature for the first time
    If you have created an anti-crawler rule, you can skip this step and click Add in the upper-right corner to create a rule.
  6. In the Configure Scenarios step, configure the basic information about the scenario in which you want to protect apps and click Next.
    Parameter description:
    • Scenario: Specify the type of scenario in which you want to protect the apps. Examples: logon, registration, and order placement.
    • Service Type: Select App to protect native iOS and Android apps.
      Note HTML5 apps are not native iOS or Android apps. If you want to protect HTML5 apps, set the Service Type parameter to Websites.
    • Traffic Characteristics: Add match conditions for requests destined for the apps that you want to protect by using the anti-crawler rule. For more information about header fields in the match conditions, see Fields in match conditions. You can specify up to five match conditions.
      Notice After you enter an IP address, you must press Enter.
  7. In the Configure Protection Rules step, configure detailed settings for the anti-crawler rule and click Next.
    Parameter description:
    • Check Invalid App Signature: After you enable this feature, the anti-crawler rule detects and controls the apps requests that have invalid signatures or do not have signatures. You cannot disable this feature. You can configure Action to handle these requests destined for the apps. If you set the Action parameter to Monitor, the anti-crawler rule allows the traffic that matches the rule and records the traffic in security reports and logs. If you set the Action parameter to Block, the anti-crawler rule blocks the traffic that matches the rule.
    • Check Abnormal Device Behaviors: After you enable this feature, the anti-crawler rule detects and controls the requests from the device that has abnormal characteristics.
      The following characteristics of a device are considered abnormal characteristics:
      • Use Simulators: A simulator is used.
      • Use Proxies: A proxy is used.
      • Use Rooted Device: A rooted device is used.
      • Debugging Mode: The debugging mode is enabled.
      • App hooked: Hooks are inserted.
      • App Open: Multiple protected apps processes run on the device at the same time.

      You can set the Action parameter to Monitor or Block based on your business requirements. If you set the Action parameter to Monitor, the anti-crawler rule allows the traffic that matches the rule and records the traffic in security reports and logs. If you set the Action parameter to Block, the anti-crawler rule blocks the traffic that matches the rule..

    • Action: You can set this parameter to Monitor or Block. This setting takes effect for Check Invalid App Signature and Check Abnormal Device Behaviors.
    • IP Address Throttling, Device Throttling, and Custom Session-based Throttling: After you enable these features, you can configure throttling conditions to filter abnormal requests. This way, HTTP flood attacks can be mitigated.
      • IP Address Throttling: You can specify throttling conditions for IP addresses. If the number of requests from an IP address within the specified time period exceeds the threshold, WAF performs the monitor or block action on subsequent requests. You can also specify the period during which the monitor or block action is performed. You can specify up to three conditions. For more information, see Create a custom protection policy.
      • Device Throttling: You can specify throttling conditions for devices. If the number of requests from the same device within the specified time period exceeds the threshold, WAF performs the monitor or block action on subsequent requests. You can also specify the period during which the monitor or block action is performed. You can specify up to three conditions.
      • Custom Session-based Throttling: You can specify throttling conditions for sessions. If the number of requests from the same session within the specified time period exceeds the threshold, WAF performs the monitor or block action on subsequent requests. You can also specify the period during which the monitor or block action is performed. You can specify up to three conditions. For more information, see Create a custom protection policy.
  8. Optional:In the Verify Actions step, check whether the anti-crawler rule is in effect.
    This step is optional. To skip this step, you can click Skip in the lower-left corner. Before you publish the rule, we recommend that you complete this step.
    Parameter description:
    • Public IP Address Test: Enter the public IP address of your test device, for example, a mobile phone. During the test, the anti-crawler rule takes effect only for the public IP address. Therefore, the test does not affect your business.
      Notice If you want to obtain the public IP address of your test device, you can click Alibaba Network Diagnose Tool. Then, on the page that appears, search for local IP. You can also use a browser to search for the IP address of your test device.
    • SDK Signature Verification: Click Start Test to verify that the SDK signature of apps is valid.
      Note Make sure that Anti-Bot SDK for apps is integrated into the test device. If the Anti-Bot SDK is not integrated into the device, the signature verification fails, normal requests are blocked, and the test cannot be completed.
    • Action Test: Check whether the Block action is in effect. After you click Start Test for an action, WAF immediately publishes the anti-crawler rule to the test device. WAF also provides the demonstration diagram and description of the test results. We recommend that you read the demonstration diagram and description of the test results.

      After the test is complete, you can click I Have Completed Test to go to the next step. If the test result shows exceptions, you can click Go Back to optimize the anti-crawler rule. Then, perform the test again.

      For more information about the exceptions that may occur during a test and about the solutions to handle these exceptions, see FAQ.

  9. In the Preview and Publish Protection Rules step, confirm the content of the anti-crawler rule and click Publish.
    After the anti-crawler rule is published, the rule immediately takes effect.
    Note If this is your first time to create an anti-crawler rule, you cannot view the rule ID until the rule is published. The rule ID is displayed on the Bot Management tab of the Security Report page. You can use the ID of an anti-crawler rule to check for requests that match the rule in Log Service for WAF.

FAQ

Error message Cause Solution
No valid test requests are detected. See WAF documentation or contact us to analyze the possible causes. The test request fails to be sent or is not sent to WAF. Verify that the test request is sent to the IP address that maps the CNAME provided by WAF.
The header fields in the test request do not match the header fields that you configure for Traffic Characteristics in the anti-crawler rule. Modify the settings of Traffic Characteristics in the anti-crawler rule.
The source IP address of the test request is different from the public IP address that you specify in the anti-crawler rule. Use the correct public IP address. We recommend that you click Alibaba Network Diagnose Tool to obtain your public IP address.
The test requests failed the verification. See WAF documentation or contact us to analyze the possible causes. No real user access is simulated. For example, the debugging mode or automation tools are used. Simulate real user access during the test.
An invalid service type is selected. For example, you select Websites for apps protection. Change the value of the Service Type parameter.
An intermediate domain name is used. The anti-crawler rule is incorrectly configured. Modify the anti-crawler rule. Specically, select Use an Intermediate Domain Name and select the source domain name from the drop-down list.
Frontend incompatibility issues occur. Contact customer service in the DingTalk group or submit a ticket.
Verification is not triggered. See WAF documentation or contact us to analyze the possible causes. No test rule is generated. Perform the test several times until the test rule is generated.
No valid test requests are detected or blocked. See WAF documentation or contact us to analyze the possible causes. The test request fails to be sent or is not sent to WAF. Verify that the test request is sent to the IP address that maps the CNAME provided by WAF.
The header fields in the test request do not match the header fields that you configure for Traffic Characteristics in the anti-crawler rule. Modify the settings of Traffic Characteristics in the anti-crawler rule.
The source IP address of the test request is different from the public IP address that you specify in the anti-crawler rule. Use the correct public IP address. We recommend that you click Alibaba Network Diagnose Tool to obtain your public IP address.
The test request is blocked. See WAF documentation or contact us to analyze the possible causes. Some code may have logic issues when Anti-Bot SDK is integrated. As a result, requests contain invalid signatures. For example, the content of a signature does not match the actual request, or the request does not have a signature. Check whether a signature issue occurs and fix the issue at the earliest opportunity. For more information, see App protection.
Proxies are used, or you do not use a real device. Use a real device to perform the test.