Threat intelligence provides information about suspicious IP addresses of dialers, on-premises data centers, and malicious scanners based on the powerful computing capabilities of Alibaba Cloud. This feature also maintains a dynamic IP library of malicious crawlers and prevents crawlers from accessing your site or specific directories.

Notice This topic uses the new version of the Web Application Firewall (WAF) console released in January 2020. If your WAF instance was created before January 2020, you cannot set bot threat intelligence rules.

Prerequisites

  • A Web Application Firewall instance that is deployed in a region inside mainland China and the Bot Manager feature are available.Bot Manager
  • The website is associated with the Web Application Firewall instance. For more information, see Add domain names.

Background information

Bot threat intelligence rules can block requests from crawlers that are recorded in the Alibaba Cloud crawler library. The Alibaba Cloud crawler library is updated in real time based on the analysis of network traffic that flows through Alibaba Cloud, and captures the characteristics of requests that are initiated from crawlers. The Alibaba Cloud crawler library contains IP addresses of crawlers, public clouds, and on-premises data centers. The IP addresses are dynamically calculated and updated based on the threat intelligence collected from network traffic that flows through Alibaba Cloud.
Note IP addresses of public clouds and on-premises data centers are also contained in the crawler library because a large number of crawlers are deployed on cloud servers. However, general users rarely access your workloads through the source IP address of a public cloud or on-premises data center.

You can set a bot threat intelligence rule that chooses different actions to manage different requests based on the type of the threat intelligence library. For example, you can set a rule that blocks certain requests, or requires JavaScript verification or CAPTCHA verification to verify certain requests. You can also use a bot threat intelligence rule to protect important endpoints against certain threats. This helps you minimize the negative impacts on the service logic.

Procedure

  1. Log on to the Web Application Firewall console.
  2. In the top navigation bar, select the resource group to which the instance belongs and the region, Mainland China or International, in which the instance is deployed.
  3. In the left-side navigation pane, choose Protection Settings > Website Protection.
  4. In the upper part of the Website Protection page, select the domain name for which you want to configure the whitelist.Switch Domain Name
  5. Click the Bot Management tab and find the Bot Threat Intelligence section. Turn on the Status switch and click Settings. Bot threat intelligence
  6. In the Bot Threat Intelligence rule list, find the target threat intelligence library by Intelligence Name, and turn on the Status switch.Bot threat intelligence rules
    The following table lists the bot threat intelligence libraries that are supported by WAF.
    Intelligence library Description
    Malicious Scanner Fingerprint Blacklist This library contains characteristics of common scanners.
    Malicious Scanner IP Blacklist This library contains malicious IP addresses that are dynamically updated based on the source IP addresses of scan attacks detected on Alibaba Cloud.
    Credential Stuffing IP Blacklist This library contains malicious IP addresses that are dynamically updated based on the source IP addresses of credential stuffing and brute-force attacks detected on Alibaba Cloud.
    Fake Crawler Blacklist This library identifies crawlers that use the user agent of authorized search engines, such as BaiduSpider, to disguise as authorized programs.
    Notice Before you enable this library, make sure that you have configured the whitelist of crawlers. Otherwise, false positives may occur. For more information, see Set a threat intelligence rule to allow requests from specific crawlers.
    Malicious Crawler Blacklist This library contains malicious IP addresses that are dynamically updated based on the source IP addresses of crawlers detected on Alibaba Cloud. This library is categorized into three severity levels: low, medium, and high. A higher severity indicates more IP addresses in the library, and a higher false positive rate.
    Note We recommend that you set up two-factor authentication, such as CAPTCHA and JavaScript verification, for the high-severity library. In scenarios where two-factor authentication cannot be implemented, such as API calls, we recommend that you set threat intelligence rules based on the low-severity library.
    IDC IP List This library contains IP addresses of public clouds and on-premises data centers, including Alibaba Cloud, Tencent Cloud, Meituan Open Services, 21Vianet, and other public clouds. Attackers typically use CIDR blocks of public clouds or on-premises data centers to deploy crawlers or as proxies to access sites. General users rarely access sites in this way.
    After you enable the default rule, requests initiated from IP addresses in the threat intelligence library to any directory of the protected domain trigger the Monitor action. This action allows the requests to the destination directories and records the events.

    If you need to modify the default rule, such as the protected URL or action, see the following section on how to customize a threat intelligence rule.

  7. Optional:Customize a threat intelligence rule.
    1. Find the target rule, and click Edit in the Actions column.
    2. In the Edit Rule dialog box that appears, set the following parameters.Edit a bot threat intelligence rule
      Parameter Description
      Protected Path
      • URL: specifies the URL that you want to protect, such as /abc and /login/abc. You can also enter a single forward slash (/) to include all directories.
      • Matching: specifies a condition for matching the URL.
        • Precise Match: The destination URL must be an exact match of the protected URL.
        • Prefix Match: The prefix of the destination URL matches the protected URL.
        • Regular Expression Match: The destination URL matches the specified regular expression.

      You can click Add Protected URL to add more URLs. You can add up to 10 URLs.

      Action Specifies the action to be performed after the match conditions of the rule are met. Supported actions include:
      • Monitor: allows the request to the destination directory and records the event.
      • Block: blocks the request.
      • JavaScript Validation: requires JavaScript verification. Request are forwarded to the destination directory only after they pass the verification.
      • Captcha: requires CAPTCHA verification on the client side. Requests are forwarded to the destination directory only after they pass the verification.
        Note CAPTCHA only supports synchronous requests. To verify asynchronous requests, such as Ajax requests, contact the Alibaba Cloud security team. If you cannot determine whether the protected URL supports CAPTCHA, we recommend that you create a custom protection policy, such as an ACL rule, to run a test.
      • Strict Captcha: requires CAPTCHA verification on the client side. Requests are forwarded to the destination directory only after they pass the verification. CAPTCHA verification has a stricter standard to verify visitor identities.
    3. Click Confirm.