Identify and process bot traffic - ESA bot management - Edge Security Acceleration

Bot management identifies and responds to automated traffic. To classify and process traffic, ESA bot management uses data collection, detection engines, dynamic risk scoring, verification challenges, and continuous learning.

Identify bot traffic

ESA bot management provides two modes. Smart Mode uses AI-based decision-making to identify bot traffic. Professional Mode lets you configure identification policies such as whitelists, blacklists, request behavior analysis, and request feature analysis.

AI decision-making

This mode uses AI models and threat intelligence to score requests by risk level, classifying them as Verified Bots, Definite Bots, or Likely Bots.

Apply different actions to each category:

Verified Bots are typically search engine crawlers that benefit your SEO and site visibility. Configure the Allow action for this traffic.
Traffic categorized as Definite Bots typically contains malicious crawler requests that threaten your website. Configure the Block action for this traffic.
The threat level for Likely Bots falls between the other two categories. Because this traffic mixes legitimate and malicious requests, configure a Slider CAPTCHA to filter automated traffic.

Whitelist

A whitelist ensures trusted traffic and beneficial crawlers access your content without being blocked.

Policy	Use cases
Legitimate search engine whitelist	Crawlers from search engines like Google and Bing bring sustainable, free, and organic traffic critical for content and commercial websites. ESA bot management lets you allow specific search engine crawlers to bypass all bot management checks.
IP whitelist	Add the source IP addresses of known-safe requests to an IP whitelist to have them bypass bot protection.

Blacklist

A blacklist blocks globally recognized malicious bots. Requests that match the blacklist are immediately marked as malicious. You can also use it to proactively block traffic from insecure or malicious infrastructure.

Policy	Use cases
IDC blacklist	If your users do not typically connect from a public cloud or an IDC, block all IDC requests to mitigate risk. Whitelist known legitimate services from IDCs, such as payment gateway callbacks or monitoring services.
Bot threat intelligence library	Block requests from IP addresses with known malicious crawling activity. This library is maintained by Alibaba Cloud and continuously updated with platform-wide data.

Identify bots by request characteristics

This method analyzes static properties of HTTP/HTTPS requests, such as headers, protocol integrity, and TLS fingerprints, to identify non-standard behavior patterns.

Policy	Use cases
JavaScript challenge	Challenges the client to verify JavaScript execution. Requests from non-browser tools that cannot execute JavaScript are considered malicious and are blocked.
Dynamic token challenge	For each client request, ESA generates a dynamic, one-time encrypted token based on request context such as timestamp, user behavior, and IP address. ESA signs this token cryptographically to prevent tampering. When enabled, ESA verifies each request against its signature and blocks those that fail.

Identify bots by request behavior

Traffic behavior identification analyzes request patterns to distinguish human users from automated scripts.

Process bot traffic

Basic protection policies

Policy	Use cases
Allow	Grants access to known legitimate bot traffic, such as search engine crawlers and partner API calls.
Block	Blocks clearly malicious bots, such as those used for DDoS attacks or vulnerability scanning. Use this for traffic identified as Definite Bots.
Monitor	Monitors and logs suspicious traffic without blocking it, allowing for further analysis.
Slider CAPTCHA	Uses a human-computer verification test, such as a slider or a click-based challenge, to distinguish between humans and automated scripts.

Custom throttling

To allow certain bot requests while preventing excessive access, use Custom Throttling to limit request rates from a single IP or session. When the rate exceeds a threshold, a protective action triggers.

Note

Custom throttling is available only in Professional Mode.

Configuration example 1

Requirement: Limit requests from a single IP address to a maximum of 10 requests per minute. If this limit is exceeded, block the IP for one hour. Alternatively, limit requests from the same IP to 100 requests per 30 minutes. If this limit is exceeded, issue a Slider CAPTCHA for the next 10 minutes.
Configuration: Add the following two conditions. Conditions are evaluated with OR logic, and you can add up to three conditions per rule.
- Condition 1: Set Statistical Interval to 60, threshold to 10, Action to Block, and Throttling Interval to 3600.
- Condition 2: Set Statistical Interval to 1800, threshold to 100, Action to Slider CAPTCHA, and Throttling Interval to 600.

Configuration example 2

Requirement: For requests that contain api in a header, limit the access rate to a maximum of 100 requests per 10 minutes. If this limit is exceeded, block the requests for one minute.
Settings: Statistical Interval is 600, threshold is 100, Action is Block, and Throttling Interval is 60.

Parameter limits

Valid ranges for custom throttling parameters:

Parameter type	Range
Statistical Interval	5 to 1,800 (seconds)
Threshold	2 to 50,000 (times)
Action	Options: Block Monitor Slider
Throttling Interval	60 to 86,400 (seconds)