The allowed crawlers function maintains a whitelist of authorized search engines, such as Google, Bing, Baidu, Sogou and Yandex. The crawlers of these search engines are allowed to access all pages on domain names.
Prerequisites
- A WAF instance that meets the following requirements is purchased:
- The instance is billed on a subscription basis.
- Bot Management is enabled. This feature is a value-added service.
For more information, see Purchase a WAF instance.
- Your website is added to WAF. For more information, see Add a website.
Background information
Rules defined in the function allow requests from specific crawlers to the target domain name based on the Alibaba Cloud crawler library. The Alibaba Cloud crawler library is updated in real time based on the analysis of network traffic that flows through Alibaba Cloud, and captures the characteristics of requests that are initiated from crawlers. The crawler library is updated dynamically and contains crawler IP addresses of mainstream search engines, including Google, Baidu, Sogou, Bing, and Yandex.