In this tutorial, you configure scenario-specific anti-crawler rules for two common scenarios: a webpage logon page and a solution details page. Each example walks through how to identify target request characteristics and customize protection rules accordingly.
Example 1: Alibaba Cloud official website logon page
This example uses the logon page of the Alibaba Cloud International site (account.alibabacloud.com) to describe how to configure a scenario-specific anti-crawler protection rule.
Identify the target request
When you click the logon button on the Alibaba Cloud official website, the Alibaba Cloud International site logon page (account.alibabacloud.com) displays the System error. Please try again later. error message. To identify the logon request characteristics, open the Network panel in your browser DevTools and select the /newlogin/login.do request to view its General information.
The target request has the following characteristics:
| Field | Value |
| Request URL | https://passport.alibabacloud.com/newlogin/login.do?fromSite=6&appName=intl-aliyun |
| Request Method | POST |
| Status Code | 200 |
Key fields in the Request Headers:
:authority: passport.alibabacloud.com:method: POST:path: /newlogin/login.do?fromSite=6&appName=intl-aliyun
You can use these fields to troubleshoot logon issues.
Configure the protection rule
To log on, you must click the
icon on the Alibaba Cloud official website page to trigger the logon request. This is the final protection target in this example. Enter Logon for the protection scenario.The webpage logon process is completed in a browser environment. Select Websites for the protection target type.
The logon button is on a page under the account.alibabacloud.com domain, but the logon request is submitted to the passport.alibabacloud.com domain. This is a cross-domain scenario. Configure the protection policy on the passport.alibabacloud.com domain, select Use Intermediate Domain Name, and enter the account.alibabacloud.com domain.
The logon request is characterized by a URL that contains /newlogin/login.do and the POST request method. Define the protection target characteristics based on these characteristics.
Example 2: Alibaba Cloud official website solution details page
This example uses the solution details page of the Alibaba Cloud International site (alibabacloud.com/solutions) to describe how to configure a scenario-specific anti-crawler protection rule.
Identify the page structure
The Alibaba Cloud solution details page contains three tabs at the top: Featured Solutions, Solutions by Use Case (currently selected), and Solutions by Industry. Below the tabs, solutions such as 1688 Cloud Hub, AI Service Dispatch, Backup and Archive, and China Gateway are displayed as cards.
Configure the protection rule
The final protection targets in this example are the solution subpages under the alibabacloud.com/solutions page. Enter Solution details page for the protection scenario.
Access to this page is completed in a browser environment. Select Websites for the protection target type.
On the Alibaba Cloud solution details page (alibabacloud.com/solutions), the URL of a single solution page uses the
/solutions/xxxstructure. Set the protection target characteristics toURLPath contains /solutions/and the request method toGET. This example does not involve cross-domain requests. You do not need to select Use Intermediate Domain Name. You can also add more protection conditions, such as user-agent, param, and referer, based on your business scenario.