Problem description
When your service is added to an Anti-DDoS Proxy instance, you may experience issues such as slow response, high latency, and access failures.
Problem analysis
Collect the affected IP addresses and test connectivity by using tools such as traceroute or MTR to identify the cause.
mtr --no-dns [$IP]
Note [$IP] indicates the affected IP address that you collect.
The following command output shows that latency occurs on a node:
Determine if the latency is acceptable.
The traffic proxying and security monitoring of Anti-DDoS Proxy can introduce some latency, which varies by client regions and ISPs. The table below shows reference latency values after adding your service to an Anti-DDoS Proxy instance.
If latency is within the normal range, consider the following methods to reduce it:
Select a node geographically close to your origin server. This choice ensures that traffic is managed by a node nearer to your server, reducing the data transmission distance and the number of hops, which in turn minimizes latency. Anti-DDoS Proxy offers nodes in regions such as China (Beijing) and China (Hangzhou). To choose a node, please reach out to your account manager. For more information, see Purchase an Anti-DDoS Proxy instance.
Utilize the Sec-Traffic Manager feature of Anti-DDoS Proxy to protect your Alibaba Cloud services. This feature routes network traffic through Anti-DDoS Proxy only during an attack, allowing normal traffic to directly reach the origin server when no attack is present. For more information, see Sec-Traffic Manager.
Employ network acceleration services like Alibaba Cloud CDN (CDN), Dynamic Content Delivery Network (DCDN), or Global Traffic Manager (GTM). While Anti-DDoS Proxy safeguards against DDoS and other network threats, services such as CDN, DCDN, and GTM are designed to enhance network performance, access speed, and user experience, and to reduce latency. Note that these acceleration services are available as paid options.
If latency significantly exceeds the normal range, examine whether the Host value of the latency node is the IP address of your origin server or the IP address of the Anti-DDoS Proxy instance.
Solutions
If you want to immediately access your service, we recommend that you bypass Anti-DDoS Proxy and directly access the origin server. This ensures normal access to your service. Then, troubleshoot the issue based on the following sections.
Troubleshooting for the exceptions on origin servers
Troubleshoot the exceptions based on the type of your origin server.
Type of origin server | Troubleshooting |
Server Load Balancer (SLB) instance | Use the TCPing tool to ping the IP address and port of the SLB instance and check whether an exception occurs. For more information, see Troubleshooting for traffic scrubbing events. Check the status of the SLB instance. For example, check whether the number of connections to the instance exceeds the Max Connection specification of the instance. Check whether access control policies are configured. If access is denied, allow the back-to-origin CIDR blocks of your Anti-DDoS Proxy instance on the origin server. For more information, see Allow back-to-origin IP addresses to access the origin server. Check whether the security software or IP blocking policy on the backend server of the SLB instance denies access from the back-to-origin CIDR blocks of your Anti-DDoS Proxy instance by mistake. If access is denied, allow the back-to-origin CIDR blocks of your Anti-DDoS Proxy instance on the origin server. For more information, see Allow back-to-origin IP addresses to access the origin server.
Note If a backend server is associated with an SLB instance, the backend server cannot identify the originating IP address of the client because Layer 7 load balancing is not used. The backend server considers all requests as being initiated from the back-to-origin CIDR blocks of an Anti-DDoS Proxy instance. As a result, the backend server considers that each back-to-origin CIDR block initiates a large number of requests. In this case, the back-to-origin CIDR blocks of an Anti-DDoS Proxy instance may be blocked by the security software by mistake. You must allow these IP addresses on the origin server. Check whether the IP address of the SLB instance is exposed. If you cannot determine the status of the IP address or the IP address has been exposed, we recommend that you use another SLB instance. Otherwise, attackers may bypass Anti-DDoS Proxy to attack the origin server.
|
Elastic Compute Service (ECS) instance | Use the TCPing tool to ping the IP address and port of the ECS instance and check whether an exception occurs based on logs. For more information, see Troubleshooting for traffic scrubbing events. Check whether exceptions occur on the ECS instance. Exceptions include high CPU utilization, slow processing of database requests, and high bandwidth of outbound traffic. You can also check blackhole filtering events and traffic scrubbing events. Check whether access control policies, such as security groups and security software, are configured. If access is denied, allow the back-to-origin CIDR blocks of your Anti-DDoS Proxy instance on the origin server. For more information, see Allow back-to-origin IP addresses to access the origin server. Check whether a security group that allows requests from the originating IP addresses of your non-website service to the ECS instance is added. Check whether the IP address of the ECS instance is exposed. If you cannot determine the status of the IP address or the IP address has been exposed, attackers may bypass Anti-DDoS Proxy to attack the origin server. In this case, we recommend that you change the IP address of the ECS instance. For more information, see Change the public IP address of an ECS origin server.
|
Server not deployed on Alibaba Cloud | Use the TCPing tool to ping the IP address and port of the server and check whether an exception occurs based on logs. For more information, see Troubleshooting for traffic scrubbing events. Check whether exceptions occur on the server. Exceptions include high CPU utilization, slow processing of database requests, and high bandwidth of outbound traffic. Check whether access control policies, such as a blacklist, a whitelist, and security software, are configured. If access is denied, allow the back-to-origin CIDR blocks of your Anti-DDoS Proxy instance on the origin server. For more information, see Allow back-to-origin IP addresses to access the origin server. Check whether the IP address of the server is exposed. If you cannot determine the status of the IP address or the IP address has been exposed, we recommend that you change the IP address of the server. Otherwise, attackers may bypass Anti-DDoS Proxy to attack the origin server.
|
Troubleshooting for an Anti-DDoS Proxy instance
View the status of the instance on the Instances page of the Anti-DDoS Pro console. Then, determine the handling methods based on the status.
Scrubbing
If network traffic exceeds the traffic scrubbing threshold, the instance scrubs traffic. Traffic scrubbing events may cause slow response or latency.
Blackholing
If volumetric attacks are launched to your server and blackholing filtering is triggered for your server, the server is accessible only from Alibaba Cloud and services in the same region as the server. Traffic from other sources is denied.
Troubleshooting for traffic scrubbing events
The following figure shows that traffic scrubbing events occurred and caused the exception. You can use the TCPing tool to ping the affected and unaffected ports and check whether latency or packet loss occurs.
Identify the cause and resolve the issue based on the following table.
Latency and packet loss on affected ports | Latency and packet loss on unaffected ports | Solution |
Yes | No | The traffic scrubbing policy does not cause high latency and packet loss. We recommend that you check the status and attack mitigation capabilities of the backend server. If the backend server does not have sufficient capabilities to mitigate attacks, adjust the mitigation policies of your Anti-DDoS Proxy instance to harden the security of the server. You can analyze the attack mitigation capabilities of your server and adjust the mitigation policies based on the following details: |
Yes | Yes | The traffic scrubbing policy causes latency and packet loss. |
No | No | The traffic scrubbing policy does not cause latency or packet loss. |
No | Yes | This case does not exist. |
Troubleshooting for blackhole filtering events
The following figure shows that blackhole filtering events occurred. In this case, check the IP address that is in the blackholing state and check whether the affected service used the IP address. 
We recommend that you deactivate blackhole filtering. Each Alibaba Cloud account can deactivate blackhole filtering up to five times per day. For more information, see Deactivate blackhole filtering.
More information
This section describes how to use TCPing to check the port status, detect TCP latency, and view connection information by using TCP connections. Click TCPing to download the TCPing tool.
Use TCPing on Windows
Copy the TCPing tool to the specified directory in Windows and run the tcping [$Domain_Name] [$Port] command.
The following output is returned:
Probing 192.168.XX.XX:80/tcp - Port is open - time=19.550ms
Probing 140.XXX.XXX.8:80/tcp - Port is open - time=8.761ms
Probing 192.168.XX.XX:80/tcp - Port is open - time=10.899ms
Probing 192.168.XX.XX:80/tcp - Port is open - time=13.013ms
Ping statistics for 192.168.XX.XX:80
4 probes sent.
4 successful, 0 failed.
Approximate trip times in milliseconds:
Minimum = 8.761ms, Maximum = 19.550ms, Average = 13.056ms
Use TCPing on Linux
Run the following commands in sequence to install TCPing:
tar zxvf tcping-1.3.5.tar.gz
cd tcping-1.3.5
make tcping.linux
Run the following command to test connectivity:
for ((i=0; i<10; ++i)) ; do ./tcping www.example.com 80;done
The following output is returned:
www.example.com port 80 open.
www.example.com port 80 open.
www.example.com port 80 open.
www.example.com port 80 open.
www.example.com port 80 open.
www.example.com port 80 open.
www.example.com port 80 open.
www.example.com port 80 open.
www.example.com port 80 open.
www.example.com port 80 open.
Applicable scope
Anti-DDoS Proxy (Chinese Mainland) and Anti-DDoS Proxy (Outside Chinese Mainland).