:Troubleshoot slow response, high latency, and access failures for services protected by Anti-DDoS Proxy

Step 1: Collect affected IP addresses and test connectivity

Run MTR or traceroute against the affected IP address to identify where latency or packet loss occurs in the network path.

When to use MTR: MTR combines traceroute and ping for continuous path monitoring. Use it when you suspect packet loss or high latency at specific hops between your client and Anti-DDoS Proxy, or between the proxy and the origin server.

This section uses MTR as an example. Use MTR to analyze network paths covers both tools in detail.

mtr --no-dns [$IP]

Replace [$IP] with the affected IP address.

The following example output shows that latency occurs on a specific node:

Step 2: Determine whether the latency is within the normal range

Anti-DDoS Proxy adds latency from traffic proxying and security inspection. Actual latency varies by client region and ISP. The following table lists reference values.

If your latency falls within the reference range, try the tips in Reduce latency within the normal range. If it significantly exceeds the range, proceed to the next step.

Step 3: Identify where latency occurs

Examine the MTR output to determine whether the high-latency node is your origin server or your Anti-DDoS Proxy instance.

• Latency is on the origin server -- Troubleshoot origin server issues.

• Latency is on the Anti-DDoS Proxy instance -- Troubleshoot Anti-DDoS Proxy instance issues.

Server Load Balancer (SLB) instance

1. Use TCPing to ping the SLB instance IP and port. Check for latency or packet loss. Appendix: Use TCPing to test connectivity.

2. Check the SLB instance status. Verify that the connection count does not exceed the Max Connection specification.

3. Check whether access control policies on the SLB instance deny the Anti-DDoS Proxy back-to-origin CIDR blocks. If so, allow them. Allow back-to-origin IP addresses to access the origin server.

4. Check whether security software or IP blocking on the SLB backend server blocks the back-to-origin CIDR blocks. If so, allow them. Allow back-to-origin IP addresses to access the origin server.

   When Layer 7 load balancing is not used, the backend server sees all requests as originating from the Anti-DDoS Proxy back-to-origin CIDR blocks. The high request volume from each block may trigger security software to block them. Add these CIDR blocks to the origin server allow list.

5. Check whether the SLB instance IP address has been exposed. If exposed or unknown, use a different SLB instance to prevent attackers from bypassing Anti-DDoS Proxy.

Elastic Compute Service (ECS) instance

1. Use TCPing to ping the ECS instance IP and port. Check logs for exceptions. Appendix: Use TCPing to test connectivity.

2. Check the ECS instance for performance issues such as high CPU utilization, slow database queries, or high outbound bandwidth. Also check for blackhole filtering and traffic scrubbing events.

3. Check whether security groups or security software block the back-to-origin CIDR blocks. If so, allow them. Allow back-to-origin IP addresses to access the origin server.

4. If you use a non-website service, check whether a security group rule allows requests from the originating IP addresses to the ECS instance.

5. Check whether the ECS instance IP address has been exposed. If exposed or unknown, change it to prevent attackers from bypassing Anti-DDoS Proxy. Change the public IP address of an ECS origin server.

Server not deployed on Alibaba Cloud

1. Use TCPing to ping the server IP and port. Check logs for exceptions. Appendix: Use TCPing to test connectivity.

2. Check the server for performance issues such as high CPU utilization, slow database queries, or high outbound bandwidth.

3. Check whether access control policies (blocklist, allowlist, or security software) block the back-to-origin CIDR blocks. If so, allow them. Allow back-to-origin IP addresses to access the origin server.

4. Check whether the server IP address has been exposed. If exposed or unknown, change it to prevent attackers from bypassing Anti-DDoS Proxy.

Troubleshoot traffic scrubbing events

The following figure shows traffic scrubbing events on the Instances page.

When to use TCPing here: Compare affected ports with unaffected ports using TCPing. This reveals whether the scrubbing policy or your backend server is causing the latency.

Use TCPing to ping both affected and unaffected ports. Check for latency or packet loss, then use the following table to identify the cause.

Troubleshoot blackhole filtering events

1. The following figure shows a blackhole filtering event on the Instances page. Identify the IP address that is in the blackholing state and verify whether the affected service uses that IP address.

2. Deactivate blackhole filtering. Each Alibaba Cloud account can deactivate it up to five times per day. Deactivate blackhole filtering.

3. To prevent blackhole filtering from being triggered again after deactivation, upgrade your Anti-DDoS Proxy instance.

Path 1: Anti-DDoS Proxy > WAF (CNAME mode) > Cloud Firewall > backend server

In the WAF console, go to the Security report page. Check whether WAF protection rules mistakenly block the source IP address. If so, add the IP to the allowlist or allow the matched rule ID. Configure protection rules for the whitelist module to allow specific requests.

Path 2: Anti-DDoS Proxy > Cloud Firewall > WAF (cloud native mode) > backend server

1. In the Cloud Firewall console, go to the Log audit page. Check whether the IP address is mistakenly blocked by intrusion prevention system (IPS) rules.

2. Add the source IP address to an address book.

3. Add the address book to the IPS whitelist.

Use TCPing on Windows

Copy the TCPing tool to the target directory and run the following command:

tcping [$Domain_Name] [$Port]

[$Domain_Name] is the domain name or IP address you want to test.

[$Port] is the port you want to test.

Example output:

Probing 192.168.XX.XX:80/tcp - Port is open - time=19.550ms
Probing 140.XXX.XXX.8:80/tcp - Port is open - time=8.761ms
Probing 192.168.XX.XX:80/tcp - Port is open - time=10.899ms
Probing 192.168.XX.XX:80/tcp - Port is open - time=13.013ms

Ping statistics for 192.168.XX.XX:80
     4 probes sent.
     4 successful, 0 failed.
Approximate trip times in milliseconds:
     Minimum = 8.761ms, Maximum = 19.550ms, Average = 13.056ms

Use TCPing on Linux

1. Install TCPing:

      tar zxvf tcping-1.3.5.tar.gz
      cd tcping-1.3.5
      make tcping.linux

2. Test connectivity: Example output:

      for ((i=0; i<10; ++i)) ; do ./tcping www.example.com 80;done

      www.example.com port 80 open.
      www.example.com port 80 open.
      www.example.com port 80 open.
      www.example.com port 80 open.
      www.example.com port 80 open.
      www.example.com port 80 open.
      www.example.com port 80 open.
      www.example.com port 80 open.
      www.example.com port 80 open.
      www.example.com port 80 open.