All Products
Search
Document Center

After services are added to an Anti-DDoS Pro or Anti-DDoS Premium instance, issues, such as slow response, high latency, and access failures, occur. What do I do?

Last Updated: Sep 17, 2021

Problem description

After services are added to an Anti-DDoS Pro or Anti-DDoS Premium instance, issues, such as slow response, high latency, and access failures, occur.

Causes

  1. Collect the affected IP addresses and test connectivity by using tools, such as traceroute or MTR, to identify the cause.
    Note:
    mtr --no-dns [$IP]
    Note: [$IP] indicates the affected IP address that you collect.
    The command output shows that latency occurs on a node.
  2. Check whether the Host value of the node is the IP address of your origin server or the IP address of the Anti-DDoS Pro or Anti-DDoS Premium instance.

Solutions

If you want to immediately access the services, we recommend that you bypass Anti-DDoS Pro or Anti-DDoS Premium and directly access the origin server. This ensures normal access to the services. Then, handle the issues based on the following sections.

Troubleshooting for the exceptions on origin servers

Troubleshoot the exceptions based on the type of your origin server.

Type of origin server Troubleshooting
Server Load Balancer (SLB) instance
  1. Use the TCPing tool to ping the IP address and port of the SLB instance and check whether an exception occurs. For more information, see Troubleshooting for traffic scrubbing events.
  2. Check the status of the SLB instance. For example, check whether the number of connections to the instance exceeds the Max Connection specification of the instance.
  3. Check whether access control policies are configured.
    If access control policies are configured, allow the back-to-origin IP addresses of an Anti-DDoS Pro or Anti-DDoS Premium instance on the origin server. For more information, see Allow back-to-origin IP addresses to access the origin server.
  4. Check whether the security software or IP blocking policy on the backend server of the SLB instance denies access from the back-to-origin IP addresses of an Anti-DDoS Pro or Anti-DDoS Premium instance by mistake. If access is denied, allow the back-to-origin IP addresses of an Anti-DDoS Pro or Anti-DDoS Premium instance on the origin server. For more information, see Allow back-to-origin IP addresses to access the origin server.
    Note : If a backend server is associated with an SLB instance, the backend server cannot identify the actual source IP address of the client because Layer 7 load balancing is not used. The backend server considers all requests as being initiated from the back-to-origin IP addresses of an Anti-DDoS Pro or Anti-DDoS Premium instance. As a result, the backend server considers that each back-to-origin IP address initiates a large number of requests. In this case, the back-to-origin IP addresses of an Anti-DDoS Pro or Anti-DDoS Premium instance may be blocked by the security software by mistake. You must allow these IP addresses on the origin server.
  5. Check whether the IP address of the SLB instance is exposed. If you cannot determine the status of the IP address or the IP address has been exposed, we recommend that you use another SLB instance. Otherwise, attackers may bypass Anti-DDoS Pro or Anti-DDoS Premium to attack the origin server.
Elastic Compute Service (ECS) instance
  1. Use the TCPing tool to ping the IP address and port of the ECS instance and check whether an exception occurs based on logs. For more information, see Troubleshooting for traffic scrubbing events.
  2. Check whether exceptions occur on the ECS instance. Exceptions include high CPU utilization, slow processing of database requests, and high bandwidth of outbound traffic. You can also check blackhole filtering events and traffic scrubbing events.
  3. Check whether access control policies, such as security groups and security software, are configured. If access control policies are configured, allow the back-to-origin IP addresses of an Anti-DDoS Pro or Anti-DDoS Premium instance on the origin server. For more information, see Allow back-to-origin IP addresses to access the origin server.
  4. Check whether a security group that allows requests from the actual source IP addresses of non-website services to the ECS instance is added. For more information, see "The access to non-website services is abnormal. What do I do?"
  5. Check whether the IP address of the ECS instance is exposed. If you cannot determine the status of the IP address or the IP address has been exposed, attackers may bypass Anti-DDoS Pro or Anti-DDoS Premium to attack the origin server. In this case, we recommend that you change the IP address of the ECS origin instance. For more information, see Change the public IP address of an ECS origin server.
Server that is not deployed on Alibaba Cloud
  1. Use the TCPing tool to ping the IP address and port of the server and check whether an exception occurs based on logs. For more information, see Troubleshooting for traffic scrubbing events.
  2. Check whether exceptions occur on the server. Exceptions include high CPU utilization, slow processing of database requests, and high bandwidth of outbound traffic.
  3. Check whether access control policies, such as a blacklist, a whitelist, and security software, are configured.
    If access control policies are configured, allow the back-to-origin IP addresses of an Anti-DDoS Pro or Anti-DDoS Premium instance on the origin server. For more information, see Allow back-to-origin IP addresses to access the origin server.
  4. Check whether the IP address of the server is exposed. If you cannot determine the status of the IP address or the IP address has been exposed, we recommend that you change the IP address of the server. Otherwise, attackers may bypass Anti-DDoS Pro or Anti-DDoS Premium to attack the origin server.

Troubleshooting for an Anti-DDoS Pro and Anti-DDoS Premium instance

View the status of the instance on the Instances page of the Anti-DDoS Pro console. Then, determine the handling methods based on the status.

  • Mitigating
    If network traffic exceeds the traffic scrubbing threshold, the instance scrubs traffic. Traffic scrubbing events may cause slow response or latency.
  • Blackholing
    If volumetric attacks are launched to your server and blackholing filtering is triggered for your server, the server is accessible only from Alibaba Cloud and services in the same region as the server. Traffic from other sources is denied.

Troubleshooting for traffic scrubbing events

A traffic scrubbing event occurred and caused an exception. You can use the TCPing tool to ping the affected and unaffected ports and check whether latency or packet loss occurs.

 

Identify the cause and address the issue based on the following table.
Latency and packet loss on affected ports

 

Latency and packet loss on unaffected ports

 

Solution
Yes No

The traffic scrubbing policy does not cause latency or packet loss.

We recommend that you check the attack mitigation capabilities of the backend server. If the backend server does not have sufficient capabilities to mitigate attacks, harden the security of the server. You can analyze the attack mitigation capabilities of your server and adjust the mitigation policies based on the following details. Alternatively, you can submit a ticket to contact Alibaba Cloud technical support.

  • Access statistics
  • Service interaction process
  • Performance data
Yes Yes The traffic scrubbing policy causes latency and packet loss.

Submit a ticket to contact the after-sales technical support.

No No The traffic scrubbing policy does not cause latency or packet loss.
No Yes This case does not exist.

Troubleshooting for blackhole filtering events

  1. Blackhole filtering event occurred. In this case, check the IP address that is in the blackholing state and whether the affected service used the IP address.
  2. We recommend that you enable the Deactivate Blackhole feature to deactivate blackhole filtering. Each Alibaba Cloud account can deactivate blackhole filtering up to five times per day. For more information, see Deactivate blackhole filtering.

Other scenarios

If the issue persists, we recommend that you submit a ticket to contact Alibaba Cloud technical support. The following information must be included in the ticket to help the technical support resolve the issue quickly.
Region Source IP address IP address of the Anti-DDoS Pro or Anti-DDoS Premium instance that protects your services Ping information Traceroute or tracert information TCPing or port connection information
Mainland China or outside mainland China Example:192.168.XX.XX Example:203.168.XX.XX Results of more than 10 consecutive ping requests Tracert or traceroute information from the source IP address to the IP address of the Anti-DDoS Pro or Anti-DDoS Premium instance Results of more than 10 consecutive TCPing requests or the port connection information
The following information also helps the Alibaba Cloud technical support identify the issue:
  • The type of origin server, such as an SLB instance, an ECS instance, or a server that is not deployed on Alibaba Cloud.
  • The IP address and logs for the origin server. The logs include the CPU utilization, memory usage, bandwidth, and number of connections. The origin server can be an SLB instance, an ECS instance, or a server that is not deployed on Alibaba Cloud.
  • Whether an access control policy is configured for the origin server.
  • Whether security software, such as Fortinet and the iptables firewall, is installed on the origin server.
  • Whether a security policy, such as the detection and filtering for specific IP addresses, is applied to the origin server.
  • Whether a traffic scrubbing or blackhole filtering event occurs on the Anti-DDoS Pro or Anti-DDoS Premium instance.
  • The type of service that is protected, such as websites, client-based games, browser games, or apps.
  • The time when the issue occurred, and whether the Anti-DDoS Pro or Anti-DDoS Premium instance is modified or deleted.

Additional information

This section describes how to use TCPing to check the port status, detect TCP latency, and view connection information by using TCP connections. Click TCPing to download the TCPing tool.

  • Use TCPing in Windows
    Copy the TCPing tool to the specified directory in Windows and run the tcping [$Domain_Name] [$Port] command.
    Note:
    • [$Domain_Name] indicates the domain name or IP address for which you want to test connectivity.
    • [$Port] indicates the port for which you want to test connectivity.
    The following output is returned:
    Probing 192.168.XX.XX:80/tcp - Port is open - time=19.550ms
    Probing 192.168.XX.XX:80/tcp - Port is open - time=8.761ms
    Probing 192.168.XX.XX:80/tcp - Port is open - time=10.899ms
    Probing 192.168.XX.XX:80/tcp - Port is open - time=13.013ms
    
    Ping statistics for 192.168.XX.XX:80
         4 probes sent.
         4 successful, 0 failed.
    Approximate trip times in milli-seconds:
         Minimum = 8.761ms, Maximum = 19.550ms, Average = 13.056ms
  • Use TCPing in Linux
    1. Run the following commands in sequence to install TCPing:
      tar zxvf tcping-1.3.5.tar.gz
      cd tcping-1.3.5
      make tcping.linux
    2. Run the following command to test connectivity:
      for ((i=0; i<10; ++i)) ; do ./tcping www.example.com 80;done
      The following output is returned:
      www.example.com port 80 open.
      www.example.com port 80 open.
      www.example.com port 80 open.
      www.example.com port 80 open.
      www.example.com port 80 open.
      www.example.com port 80 open.
      www.example.com port 80 open.
      www.example.com port 80 open.
      www.example.com port 80 open.
      www.example.com port 80 open.

Application scope

  • Anti-DDoS Pro and Anti-DDoS Premium