Server Load Balancer:FAQ about CLB listeners

Does CLB support port redirection?

Yes.

CLB supports port redirection. For example, see Redirect HTTP requests to HTTPS by using a CLB instance.

Do CLB Layer 4 listeners support port ranges?

No. To configure a TCP or UDP listener for a port range, create a Network Load Balancer (NLB) instance and enable the full-port feature for its listener. For more information, see Forward traffic across multiple ports with an NLB full-port listener.

Listener port configuration notes

Some carriers mark ports such as 25, 135, 139, 444, 445, 5800, and 5900 as high-risk and block them by default. Even if you allow traffic on these ports in security group rules, users in restricted regions cannot access your services. We recommend that you use other non-high-risk ports for your services.

• For information about ports used by applications on Windows Server, see Service overview and network port requirements for Windows.

• For more information about common ports, see Common ports.

Configuring listeners for WebSocket

• For backend servers that use WebSocket, you can configure a TCP or HTTP listener.

• For backend servers that use WebSocket Secure, you can configure a TCP or HTTPS listener.

Effect and impact of listener configuration changes

Changes take effect immediately and apply only to new requests. Existing connections are not affected.

Accessing URLs with special characters

Special characters in URLs must be URL-encoded to be accessed correctly. For example, the hash symbol (#) must be encoded as %23, and the URL should be entered as http://www.example.com/%23/. For the complete encoding rules, see RFC 3986.

Packet drops despite available bandwidth

This issue may occur for the following reasons:

• Alibaba Cloud's bandwidth monitoring data is averaged over one minute. Therefore, the chart may show usage below the limit even if a momentary traffic spike exceeds it, because the one-minute average remains within the limit.

• CLB uses a server cluster to distribute and forward requests. The specified maximum bandwidth is shared among these servers. If the amount of data a single client connection downloads exceeds the threshold of a single server, packets may be dropped. For more information about how the upper limit for download traffic on a single connection is calculated, see Failure to reach maximum bandwidth.

Monitored traffic exceeds rate limit

The load balancing system uses a cluster deployment to provide services for CLB instances and implements distributed rate limiting. The peak rate limit for a single node = Configured total bandwidth of the load balancer/(N-1), where N is the number of nodes in the cluster. Therefore, the overall rate limit is slightly higher than the configured value.

Failure to reach maximum bandwidth

• Scenario: Connections to a pay-by-bandwidth public-facing CLB instance may not reach the configured maximum bandwidth in specific scenarios, such as single-client stress testing or transferring very large data packets.

• Cause:

  CLB distributes external requests across a server cluster, and the configured maximum bandwidth is shared among these servers.

  The maximum download traffic for a single connection is calculated by using the following formula: Peak download speed per connection = Total configured bandwidth / (N-1). In this formula, N is the number of sub-nodes in the cluster. The value of N is 4 for a Layer-4 listener and 8 for a Layer-7 listener. For example, if you set a bandwidth limit of 10 Mbps in the console, the total bandwidth can reach 10 Mbps when multiple clients download data at the same time. The maximum download speed for a single client is 10/(4-1) = 3.33 Mbps.

• Recommendations:

  • Use the pay-by-data-transfer metering method for the public-facing CLB instance.

  • Use an Application Load Balancer (ALB) or Network Load Balancer (NLB) instance with an elastic IP address (EIP) and an EIP bandwidth plan. This solution provides sufficient elasticity and removes this limitation.

Failure to reach maximum QPS

• Scenario: In scenarios with a small number of persistent connections, not all servers in the forwarding group may be allocated to handle them. As a result, the CLB instance cannot reach its maximum QPS.

• Cause:

  CLB uses a cluster of servers to provide services. All external requests are distributed across these servers for forwarding. Therefore, the maximum QPS of the CLB instance is shared among the servers in the cluster.

  The maximum QPS for a single backend server is calculated by using the following formula: Maximum QPS of a single backend server = Total QPS of the instance / (N - 1). N is the number of backend servers in the server group. For example, a CLB instance of the slb.s1.small specification provides a total of 1,000 QPS. If the server group contains 8 backend servers, the maximum QPS for a single backend server is 1,000 / (8 - 1) = 142 QPS.

  Note

  New purchases of pay-by-specification CLB instances will no longer be available after 00:00:00 on June 1, 2025 (UTC+8). For more information, see End of Sale for Pay-by-specification CLB Instances.

• Recommendations:

  • Use short-lived connections from a single client for stress testing.

  • Reduce connection reuse based on your business requirements.

  • Upgrade the specification of the CLB instance. For more information, see Upgrade or downgrade a pay-as-you-go (pay-by-specification) instance.

  • Use an ALB instance. This solution provides sufficient elasticity.

Failure to reach peak new connection rate

• Scenario: When you purchase a CLB instance and select the pay-by-specification billing method, the new connection rate (CPS) may not reach the level stated in the specification in certain scenarios, such as single-client stress testing or traffic from a single source.

  Note

  New purchases of pay-by-specification CLB instances will no longer be available after 00:00:00 on June 1, 2025 (UTC+8). For more information, see End of Sale for Pay-by-specification CLB Instances.

• Cause:

  CLB uses a cluster deployment architecture to ensure high availability and scalability. Incoming connection requests are distributed among multiple servers in the cluster. Therefore, the peak CPS of a CLB instance is shared across these servers.

  The formula for the peak CPS of a single server is: Peak CPS per server = Total CPS of the instance / (N-1), where N is the number of servers in the forwarding group.

  For example, if you purchase a CLB instance of the slb.s1.small specification, its stated CPS is 3,000. When multiple clients access the instance concurrently, the overall CPS can reach 3,000. If the number of servers is 4, the CPS limit for a single server is 3,000 / (4 - 1) = 1,000 CPS.

• Recommendations:

  • Change the billing method to pay-as-you-go. Pay-as-you-go CLB instances do not require a specified specification and offer higher performance limits than most specification-based instances, which helps prevent performance bottlenecks.

  • Upgrade to a Network Load Balancer (NLB): For scenarios with high concurrency and a high rate of new connections, we recommend using NLB. NLB offers superior performance and elasticity compared to CLB. A single NLB instance supports 100 million concurrent connections, making it ideal for large-scale scenarios and helping you avoid the CPS limitations of CLB.

Connection timeout ranges by listener

• TCP listener connection timeout: 10 to 900 seconds.

• HTTP listener:

  • Idle connection timeout: 1 to 60 seconds.

  • Request timeout: 1 to 180 seconds.

• HTTPS listener:

  • Idle connection timeout: 1 to 60 seconds.

  • Request timeout: 1 to 180 seconds.

Connection timeouts to CLB service address

The following issues can cause connection timeouts to the CLB service address:

• The service address is under security protection.

  This includes traffic blackholing, scrubbing, and Web Application Firewall (WAF) protection. After a connection is established, WAF may send RST packets to both the client and the backend server.

• Client port exhaustion

  This often occurs during stress testing. Client port exhaustion can cause connection failures. By default, the load balancer removes the TCP connection's timestamp attribute. This prevents the Linux kernel's tw_reuse feature (reuse of connections in a TIME_WAIT state) from working, leading to a build-up of TIME_WAIT connections and a shortage of client ports.

  Solution: Use persistent connections instead of short-lived connections on the client side. Close connections by sending an RST packet (by setting the SO_LINGER socket attribute) instead of a FIN packet.

• The backend server's accept queue is full

  If the backend server's accept queue is full, it does not reply with SYN-ACK packets, causing the client to time out.

  The default value of net.core.somaxconn is 128. Evaluate your service traffic and adjust this value to meet your actual needs. Then, run the sysctl -w net.core.somaxconn=<new_value> command to change this parameter and restart the application on the backend server.

• Accessing a Layer 4 load balancer's service address from its own backend server

  A Layer 4 listener (TCP/UDP) does not allow a backend server to access its own service address through the load balancer. This configuration will cause connection failures. A common scenario is when a backend application redirects to the CLB's service address by constructing a URL.

  Solutions:

  • Access the service from a different client, not from the Layer 4 load balancer's backend server.

  • Migrate to Network Load Balancer (NLB) and disable the Client IP Preservation feature in the server group. This allows ECS instances in the server group to act as both backend servers and clients. To obtain the client's source IP address, enable Proxy Protocol. For more information, see How can an ECS instance act as both a backend server and a client in NLB?.

• Improper handling of RST packets for connection timeouts

  If a TCP connection is idle for 900 seconds, the load balancer sends an RST packet to both the client and the server to close it. Some applications handle these RST packets improperly and may try to send data on a closed connection, causing an application timeout.

  Note

  900 seconds is the system's default value and can be adjusted as needed.

HTTP and HTTPS connection timeouts

• For a persistent HTTP connection, a maximum of 100 consecutive requests can be sent. The connection is closed after this limit is exceeded.

• The idle timeout between two HTTP or HTTPS requests over a persistent connection is configurable from 1 to 60 seconds (with a potential error of 1 to 2 seconds). The TCP connection is closed after the timeout. If you require persistent connections, try to send a heartbeat request within 13 seconds.

• The timeout for the TCP three-way handshake between the load balancer and a backend ECS instance is 5 seconds. If the handshake times out, CLB selects the next ECS instance. You can identify this by checking the upstream response time in the access logs.

• The load balancer's response timeout for an ECS instance is configurable from 1 to 180 seconds. After this period, a 504 or 408 status code is typically returned to the client. Checking the upstream response time in the access logs can help locate the issue.

• The HTTPS session reuse timeout is 300 seconds. After this, the same client needs to perform a full SSL handshake again.

Backend connection behavior on client disconnect

The load balancer does not disconnect from the backend server during read and write operations.

Enabling persistent connections to backend servers

Classic Load Balancer (CLB) does not support persistent connections to backend servers. For this functionality, use an Application Load Balancer (ALB) instance, configure an HTTP or HTTPS listener, and enable persistent connections in the corresponding ALB server group. For more information, see Create and manage a server group.

Troubleshooting high latency on CLB

Accessing a service through CLB introduces slightly more latency than accessing the backend server directly; this is expected. A Layer 7 CLB listener uses a reverse proxy architecture (Tengine), where requests are forwarded by CLB. This adds latency from an extra network hop and protocol processing. A Layer 4 listener uses LVS for forwarding, and the additional latency is usually minimal.

If the latency is significantly high, follow these steps to troubleshoot:

1. Enable access logs and analyze latency fields: Enable CLB access logs and focus on the following fields:

   • request_time: The time interval in seconds from when CLB receives the first request packet to when it returns the response.

   • upstream_response_time: The time in seconds between establishing a connection with the backend and closing the connection after receiving the full response.

2. Identify the source of the latency:

   • If the upstream_response_time is high, the latency is usually caused by slow processing on the backend servers. Check the performance of your backend applications, the efficiency of database queries, and resource usage such as CPU and memory, or add more backend servers to distribute the load.

   • If request_time is much greater than upstream_response_time, the latency is likely on the network link from the client to CLB. You can continuously run ping tests or MTR route traces from the client to the CLB service address to troubleshoot network link issues.

3. Cross-region access scenarios: If the client and the CLB instance are in different regions, network latency due to physical distance is unavoidable. We recommend using the Global Accelerator (GA) service to optimize the cross-region access experience.

Troubleshooting 502, 503, and 504 errors

Receiving a 502, 503, or 504 error when accessing a service through CLB usually indicates that the backend server did not process the request correctly. The meanings of these error codes are as follows:

• 502 Bad Gateway: CLB cannot forward the request to the backend server or receive a response from it. Common causes include an unreachable backend service or all health checks failing.

• 503 Service Temporarily Unavailable: This is usually due to traffic exceeding limits or an unavailable backend server. This error code is returned when the instantaneous traffic of a request exceeds the limit of the CLB instance's specification.

• 504 Gateway Time-out: The backend server response timed out. Common causes are long processing times on the backend or a backend server connection timeout.

Step 1: Check access logs

We recommend that you first enable CLB access logs and check the following fields in the logs: status (the status code that CLB returns to the client) and upstream_status (the status code that the backend server returns to CLB).

• If status is the same as upstream_status, it is likely that CLB directly passed through an abnormal status code from the backend server. You should first investigate why the backend server returned this status code.

• If upstream_status is "-" or is different from status, CLB returns the error code. You can refer to the following points for troubleshooting.

Troubleshooting 502 errors

• All backend servers fail health checks: When all backend servers associated with a listener fail their health checks, CLB cannot forward requests and returns a 502 error. Check the health check status in the console and troubleshoot the cause of the failure, such as a security group that does not allow traffic from 100.64.0.0/10 (the CLB system address range), a mismatched health check status code configuration, or a non-existent health check path. See CLB Health Check FAQ.

• CLB converts backend error status codes to 502: If a backend server returns certain error status codes, such as 504 or 444, CLB may return a 502 error to the client. Check the upstream_status field in the access logs to identify the actual status code returned by the backend server and troubleshoot the backend server error.

• Backend service error: A 502 error can also be caused by high load on the backend server, an abnormal response format, or an improperly closed connection. Check the backend server logs and resource usage, such as CPU and memory.

Troubleshooting 503 errors

• Traffic exceeds instance specification limits: A 503 error is returned when the QPS, bandwidth, or number of new connections for incoming traffic exceeds the current specification limits. You can obtain these monitoring metrics from CloudMonitor.

• Instantaneous traffic exceeds the limit but is not shown in monitoring: CloudMonitor displays data at a one-minute granularity and may not reflect traffic bursts that exceed the limit within a second. You can check the number of requests per second in the access logs. When the upstream_status in the logs is "-", this indicates that the request was not sent to the backend server.

Troubleshooting 504 errors

• Backend response timeout: If a backend server does not return a response within the connection timeout period configured for the listener, CLB returns a 504 error. We recommend that you check the upstream_response_time field in the access logs to confirm the actual response time of the backend server, and adjust the connection timeout period of the listener accordingly.

• Backend connection timeout: The timeout for the TCP three-way handshake between a CLB instance and a backend ECS instance is 5 seconds. If the upstream_response_time in the access log is too long, this may indicate a connection error with the backend server. You can capture packets to investigate the cause.

• High backend load: High resource usage on the backend server, such as CPU or memory, can cause the response time to exceed the timeout period. Troubleshoot and optimize the performance of the backend service, or add more backend servers to distribute the load.

Troubleshooting service access failures

If you cannot access your service after configuring CLB, follow these steps to troubleshoot the issue layer by layer:

1. Verify DNS resolution: If you use a domain name for access, verify that the domain name is correctly resolved to the service address of the CLB instance. You can use the nslookup or dig command to verify the resolution result. Incorrect DNS resolution is a common cause of access failures.

2. Confirm listener configuration: In the CLB console, check whether a listener has been created and confirm that the listener port and protocol are correctly configured. If a listener is not added or is misconfigured, requests cannot be forwarded.

3. Confirm health check status: In the CLB console, check the health check status of the backend servers. When all backend servers fail their health checks, CLB cannot forward requests.

4. Check the security group and firewall: Verify that the security group rules and firewall settings of the backend server allow traffic on the backend service port and from the CLB system CIDR block 100.64.0.0/10.

5. Verify that the backend service is running properly: Log on to the backend server directly and run the telnet <Private IP address of the backend server> <Port> command (for Layer 4) or the curl -I http://<Private IP address of the backend server> command (for Layer 7) to confirm that the backend service can respond properly.

6. Troubleshoot the network link: Test access to the CLB service address from different network environments. If an issue occurs only on your local network, you can perform further troubleshooting by using continuous ping tests or MTR route tracing.

Domain access failure with successful IP access

The most common reason is that the domain name has not completed its ICP filing.

According to regulatory requirements, when you use a domain name for public access in the Chinese mainland, the domain name must have an ICP filing. Access to a domain name without an ICP filing will be blocked, resulting in a 403 status code or a reset connection.

We recommend that you follow these steps to troubleshoot and resolve the issue:

1. Confirm the ICP filing status of the domain name: Log on to the Alibaba Cloud ICP Filing system to check whether the domain name has completed its filing. If not, complete the ICP filing first. For more information, see ICP filing process.

2. Confirm whether an ICP filing transfer is required: If the domain name has an ICP filing with another cloud service provider but is being used with Alibaba Cloud for the first time, you must also perform an ICP filing transfer to move the filing information to Alibaba Cloud. An incomplete transfer can also result in blocked access.

3. Rule out other causes: If the domain has completed its ICP filing and access filing, check that the domain correctly resolves to the CLB service address (you can use the nslookup or dig command to verify this), and that the CLB listener port and protocol configuration match the way the domain is accessed.

Access control impact on private network

Yes. Access control is applied at the listener level and affects both private and public network access. If your allowlist only permits specific public IP addresses, it will block requests from internal IP addresses that are not on the list. To avoid affecting internal services, add the relevant internal network segments to the allowlist, or use Cloud Firewall to restrict public access to the EIP.

Troubleshooting timeouts during stress testing

If you perform a stress test on a Layer-7 CLB and receive 504 status codes and request timeouts, and the upstream_response_time in the logs clusters around 5 seconds, the issue is typically a connection timeout caused by a failed TCP three-way handshake between the CLB and the backend servers. A common reason for this is that the connection tracking table (nf_conntrack) on a backend server is full, which causes the server to drop packets for new connections.

Log in to the backend server to check the /var/log/messages log. The issue is confirmed if the following error message appears:

nf_conntrack: table full, dropping packet

Solution: Adjust the nf_conntrack parameters (adjust the parameter values based on your actual business requirements):

sysctl -w net.netfilter.nf_conntrack_max=1048576
sysctl -w net.netfilter.nf_conntrack_buckets=262144
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=3600

Note: The preceding command takes effect only temporarily and does not persist after the instance is restarted. To make the change permanent, add the parameters to /etc/sysctl.conf.

Site inaccessibility due to database failure

Scenario: Multiple websites, such as a static website www.example.com and a dynamic website app.example.com, are attached to a CLB listener. When the backend database of the dynamic website fails, the static website also becomes inaccessible and returns an HTTP 502 error.

Cause: Both websites share the same listener, and the listener's Health Check Domain Name is configured to the domain of the dynamic website. When the backend of the dynamic website fails, all backend servers fail their health checks. As a result, CLB stops forwarding traffic to the backend, which affects access to all sites under the listener.

Solution: Use separate CLB instances for the dynamic and static websites to isolate the services. This way, a failure in the dynamic website will not affect access to the static website.

Causes of session persistence failure

• Session persistence is not enabled: Check whether session persistence is enabled in the listener configuration.

• HTTP/HTTPS listener issue: An HTTP or HTTPS listener will not insert a session persistence cookie into a response if the backend server returns a 4xx status code.

  Solution: Switch to a TCP listener, which uses the client's source IP address for session persistence. You can also insert a cookie on the backend ECS instance and add a cookie check for extra assurance.

• 302 redirect issue: A 302 redirect can break session persistence by changing the SERVERID string.

  When the load balancer inserts a cookie, if the backend ECS instance returns a 302 redirect response, the SERVERID string in the cookie is changed, causing session persistence to fail.

  Troubleshooting: Capture the request and response in a browser or use a packet capture tool to analyze whether a 302 response is present. Compare the SERVERID strings in the cookies before and after to see if they are different.

  Solution: Switch to a TCP listener, which uses the client's source IP address for session persistence. You can also insert a cookie on the backend ECS instance and add a cookie check for extra assurance.

• Session persistence timeout is too short: If the session persistence timeout is set to a small value, session persistence may fail.

Viewing the session persistence string

You can press F12 in your browser to check if the response contains the 'SERVERID' string or a user-specified keyword, or run curl www.example.com -c /tmp/cookie123 to save the cookie, and then use curl www.example.com -b /tmp/cookie123 to access the URL.

Testing session persistence with curl

1. Create a test page.

   On each backend ECS instance, create a test page that displays the instance's private IP address. This IP identifies which server handles a request, allowing you to verify that session persistence is effective by checking for a consistent IP.

   

2. Run the curl command on a Linux system.

   Assume that the load balancer service IP address is 10.170.XX.XX, and the test page URL is http://10.170.XX.XX/check.jsp.

   1. Log on to the Linux server that you want to use for testing.

   2. Run the following command to query the cookie value inserted by CLB.

      curl -c test.cookie http://10.170.XX.XX/check.jsp

      Note

      The default mode for Alibaba Cloud session persistence is cookie insertion. However, a curl test does not save or send cookies by default. Therefore, you must first save the cookie for the test. Otherwise, the curl test results will be random, which may lead you to mistakenly believe that session persistence is not working.

   3. Run the following command to perform continuous testing.

      for ((a=1;a<=30;a++));
          do curl  -b test.cookie http://10.170.XX.XX/check.jsp  | grep '10.170.XX.XX';
          sleep 1;
      done

      Note

      a≤30 specifies the number of repeated tests. You can modify this value as needed. grep '10.170.XX.XX' is used to filter the displayed IP information. Modify the IP address based on the internal IP address of the backend ECS instance.

   4. Observe the IP addresses returned by the test. If it is always the same private IP address of a backend ECS instance, session persistence is effective. Otherwise, there may be an issue with session persistence.

Configuring backend cookies for Rewrite Cookie mode

If you select the Rewrite Cookie mode for Layer 7 session persistence in CLB, you must configure a cookie with the specified name on the backend servers. The cookie name must match the cookie name configured in the CLB console. The following sections provide configuration examples for common web servers:

server {
    listen 8080;
    server_name wqwq.example.com;
    location / {
        add_header Set-Cookie name=xxxx;
        root   html;
        index  index.html index.htm;
    }
}

Style loading issues with HTTPS listeners

Symptom:

You create an HTTP and an HTTPS listener, and both listeners use the same backend servers. When you access the website through the port of the HTTP listener, the website displays correctly. However, when you access it through the HTTPS listener, the website's layout is distorted.

Cause:

By default, CLB does not block the loading and transmission of JavaScript files. Possible causes include:

• The certificate is not compatible with the browser's security level.

• The certificate is from an unauthorized third-party issuer. You need to contact the certificate issuer to check for issues.

Solution:

1. When you open the website, load scripts as prompted by the browser.

2. Add the corresponding certificate to the client.

Backend certificate for HTTP-to-HTTPS redirection

No. You only need to configure the relevant certificate in the HTTPS listener of the CLB instance. For more information, see Configure an SSL certificate.

Browser shows outdated certificate after update

This situation usually occurs because CLB is connected to WAF 2.0 in transparent proxy mode, and the certificate on the WAF side has not been updated. WAF synchronizes certificates from CLB periodically. To force an immediate synchronization, you can disable and then re-enable traffic redirection on the WAF side. This action forces a refresh of the certificate status. Note that this operation causes a brief service interruption of 1 to 2 seconds.

Backend HTTP protocol version for listeners

• When the client request uses HTTP/1.1 or HTTP/2, the Layer 7 listener uses HTTP/1.1 to access the backend server.

• When the client request uses a protocol other than HTTP/1.1 and HTTP/2, the Layer 7 listener uses HTTP/1.0 to access the backend server.

Obtaining client protocol version on backend

Yes.

URL-based rate limiting support

CLB does not support URL-based rate limiting. It only supports bandwidth limiting at the listener level.

ALB supports URL-based rate limiting. You can configure listener forwarding rules to set a QPS limit for a specific path. This requires using the "Forward To" action. Refer to the following figure:

The Transfer-Encoding: chunked header

Transfer-Encoding: chunked is a standard field in the HTTP protocol that indicates the message body is transferred in chunks. Layer-7 CLB is implemented based on the Tengine reverse proxy and uses chunked transfer encoding when forwarding requests to backend servers. Therefore, backend servers will see this field in the request header. This is normal behavior for a reverse proxy and does not affect your services. Layer-4 CLB only forwards traffic, so this field does not appear.

Backend response headers removed by L7 CLB

To implement session persistence, CLB removes fields such as Date, Server, X-Pad, and X-Accel-Redirect from the response header. If you need to retain these fields, you can add a prefix to your custom response headers, such as xl-server, or use a Layer 4 TCP listener instead.

Using WAF protection with CLB

Impact of disabling the public NIC

If an ECS instance has a public IP address, disabling its public network interface controller (NIC) disrupts the load balancing service.

This is because when a public NIC is present, the default route goes through the public network. Disabling it prevents response packets from being sent, thus affecting the load balancing service. We recommend not disabling the public NIC. If you must disable it, you need to change the default route to use the private network to avoid service disruption. However, you must first consider whether your business depends on the public network, such as for accessing RDS instances.

Support for requests with a TOA field

No. A TCP Option Address (TOA) field in a client request conflicts with the TOA field used internally by the load balancer, which prevents you from obtaining the client's real IP address.