CLB error status codes

Five quick checks

Before troubleshooting specific status codes, check the following five configurations in order, as they are the most frequent causes of CLB failures.

1. Run the CLB instance diagnosis tool. On the CLB instances page, find the target instance and click Start Diagnostics in the Instance Diagnostics column. It checks for common issues with the instance configuration, listeners, and backend services.

2. Verify that the Health Check Status for the listener is Healthy. On the listener page of the instance, check the Health Check Status of the backend server group. If the status is abnormal, see Health Check FAQ.

3. Ensure the backend ECS instance does not block the 100.64.0.0/10 CIDR block. This is an internal CIDR block reserved by Alibaba Cloud for communication between CLB and backend servers and poses no security risk. You do not need to add an allow rule for this CIDR block in the security group for the ECS instance. However, if the iptables rules or third-party security software on the backend ECS instance blocks this CIDR block, CLB cannot access the backend, which can trigger 502, 504, or other errors.

4. Verify that the port configured for the backend server in the server group matches the port that the backend service is actually listening on. The port configured for each backend server in a CLB server group must match the port on which the service process is listening. For example, if you configure port 8080 in the server group but the backend service is listening on port 80, connections will fail. You can run ss -tlnp | grep ':<port> ' or netstat -tlnp | grep ':<port> ' on the backend ECS instance to check the listening port.

5. For HTTPS listeners: Verify that the certificate has not expired and that the domain name bound to the certificate matches the access domain. On the listener details page, check the bound certificate and its expiration date. An expired certificate or a domain mismatch can cause an SSL handshake to fail or return an error status code.

502 (Bad Gateway)

This error occurs when an HTTP or HTTPS listener receives a request from a client but CLB cannot forward it to a backend server or receive a response.

Troubleshooting approach: Check the value of the upstream_status field in the access log and determine the next troubleshooting steps based on the value.

• If upstream_status is 502: The backend service itself returned a 502 error, and CLB passed it through to the client. Investigate your backend Nginx, gateway, or application logs.

• When upstream_status is another value (such as 504, 444, or 500): The status returned by CLB in the access log does not match the upstream_status, which indicates that CLB converted the status code. Investigate why the backend service returned this status code based on the actual value of upstream_status (check the backend Nginx, gateway, or application logs).

• If upstream_status is - or empty: CLB did not receive a response from the backend, which indicates that the request did not reach the backend or the backend closed the connection unexpectedly before responding. Check the following items in order:

  • Abnormal TCP communication between CLB and the backend server. Check whether the backend service is running, the service port is listening correctly, and iptables rules or third-party security software on the backend ECS instance is blocking the 100.64.0.0/10 CIDR block. You can also capture packets to verify that the TCP handshake is successful.

  • The backend server backlog is full. This causes new connection requests to be rejected or dropped. You can run netstat -s | grep -i listen on the backend server to check for a drop count.

  • The backend server did not process the request in time. Check the backend server's logs and review its CPU and memory utilization to identify performance bottlenecks.

  • The packet sent by the client exceeds the MTU of the backend server. This can manifest as successful health checks and normal processing of small packets, but failures for larger packets. We recommend capturing packets on the backend server to analyze whether the packet size is compliant.

  • The backend server's response has an invalid format or invalid HTTP headers. Capture packets on the backend server to analyze whether the response format is valid.

  • All backend servers in the server group are failing health checks. With no available backends, CLB returns a 502 error. Troubleshoot the listener's health check configuration and the status of your backend services. For more information, see How do I troubleshoot health check failures?

  • The listener's health check is normal, but the server group associated with a forwarding rule is abnormal. Health checks apply only to the default server group bound to a listener and do not cover server groups configured in forwarding rules. If a request matches a forwarding rule, you must check the status of the backend servers in the server group associated with that rule.

400 (Bad Request)

The HTTP request has an invalid format.

• The backend server returns a 400 error. The upstream_status is 400, and CLB passes the response through. This issue commonly occurs when CLB uses the HTTP protocol to access a backend server that uses the HTTPS protocol, or when the backend server has special message validation logic. Investigate why the backend service is returning a 400 error.

• The HTTP header format of the request sent from the client is incorrect. For example, the value of the Content-Length header does not match the length of the request body, the request method is not specified in uppercase, or the header size exceeds the 32 KB limit. We recommend that you capture packets on the client to analyze the HTTP request format and compare it with a valid request.

• The client sent an HTTP request to the CLB instance's HTTPS listener port. CLB rejects the request and returns a 400 error. Check whether the client is using the correct protocol.

• The client actively closed the connection before the HTTP request was fully sent. We recommend capturing packets to investigate why the client disconnected.

405 (Method Not Allowed)

The request method is not supported.

• The backend server does not support the request method used by the client. The upstream_status is 405. You can run the curl -X method ip:port command on the backend server to verify, where method is the client request method, ip is the backend IP address, and port is the backend port. Check if the client request method is supported by the backend server.

• CLB does not support the TRACE request method. Use a different method.

408 (Request Timeout)

The request timed out, and CLB actively closed the connection.

• The client transmitted only partial data. The client sent only partial data (such as only the HTTP header) within the request timeout period (default 60 seconds). Capture packets to check for client-side performance bottlenecks or abnormal behavior.

• Poor network link quality from the client to CLB. The TCP Round Trip Time (RTT) is high or packet loss occurs. You can check the request_time and tcpinfo_rtt fields in the access log, or perform network diagnostics on the client.

• Excessive traffic to the CLB instance is triggering bandwidth throttling and packet loss. Use CloudMonitor to check the instance's outbound bandwidth and dropped connection metrics.

414 (URI Too Long)

The length of the client request URI exceeds the limit, and CLB rejected the request.

• The backend server itself returns a 414 error. upstream_status=414. The backend server has a stricter URI length limit than CLB. You can shorten the client URI or increase the backend server's URI length limit.

• The client request URI exceeds the 32 KB limit of CLB. Shorten the URI length. If you must send a large amount of data, use the POST method to place it in the request body.

499 (Client Closed Request)

The client actively closed the connection.

• Poor network link quality between the client and CLB. This is indicated by a high TCP RTT or packet loss. You can check the request_time and tcpinfo_rtt fields in the access log, or perform a packet capture to diagnose the client network.

• Excessive traffic to the CLB instance is triggering bandwidth throttling and packet loss. Use CloudMonitor to check the instance's outbound bandwidth and dropped connection metrics.

• The backend server takes too long to process requests. The processing time exceeds the client's request timeout. (In the access log, upstream_response_time represents the backend processing time). We recommend that you check the backend server for performance bottlenecks in the CPU, memory, or network.

• The request timeout period configured on the client is too short. This causes the client to close the connection before the request is complete. The request_time field in the access log indicates the total request time. We recommend that you adjust the client timeout configuration based on this field.

• The client encountered an unknown issue. This caused it to close the connection prematurely. Investigate the client for behavior that might cause early connection termination.

500 (Internal Server Error)

An internal error occurred on the backend server.

• The backend server itself returns a 500 error. The upstream_status is 500, and CLB passes through the response. Check the error logs of the backend server to troubleshoot the root cause.

• The backend server closed the connection unexpectedly before it sent a full response. Capture packets on the backend server to investigate the cause of the unexpected connection closure.

503 (Service Temporarily Unavailable)

The service is temporarily unavailable, usually because traffic limits are exceeded or the backend service is unavailable.

• The backend server itself returns a 503 error. The upstream_status is 503, and CLB passes the response through. Check the logs of the backend server to troubleshoot the root cause.

• Client request traffic triggers the rate limiting of CLB. You can view the requests per second metric in CloudMonitor. Because CloudMonitor displays data at minute-level granularity, it may not show when the per-second limit is exceeded. You can view the per-second request count in the access log. A value of - for the upstream_status field indicates that the request was not sent to a backend server.

  Solutions:

  • Upgrade the specification of the CLB instance.

  • Use Alibaba Cloud DNS to point a domain name to multiple CLB instances to distribute traffic.

  • For Layer 4 services, consider using NLB to handle more concurrent connections.

  • For Layer 7 services, consider using ALB to achieve higher QPS.

• No backend servers are configured for the CLB listener, or the configured backend servers have a weight of 0.

504 (Gateway Timeout)

The backend server timed out while responding.

• The backend server itself returns a 504 error. The upstream_status is 504, and CLB passes the response through. A common cause is that the backend server times out when it accesses other internal services. Troubleshoot these internal services.

• A timeout occurs when CLB connects to a backend server. The default connection timeout is 5 seconds and cannot be modified. You can check the access log to see if the value of the upstream_connect_time field is close to 5 seconds. We recommend that you use packet capture to investigate why the backend server is responding slowly.

• The backend server response time exceeded the request timeout period of CLB. The default is 60 seconds. You can confirm this by checking the upstream_rt metric in CloudMonitor or the upstream_response_time field in the access log.