Server Load Balancer:Troubleshoot ALB health check failures

Problem description

The Health Check Status for a listener on an ALB instance is Unhealthy.

Causes

Solutions

Health checks fail during initial configuration

Cause 1: Incorrect health check parameters

1. Log on to the ALB console.

2. In the top menu bar, select the region where the ALB instance is located.

3. In the left-side navigation pane, choose ALB > Server Groups.

4. On the Server Groups page, find the server group attached to the target ALB instance, and then click the server group ID.

5. On the server group details page, in the Health Check section, click Modify Health Check.

6. In the Modify Health Check dialog box, verify the health check parameters. We recommend using the default values.

   Health checks for ALB

Cause 2: Health check port issues

1. Log on to the ALB console.

2. In the top menu bar, select the region where the ALB instance is located.

3. In the left-side navigation pane, choose ALB > Server Groups.

4. On the Server Groups page, find the server group attached to the target ALB instance, and then click the server group ID.

5. On the server group details page, click the Backend Servers tab to view and record the backend server port.

6. On the server group details page, click the Details tab, and in the Health Check section, click Modify Health Check. In the Modify Health Check dialog box, view and record the health check parameters.

7. Log on to the backend server and use the nc or curl command to probe the server.

   Connect to an ECS instance

   # nc command:
   echo -e "[$Method] [$PATH] [$VERSION]\r\nHost: [$Domain]\r\n\r\n" | nc -t [$IP] [$Port] # Format
   echo -e "HEAD /index.html HTTP/1.0\r\nHost: www.example.org\r\n\r\n" | nc -t 127.0.0.1 80 # Example
   # curl command:
   curl -X [$Method] -H "Host: [$Domain]" -I http://[$IP]:[$Port][$PATH]  # Format
   curl -X HEAD --http1.0 -H "Host: www.example.org" -I http://127.0.0.1:80/index.html # Example

   Note

   • [$Method] is the health check method configured for the server group.

   • [$PATH] is the health check path configured for the server group.

   • [$VERSION] is the HTTP version used for the health check, for example, HTTP/1.0.

   • [$Domain] is the domain name configured for the health check. If this value is -----, the private IP address of each backend ECS instance is used by default. In this case, you can use [$IP] instead.

   • [$IP] is the private IP address of the backend ECS instance.

   • [$Port] is the health check port configured for the server group. If no health check port is manually configured, the backend ECS instance port is used by default. If a health check port is configured, the configured health check port is used.

8. Check whether the returned status code indicates a normal response for your service.

   • If the status code is normal but not in your configured healthy status codes, add it to the health check configuration.

   • If the status code indicates an error, troubleshoot using the following table.

     Status code	Description	Troubleshooting
     400	Invalid HTTP request format from the client.	Possible causes: (1) HTTP header format error, for example, empty content-length or an HTTP request sent to an HTTPS port — check the HTTP request format. (2) If specific business logic returns 4xx codes and causes health check failures, select 4xx under Health Check Response Codes in the health check configuration to treat 4xx status codes as normal responses. This prevents backend nodes from being removed by health checks due to specific business return codes.
     403	Access is forbidden. ALB itself does not return 403 status codes; a 403 response typically originates from the backend web service configuration.	Check whether the backend service (for example, Nginx or Tomcat) has access control rules that block requests from specific source IP addresses or request patterns. If the backend blocks the source of health check requests, health checks return 403 continuously. Adjust the access control configuration of the backend service to allow health check requests.
     404	The target resource is not found. This is usually caused by a mismatch between the health check path and the paths actually available on the backend server.	Troubleshoot as follows: Check the path prefix exposed by the backend gateway (for example, Nginx), such as /api/v1, and ensure the configured Health Check Path matches a path that is actually accessible on the backend. If direct access to the backend server IP returns a normal response but access via the domain name returns 404, the backend service is usually performing virtual host matching based on the domain name (Host header). Verify that the Health Check Domain configured for the health check matches the domain configuration of the backend service.
     405	The health check request method is not supported.	Check whether the backend service supports the configured health check method.
     500	An internal server error occurred and the request cannot be completed.	Check the business logic of the backend service.
     503	The server is temporarily unavailable.	Check the business logic of the backend service or determine whether the server load is too high.

Health check failures on an existing configuration

Cause 1: Security software interference

ALB uses IP addresses from a reserved CIDR block to communicate with backend ECS instances. If this CIDR block is blocked, health checks fail and ALB cannot forward requests. The following example uses iptables to check for blocks on 100.64.0.0/10.

1. Log on to the affected backend ECS instance and run the following command to list filter table rules.

   iptables -nL

   Output similar to the following indicates the backend ECS instance is blocking requests from the ALB CIDR block.

   [root@xxx Z ~]# iptables -nL
   Chain INPUT (policy ACCEPT)
   target     prot opt source               destination
   DROP       all  --  100.64.0.0/10        0.0.0.0/0
   Chain FORWARD (policy ACCEPT)
   target     prot opt source               destination
   Chain OUTPUT (policy ACCEPT)
   target     prot opt source               destination
   Chain L (0 references)
   target     prot opt source               destination

2. Run the following command to delete this rule:

   iptables -t filter -D INPUT -s 100.64.0.0/10 -j DROP

3. Confirm the ALB CIDR block is no longer blocked:

   iptables -nL

In addition to iptables rules, check the following security configurations to ensure that ALB health check traffic can reach the backend service:

1. Check the port listening status of the backend server: Confirm that the backend server is listening on the backend port configured in the server group (for example, port 80). Run the following command to check the port listening status:

   netstat -tlnp | grep <port>
   # Or use the ss command:
   ss -tlnp | grep <port>

   If the port is not in the LISTEN state, check whether the backend service has started properly.

   Note

   For a backend service deployed in Docker or another container, a reachable host port does not guarantee that the service inside the container is correctly listening on the health check port. If the host port is reachable but the health check fails, enter the container and confirm that the service is listening on the health check port (for example, port 80) and that the listening address is 0.0.0.0 (not just 127.0.0.1). Also check whether a security policy inside the container restricts ALB health check IP addresses.

2. Check the inbound rules of the backend ECS security group: Ensure that the inbound rules of the security group allow access from the following source. Otherwise, ALB health check probes may be blocked by the security group:

   • Authorization object (source address): For an upgraded ALB instance, the vSwitch CIDR block where its Local IP resides; for a non-upgraded ALB instance, the 100.64.0.0/10 CIDR block.

   • Protocol: TCP

   • Port range: The backend service port configured in the server group.

Cause 2: Incorrect route configuration

An incorrect route for the 100.64.0.0/10 CIDR block on a backend ECS instance can prevent health check responses from reaching the ALB instance. The following example uses the Linux route command to verify the configuration.

1. Log on to the backend ECS instance and run the following command to check the route configuration:

   route -n

   The route is incorrect if an entry with Destination 100.64.0.0 and Genmask 255.192.0.0 has a Gateway other than the NIC default gateway (the Gateway value where Destination is 0.0.0.0).

   [root@test-server2 ~]# route -n
   Kernel IP routing table
   Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
   0.0.0.0         <Default gateway of eth0>  0.0.0.0         UG    100    0        0 eth0
   xxx             xxx             xxx             xxx   xxx    xxx      xxx xxx
   100.64.0.0      0.0.0.0         255.192.0.0     U     0      0        0 eth0

2. Run the following command to delete the incorrect route for the 100.64.0.0/10 CIDR block:

   route del -net 100.64.0.0/10

Cause 3: High backend server load

When a backend ECS instance runs low on resources, it may fail to respond to health check probes within the configured timeout, causing health check failures. Insufficient resources can result from overall system overload or from a specific issue, such as a connection leak, that gradually exhausts system resources. Troubleshoot the following two sub-scenarios.

Sub-scenario 1: System resource overload

When CPU, memory, or disk I/O is saturated, the server cannot respond to health check probes in time. Troubleshoot and handle high load on Linux instances.

Sub-scenario 2: Connection leak exhausting resources (CLOSE_WAIT buildup)

If backend ECS instances accumulate a large number of TCP CLOSE_WAIT connections, the cause is typically that the backend application (for example, a Java service managed by jsvc.exec) does not actively call close() to release connections after handling requests. This leaves connections in the CLOSE_WAIT state and causes them to accumulate over time. Excessive CLOSE_WAIT connections consume system resources and can affect health check responses.

Troubleshooting steps:

1. Log on to the backend ECS instance and run the following command to check the current number of CLOSE_WAIT connections:

   netstat -an | grep CLOSE_WAIT | wc -l

2. If the number of CLOSE_WAIT connections keeps growing or is abnormally high, check the backend application code or service configuration to ensure that the application correctly closes connections after processing requests (for example, by calling close() or shutdown()). For frameworks that use a connection pool, check the pool's idle timeout and reclamation settings.

FAQ

Why do backend servers receive health check requests at a high frequency?

ALB uses a multi-zone distributed cluster architecture. All forwarding nodes in the cluster independently send health check probes to backend servers. As a result, the actual request frequency received by a backend server equals the per-node frequency multiplied by the number of nodes. For example, if the health check interval is set to 5 seconds and the cluster contains 10 forwarding nodes, the backend server receives about 10 health check requests every 5 seconds, or nearly 2 requests per second. This is expected behavior by design and does not indicate an issue.

If the health check volume affects backend performance, reduce the frequency using either of the following methods:

• Adjust health check parameters: In the health check configuration, increase the Health Check Interval (range: 1–50 seconds) or raise the Healthy Threshold. Note that a longer interval delays fault detection.

• Switch to a Layer 4 health check protocol: A Layer 4 Health Check performs only a TCP handshake without sending an HTTP request, significantly reducing traffic on backend servers.