Server Load Balancer:Troubleshoot ALB health check issues

Incorrect health check parameters

Symptom: All backend servers display Unhealthy immediately after health checks are enabled.

Cause: One or more health check parameters (such as the protocol, path, expected status codes, or timeout) do not match the backend server configuration.

Solution:

1. Log on to the ALB console.

2. In the top navigation bar, select the region where the ALB instance is deployed.

3. In the left-side navigation pane, choose ALB > Server Groups.

4. On the Server Groups page, find the target server group and click the server group ID.

5. On the server group details page, click Modify Health Check in the Health Check section.

6. In the Modify Health Check dialog box, review the parameter settings. Use the default settings as a starting point if the correct values are unclear.

For details about each parameter, see Health checks.

Verify the fix: After saving the changes, wait for several health check cycles. Then check whether the Health Check Status changes to healthy.

Incorrect health check port

Symptom: Backend servers display Unhealthy even though the backend service is running correctly.

Cause: The health check port does not match the port on which the backend service listens, or the backend service does not respond correctly to health check requests.

Solution:

1. Log on to the ALB console.

2. In the top navigation bar, select the region where the ALB instance is deployed.

3. In the left-side navigation pane, choose ALB > Server Groups.

4. On the Server Groups page, find the target server group and click the server group ID.

5. On the server group details page, click the Backend Servers tab and record the backend server port.

6. Click the Details tab. Then, click Modify Health Check in the Health Check section. In the Modify Health Check dialog box, record the health check configurations.

7. Log on to a backend server and manually send a health check request to verify the response. For details about connecting to an Elastic Compute Service (ECS) instance, see Guidelines on instance connection. Use the nc command: Or use the curl command: Replace the placeholders with the values from your health check configuration:

   Placeholder	Description
   [$Method]	The health check method of the server group
   [$PATH]	The health check path of the server group
   [$VERSION]	The HTTP protocol version of the server group, such as HTTP/1.0
   [$Domain]	The health check domain name of the server group. If the domain name is -----, use the private IP address of the ECS instance instead
   [$IP]	The private IP address of the ECS instance
   [$Port]	The health check port of the server group. If no port is specified, use the backend server port

      # Format
      echo -e "[$Method] [$PATH] [$VERSION]\r\nHost: [$Domain]\r\n\r\n" | nc -t [$IP] [$Port]
   
      # Example
      echo -e "HEAD /index.html HTTP/1.0\r\nHost: www.example.org\r\n\r\n" | nc -t 127.0.0.1 80

      # Format
      curl -X [$Method] -H "Host: [$Domain]" -I http://[$IP]:[$Port][$PATH]
   
      # Example
      curl -X HEAD --http1.0 -H "Host: www.example.org" -I http://127.0.0.1:80/index.html

8. Check the returned HTTP status code to determine whether it indicates a healthy state:

   • If the status code is healthy but is not included in the health check settings, update the settings to include that status code.

   • If the status code indicates an unhealthy state, refer to the following table.

Security rules blocking health check traffic

Symptom: Backend servers display Unhealthy even though the backend service is running and responding to requests from other sources.

Cause: Firewall rules (such as iptables), security group rules, or third-party security software on the backend server block requests from ALB health check source IP addresses.

Solution (iptables example):

The following example shows how to identify and remove an iptables rule that blocks the 100.64.0.0/10 CIDR block.

1. Log on to the ECS instance and run the following command to list all rules in the filter table: If the output shows a rule that drops traffic from the ALB health check CIDR block (for example, DROP all -- 100.64.0.0/10 0.0.0.0/0), this rule is blocking health check probes.

      iptables -nL

2. Delete the blocking rule:

      iptables -t filter -D INPUT -s 100.64.0.0/10 -j DROP

3. Verify that the rule has been removed: Confirm that no rules block traffic from the ALB CIDR block.

      iptables -nL

Verify the fix: Wait for several health check cycles. Then check whether the Health Check Status changes to healthy in the ALB console.

Incorrect route configuration

Symptom: Backend servers display Unhealthy on a non-upgraded ALB instance even though the backend service is running and firewall rules are correct.

Cause: The route for 100.64.0.0/10 on the backend ECS instance is not configured with the correct gateway. When the route points to the wrong gateway, health check response packets from the backend server cannot reach the ALB instance.

Solution:

1. Log on to the backend ECS instance and check the route configuration: The route is incorrectly configured if all of the following conditions are true: The default gateway is the Gateway value associated with the route where the Destination value is 0.0.0.0.

   • The Destination value is 100.64.0.0

   • The Genmask value is 255.192.0.0

   • The Gateway value is not set to the default gateway of the corresponding Elastic Network Interface (ENI)

      route -n

2. Delete the incorrect route:

      route del -net 100.64.0.0/10

Verify the fix: Run route -n again to confirm that the incorrect route has been removed. Wait for several health check cycles and check whether the Health Check Status changes to healthy.

Backend server overload

Symptom: Backend servers intermittently display Unhealthy, or health checks fail during traffic spikes.

Cause: The backend ECS instance is overloaded (high CPU usage, memory pressure, or disk I/O saturation). When the server cannot respond to health check probes within the configured timeout, ALB marks it as unhealthy.

Solution:

Check the system load on the backend ECS instance. For detailed instructions, see Query and analysis of system loads on Linux instances.

Verify the fix: After resolving the load issue (by scaling out, optimizing the application, or increasing instance specifications), confirm that the Health Check Status returns to healthy in the ALB console.