Troubleshoot NLB health check failures - Server Load Balancer

Symptoms

The Health Check Status for a listener of an NLB instance is Unhealthy.

Causes

If a health check fails after its initial configuration, first check the health check configuration. The failure may be caused by one of the following reasons:

Incorrect health check parameters
Listener port issues

If a health check fails after running successfully, first check the backend servers. The failure may be caused by one of the following reasons:

Security software issues
Incorrect route configuration
High backend server load
Connection limit reached on backend server

Solutions

Health checks fail after initial configuration

Cause 1: Incorrect health check parameters

Log on to the NLB console.
In the top navigation bar, select the region in which the NLB instance is deployed.
In the left-side navigation pane, choose NLB > Server Groups.
On the Server Groups page, find the server group associated with the target NLB instance, and click Modify Health Check Settings.
In the Modify Health Check Settings dialog box, verify that the health check parameters are correct. Using the default settings is recommended.

Cause 2: Listener port issues

1. Check the health check port.

Log on to the NLB console.
In the top navigation bar, select the region in which the NLB instance is deployed.
In the left-side navigation pane, choose NLB > Server Groups.
On the Server Groups page, find the server group that is attached to the target NLB instance, and then click the server group ID.
On the server group details page, click the Backend Servers tab to view and record the backend server ports.
On the server group details page, click the Details tab. In the Health Check section, click Modify Health Check Settings. In the Modify Health Check Settings dialog box, view and record the health check parameters.

TCP listener

Log on to the backend server and run the following command to test the connection to the health check port.
For information about how to log on to a backend server, see Connect to an ECS instance.
```
telnet [$IP] [$Port]
```
Note
- [$IP] is the private IP address of the backend server.
- [$Port] is the probe port for the server group's health check. By default, this is the backend server's port. If a specific health check port is configured, that port is used instead.
Check the command output.

A response similar to "telnet: connect to address [$IP]: Connection refused" indicates the connection was refused and the health check port is unhealthy.
```
Trying xxx.xxx.xxx.xxx...
telnet: connect to address xxx.xxx.xxx.xxx: Connection refused
```
A response similar to "Connected to [$IP]" indicates the health check port is listening correctly.
```
Trying 1xxx...
Connected to 1xxx.
```

UDP listener

Log on to the backend server and run the following command to check the status of the health check port.
For information about how to log on to a backend server, see Connect to an ECS instance.
```
netstat -anu | grep [$IP]:[$Port]
```
Note
- [$IP] is the private IP address of the backend server.
- [$Port] is the probe port for the server group's health check. By default, this is the backend server's port. If a specific health check port is configured, that port is used instead.
Check the command output.

If no entry for [$IP]:[$Port] is returned, the health check port is unhealthy.
```
[rxxxxx xjZ ~]# netstat -anu | grep 192.168    :53
[xxxxx pxjZ ~]#
```
If an entry for [$IP]:[$Port] is returned, the health check port is listening correctly.
```
[rxxx ~]# netstat -anu | grep 192.168.xxx.xxx:53
udp    0    0 192.168.xxx.xxx:53       0.0.0.0:*
```

2. Verify that the application is running on the backend server and uses the correct health check port. This section uses an Nginx service as an example.

Log on to the unhealthy backend server and run the following command to check the status of the Nginx service.
```
systemctl status nginx
```

A response similar to the following indicates that the service is not running.

● nginx.service - The nginx HTTP and reverse proxy server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

Run the following command to start the Nginx service.
```
systemctl start nginx
```

Run the following command again to check the Nginx service status.

systemctl status nginx

A response similar to the following indicates that the service is running.

● nginx.service - The nginx HTTP and reverse proxy server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon xxx CST; 2h 20min ago

Log on to the Network Load Balancer (NLB) console and perform the following steps.
1. In the left-side navigation pane, choose NLB > Server Groups.
2. On the Server Groups page, find the target server group and click Modify Health Check Settings in the Actions column.
3. In the Modify Health Check Settings dialog box, check the health check port [$Port]. Confirm that Health Check is enabled, the Health Check Protocol is TCP, and the Health Check Port is 80.
4. Check the health check status. If it is still unhealthy, run the following command to check the listening port of the Nginx service.
```
netstat -tanp |grep nginx
```

If the output is similar to the following, the listening port does not match the [$Port] value.

tcp        0      0 0.0.0.0:81              0.0.0.0:*               LISTEN      22855/nginx: master
tcp6       0      0 :::81                   :::*                    LISTEN      22855/nginx: master

Edit the /etc/nginx/nginx.conf file. Find and modify the listen value to [$Port], and then save and exit.

Note

If you cannot change the listen value due to your use case, you can modify the health check port for the corresponding protocol. For more information, see NLB listeners.

server {
    listen       80;
    listen       [::]:80;
    server_name  _;
    root         /usr/share/nginx/html;
    # Load configuration files for the default server block.
    include /etc/nginx/default.d/*.conf;
}

Run the following command to restart the Nginx service. After a moment, confirm that the health check is normal.
```
systemctl restart nginx
```

Health checks fail after running successfully

Cause 1: Security software issues

The NLB instance uses a VPC CIDR block to communicate with backend servers. Security software on your backend servers, such as iptables or any other third-party security policy software, must not block traffic from this CIDR block. Blocked traffic causes health check failures and prevents NLB from working correctly. This section uses iptables as an example to check for blocked private IP address ranges.

Log on to the Network Load Balancer (NLB) console and view the local IP address that the NLB instance uses to communicate with backend servers.

In the instance list, find the target instance and view the IP address in the Local IP column, for example, 192.168.20.75.

Note
An NLB instance may have multiple Local IP addresses (across multiple zones, and possibly more than one within a single zone). In the console, view and allow all Local IP addresses. Any access control based on source IP (such as iptables, security groups, or RDS whitelists) must allow all of them. If any Local IP address is blocked, health check failures or intermittent connection timeouts may occur.

Log on to the unhealthy backend server instance and run the following command to view all rules in the filter table.

iptables -nL

A response similar to the following indicates that the backend server is denying requests from the NLB instance's local IP address.

# iptables  -nL
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
DROP       all  --  192.168.20.75        0.0.0.0/0
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Run the following command to delete this rule.
```
iptables -t filter -D INPUT -s 192.168.20.75 -j DROP  # The IP address is for demonstration purposes only.
```
Note
Replace the IP address in the command with the private IP address that is used by your NLB instance.
Run the following command to confirm that requests from the NLB private IP address range are no longer blocked.
```
iptables -nL
```
Confirm that the Health Check Status for the NLB listener changes to Healthy.

Cause 2: Incorrect route configuration

An incorrect route on the backend server for the VPC CIDR block used by the NLB instance can prevent the NLB instance from receiving health check probes, causing failures. This section uses the Linux route command as an example to check the route configuration.

Log on to the Network Load Balancer (NLB) console and view the local IP address that the NLB instance uses to communicate with backend servers.
Log on to the unhealthy backend server and run the following command to check the current route configuration.
```
route -n
```
The route configuration is incorrect if a route entry exists where the Destination is the NLB instance's local IP address, the Genmask is 255.255.255.255, and the Gateway is not the default gateway for the network interface. The default gateway is the IP address listed in the Gateway column when the Destination is 0.0.0.0.
```
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         xxx             0.0.0.0         UG    0      0        0 eth0
xxx             0.0.0           255.255.0.0     U     1002   0        0 xxx
192.168.20.75   0.0.0.0         255.255.255.255 UH    0      0        0 *
```
Run the following command to delete the incorrect route.
```
ip route del blackhole 192.168.20.75 # The IP address is for demonstration purposes only.
```
Note
Replace the IP address in the command with the private IP address that is used by your NLB instance.
Confirm that the Health Check Status for the NLB listener changes to Healthy.

Cause 3: High backend server load

See Troubleshoot and resolve high load issues on Linux instances to determine if high server load is causing the failure.

Cause 4: Connection limit reached on backend server

When a backend server (such as an ACS Pod) reaches its TCP connection limit, it cannot accept new connections. If NLB is configured with TCP health checks, the health check probe fails because the backend cannot accept the new connection, and NLB automatically marks the backend as Unhealthy.

An Unhealthy backend does not receive new connections. If connection draining is not enabled, existing long-lived connections are not affected and continue to function normally.
This behavior triggers Unhealthy alerts. Configure connection count threshold alerts in Cloud Monitor to receive advance warning before the connection limit is reached.
When the backend recovers and can accept new connections again, health checks automatically pass and NLB re-marks the backend as Healthy, resuming new connection forwarding.

Note

NLB does not have a built-in feature to limit connection forwarding based on connection count. Using health checks for this purpose is an indirect mechanism. This approach applies when a backend's connection limit prevents the port from accepting new connections.

References

You can also use the NLB instance diagnostics feature to troubleshoot NLB health check failures. For more information, see NLB instance diagnostics.