Troubleshoot NLB health check failures - Server Load Balancer

A Network Load Balancer (NLB) health check verifies that backend servers are running correctly. A failed health check usually indicates a problem with a backend server. However, incorrect health check settings or backend server configurations can also cause failures. This topic shows you how to troubleshoot these failures.

Symptoms

The Health Check Status for a listener of an NLB instance is Unhealthy.

Causes

If health checks fail after you configure them for the first time, first check the health check configuration. The failure may be caused by one of the following reasons:

Incorrect health check parameters
Listener port issues

If health checks fail after they have been running successfully, first check the backend servers. The failure may be caused by one of the following reasons:

Security software issues
Incorrect route configuration
High backend server load

Solutions

Health checks fail after initial configuration

Cause 1: Incorrect health check parameters

Log on to the NLB console.
In the top navigation bar, select the region in which the NLB instance is deployed.
In the left-side navigation pane, choose NLB > Server Groups.
On the Server Groups page, find the server group for your NLB instance and click Modify Health Check Settings.
In the Modify Health Check Settings dialog box, verify that the health check parameters are correct. We recommend using the default health check parameters.

Cause 2: Listener port issues

Check the health check port

Log on to the NLB console.
In the top navigation bar, select the region in which the NLB instance is deployed.
In the left-side navigation pane, choose NLB > Server Groups.
On the Server Groups page, find the server group for your NLB instance and click the server group ID.
On the server group details page, click the Backend Servers tab to view and record the backend server ports.
On the server group details page, click the Details tab. In the Health Check section, click Modify Health Check Settings. In the Modify Health Check Settings dialog box, view and record the health check parameters.

TCP listener

Log on to the backend server and run the following command to test the connection to the health check port:
For information about how to log on to a backend server, see Remote connection guide for ECS instances.
```
telnet [$IP] [$Port]
```
Note
- [$IP] is the private IP address of the backend server.
- [$Port] is the probe port for the server group's health check. If a specific probe port is not configured, this defaults to the backend server port.
Check the command output.
A response like "telnet: connect to address [$IP]: Connection refused", as shown in the following figure, indicates that the connection was refused and the health check port is not responding.
A response like "Connected to [$IP]", as shown in the following figure, indicates that the health check port on the backend server is listening correctly.

UDP listener

Log on to the backend server and run the following command to check the status of the health check port:
For information about how to log on to a backend server, see Remote connection guide for ECS instances.
```
netstat -anu | grep [$IP]:[$Port]
```
Note
- [$IP] is the private IP address of the backend server.
- [$Port] is the probe port for the server group's health check. If a specific probe port is not configured, this defaults to the backend server port.
Check the command output.
If no entry for [$IP]:[$Port] is returned, as shown in the following figure, the health check port on the backend server is not responding.
If an entry for [$IP]:[$Port] is returned, as shown in the following figure, the health check port on the backend server is listening correctly.

Verify that the application is running on the backend server and that its port matches the health check port. This section uses an Nginx service as an example.

Log on to the unhealthy backend server and run the following command to check the Nginx service status:
```
systemctl status nginx
```
A response like the one below indicates that the service is not running.
Run the following command to start the Nginx service:
```
systemctl start nginx
```
Then, run the following command to check the Nginx service status again:
```
systemctl status nginx
```
A response like the one below indicates that the service is running.
Log on to the Network Load Balancer (NLB) console and perform the following steps:
1. In the left-side navigation pane, choose NLB > Server Groups.
2. On the Server Groups page, find the target server group and click Modify Health Check Settings in the Actions column.
3. In the Modify Health Check Settings dialog box, check the health check port [$Port].
4. Check if the health check is normal. If it is not, run the following command to check the Nginx service listener port.
```
netstat -tanp |grep nginx
```
If the output is like the following, the listening port does not match the [$Port] value.
Edit the /etc/nginx/nginx.conf file, find and modify the listen value to [$Port], and then save and exit.
Note
If you cannot change the listen value, you can modify the health check port instead. For more information, see NLB Listeners.
Run the following command to restart the Nginx service. Then, wait for a moment and confirm that the health check is normal.
```
systemctl restart nginx
```

Health checks fail after running successfully

Cause 1: Security software issues

The NLB instance communicates with backend servers using its VPC CIDR block. Ensure that security software on your backend servers, such as iptables, does not block traffic from this CIDR block. If this traffic is blocked, health checks will fail. This section uses iptables as an example to check for blocked IP address ranges.

Log on to the Network Load Balancer (NLB) console and view the local IP address the NLB instance uses to communicate with backend servers.
Log on to the unhealthy backend server instance and run the following command to view all rules in the filter table:
```
iptables -nL
```
A response like the one below indicates the backend server is denying requests from the NLB instance's local IP address.
Run the following command to delete the blocking rule:
```
iptables -t filter -D INPUT -s 192.168.20.75 -j DROP  # This IP address is for demonstration only.
```
Note
Replace the IP address in the command with the local IP address used by your NLB instance.
Run the following command to confirm that requests from the NLB VPC CIDR block are no longer blocked:
```
iptables -nL
```
Confirm that the Health Check Status for the NLB listener changes to Healthy.

Cause 2: Incorrect route configuration

An incorrect route on the backend server for the NLB instance's VPC CIDR block can prevent the NLB instance from receiving health check responses, causing failures. This section uses the Linux route command as an example to check the route configuration.

Log on to the Network Load Balancer (NLB) console and view the local IP address the NLB instance uses to communicate with backend servers.
Log on to the unhealthy backend server and run the following command to check the current route configuration:
```
route -n
```
The route configuration is incorrect if a route entry exists where the Destination is the NLB instance's local IP address, the Genmask is 255.255.255.255, and the Gateway is not the default gateway for the network interface. The default gateway is the IP address listed in the Gateway column when the Destination is 0.0.0.0.
Run the following command to delete the incorrect route:
```
ip route del blackhole 192.168.20.75 # This IP address is for demonstration only.
```
Note
Replace the IP address in the command with the local IP address used by your NLB instance.
Confirm that the Health Check Status for the NLB listener changes to Healthy.

Cause 3: High backend server load

See Troubleshoot and resolve high load issues on Linux instances to determine if high server load is causing the failure.

References

The NLB instance diagnostics feature can also help you troubleshoot health check failures.