Handling of Common Problems in Linux System

1. Why are there a large number of TCP connections in the FIN_WAIT2 state in the ECS instance of the Linux system?

Problem Phenomenon

In the ECS instance of the Linux system, there are a large number of TCP connections in the FIN_WAIT2 state.

Problem Causes

This problem may occur for the following reasons:

• In the HTTP service, the Server will actively close the connection for some reason. For example, when KEEPALIVE times out, the Server that actively closes the connection will enter the FIN_WAIT2 state.

• In the TCP/IP protocol stack, there is a concept of semi-connection, and the FIN_WAIT2 state is not counted as timeout (different from the TIME_WAIT state). If the Client is not closed, the FIN_WAIT2 state will remain until the system restarts, and more and more FIN_WAIT2 states will cause the kernel to crash.

Solution

You can reduce the value of the net.ipv4.tcp_fin_timeout parameter to speed up the closing of TCP connections in the FIN_WAIT2 state by the system.

1.1 Use VNC to remotely connect to the instance.

1.2 Modify the net.ipv4.tcp_fin_timeout parameter value.

a. Run the following command to open the /etc/sysctl.conf file.

vi /etc/sysctl.conf

b. Press i key to enter edit mode.

c. Modify the net.ipv4.tcp_fin_timeout parameter value.

For example, change the net.ipv4.tcp_fin_timeout parameter value to 10.

net.ipv4.tcp_fin_timeout = 10

d. Press the Esc key, enter :wq, save and exit editing.

1.3 Execute the following command to make the configuration take effect.

sysctl -p

2. Why are there a large number of TCP connections in the CLOSE_WAIT state in the ECS instance of the Linux system?

Problem Phenomenon

In the ECS instance of the Linux system, there are a large number of TCP connections in the CLOSE_WAIT state.

Problem Causes

The possible reason for this problem is whether the number of CLOSE_WAIT exceeds the normal range.

When the TCP connection is disconnected, it needs to wave four times. Both ends of the TCP connection can initiate a request to close the connection. If the other end initiates a close connection, but the local does not close the connection, then the connection will be in the CLOSE_WAIT state. Although the connection is already in the semi-connected state, it is unable to communicate with the peer, and the connection needs to be released in time.

Solution

It is recommended that you judge from the business level whether a connection has been closed by the peer in a timely manner, that is, close the connection in time in the program logic and check it.

2.1 Remotely connect to the ECS instance.

2.2 Check and close the TCP connection in the CLOSE_WAIT state in the program.

The corresponding read and write functions in the programming language generally include the TCP connection function to detect the CLOSE_WAIT state. The method of closing connection in Java language and C language is as follows:

• Java language

i. Use the read method to judge I/O. When the read method returns -1, it means that the end has been reached.

ii. Close the connection through the close method.

• C language

Check the return value of read.

i. If equal to 0, the connection can be closed.

ii. If it is less than 0, check the error, if it is not AGAIN, you can also close the connection.

3. Why can't the client access ECS or RDS on the server after NAT is configured?

Problem Phenomenon

After the client is configured with NAT, it cannot access the ECS and RDS of the server, including the ECS instance in the VPC configured with SNAT.

Problem Causes

This problem may be caused by setting the net.ipv4.tcp_tw_recycle and net.ipv4.tcp_timestamps parameters to 1 on the server.

When the server-side kernel parameters net.ipv4.tcp_tw_recycle and net.ipv4.tcp_timestamps are both 1, it means that the server will check the timestamp (Timestamp) in each TCP connection message. If the Timestamp is not incremental, it will not will respond to this message.

Solution

You can choose an appropriate solution based on the server-side cloud product incompatibility.

• When the remote server is ECS, modify the parameters net.ipv4.tcp_tw_recycle and net.ipv4.tcp_timestamps to 0.

• When the remote server is RDS, RDS cannot directly modify the kernel parameters. You need to modify the parameters net.ipv4.tcp_tw_recycle and net.ipv4.tcp_timestamps to 0 on the client.

3.1 Use VNC to remotely connect to the instance.

3.2 Modify the net.ipv4.tcp_tw_recycle and net.ipv4.tcp_timestamps parameter values to 0.

a. Run the following command to open the /etc/sysctl.conf file.

vi /etc/sysctl.conf

b. Press i key to enter edit mode.

c. Change the net.ipv4.tcp_tw_recycle and net.ipv4.tcp_timestamps parameters to 0.

net.ipv4.tcp_tw_recycle=0
net.ipv4.tcp_timestamps=0

d. Press the Esc key, enter :wq, save and exit editing.

3.3 Execute the following command to make the configuration take effect.

sysctl -p