Introduction to Common Kernel Network Parameters and Handling of Common Problems in Linux System

This article introduces common Linux system kernel network parameters and solutions to common problems.

Precautions

Before modifying kernel parameters, you need to pay attention to the following points:

• Proceed from the actual needs and try to have the support of relevant data. It is not recommended that you adjust the kernel parameters at will.

• Understand the specific functions of the parameters, and note that the kernel parameters may be different in different types or versions of the environment.

• Back up important data in the ECS instance. For details, see Creating a Cloud Disk Snapshot.

View and modify kernel parameters

Both /proc/sys/ and /etc/sysctl.conf support modifying kernel parameters while the instance is running. The differences are as follows:

• /proc/sys/ is a virtual file system that provides access to kernel parameters. The net under this directory stores all network kernel parameters enabled in the current system, which can be modified while the system is running, but restart the instance After that, it will become invalid, and it is generally used to temporarily verify the effect of the modification.

• /etc/sysctl.conf is a configuration file. You can modify the default values of kernel parameters by modifying the /etc/sysctl.conf file, which will not become invalid after the instance is restarted.

The files in the /proc/sys/ directory are related to the complete names of the parameters in the /etc/sysctl.conf configuration file, such as the net.ipv4.tcp_tw_recycle parameter, and the corresponding file is the /proc/sys/net/ipv4/tcp_tw_recycle file. The content is the parameter value.

Note The tcp_tw_recycle configuration has been removed from Linux kernel version 4.12, that is, the net.ipv4.tcp_tw_recycle configuration content in sysctl.conf has been removed. The net.ipv4.tcp_tw_recycle parameter can only be used when your system kernel version is lower than 4.12.

View and modify kernel parameters through the /proc/sys/ directory View and modify kernel parameters through the /etc/sysctl.conf file

1. Log in to the ECS instance of the Linux system.

2. Use the cat command to view the contents of the corresponding file.

For example, run the following command to view the value of net.ipv4.tcp_tw_recycle.

cat /proc/sys/net/ipv4/tcp_tw_recycle

3. Use the echo command to modify the file corresponding to the kernel parameter.

For example, run the following command to change the value of net.ipv4.tcp_tw_recycle to 0.

echo "0" > /proc/sys/net/ipv4/tcp_tw_recycle

Common problems and solutions of network-related kernel parameters

1. What should I do if I cannot remotely connect to the ECS instance of the Linux system and see the error message "nf_conntrack: table full, dropping packet" in the /var/log/message log?

Problem Phenomenon

Unable to remotely connect to the ECS instance, ping packet loss or ping failure occurs when pinging the target instance, and the following error messages frequently appear in the /var/log/message system log.

Feb 6 16:05:07 i-*** kernel: nf_conntrack: table full, dropping packet.
Feb 6 16:05:07 i-*** kernel: nf_conntrack: table full, dropping packet.
Feb 6 16:05:07 i-*** kernel: nf_conntrack: table full, dropping packet.
Feb 6 16:05:07 i-*** kernel: nf_conntrack: table full, dropping packet.

Problem Causes

ip_conntrack is a module for tracking connection entries of NAT in the Linux system. The ip_conntrack module will use a hash table to record the established connection records of the TCP protocol. When the hash table is full, the data packets of the new connection will be discarded, and the nf_conntrack: table full, dropping packet error will appear.

The Linux system will open up a space for maintaining each TCP connection. The size of this space is related to the nf_conntrack_buckets and nf_conntrack_max parameters. The default value of the latter is 4 times that of the former, so it is generally recommended to increase the value of the nf_conntrack_max parameter.

Note Maintaining system connections consumes memory. It is recommended that you increase the value of the nf_conntrack_max parameter when the system is idle and the memory is sufficient.

Solution

1.1 Use VNC to remotely connect to the instance.

1.2 Modify the nf_conntrack_max parameter value.

a. Run the following command to open the /etc/sysctl.conf file.

vi /etc/sysctl.conf

b. Press i key to enter edit mode.

c. Modify the nf_conntrack_max parameter value.

For example, change the parameter of the maximum value of hash table entries to 655350.

net.netfilter.nf_conntrack_max = 655350

d. Press the Esc key, enter :wq, save and exit editing.

1.3 Modify the timeout parameter nf_conntrack_tcp_timeout_established value.

For example, modify the timeout parameter value to 1200, and the default timeout period is 432000 seconds.

net.netfilter.nf_conntrack_tcp_timeout_established = 1200

1.4 Execute the following command to make the configuration take effect.

sysctl -p

2. Why does the "Time wait bucket table overflow" error message appear in the /var/log/messages log?

Problem Phenomenon

In the ECS instance of the Linux system, the "kernel: TCP: time wait bucket table overflow" error message frequently appears in the /var/log/messages log.

Feb 18 12:28:38 i-*** kernel: TCP: time wait bucket table overflow
Feb 18 12:28:44 i-*** kernel: printk: 227 messages suppressed.
Feb 18 12:28:44 i-*** kernel: TCP: time wait bucket table overflow
Feb 18 12:28:52 i-*** kernel: printk: 121 messages suppressed.
Feb 18 12:28:52 i-*** kernel: TCP: time wait bucket table overflow
Feb 18 12:28:53 i-*** kernel: printk: 351 messages suppressed.
Feb 18 12:28:53 i-*** kernel: TCP: time wait bucket table overflow
Feb 18 12:28:59 i-*** kernel: printk: 319 messages suppressed.

Problem Causes

The net.ipv4.tcp_max_tw_buckets parameter is used to adjust the number of managed TIME_WAIT states in the kernel. When the sum of the number of connections in the TIME_WAIT state in the ECS instance plus the number of connections that need to be converted to the TIME_WAIT state exceeds the net.ipv4.tcp_max_tw_buckets parameter value, The "kernel: TCP: time wait bucket table overflow" error message will appear in the /var/log/messages log. At this time, the system kernel will close some TCP connections that exceed the parameter value.

Solution

You can appropriately increase the net.ipv4.tcp_max_tw_buckets parameter value according to the actual situation. At the same time, it is recommended that you improve the TCP connection from the business level. This article describes how to modify the net.ipv4.tcp_max_tw_buckets parameter value.

2.1 Use VNC to remotely connect to the instance.

2.2 Execute the following command to view the number of TCP connections.

netstat -anp |grep tcp |wc -l

The display is as follows, indicating that the number of connections in the TIME_WAIT state is 6300.

6300 TIME_WAIT
40 LISTEN
20 ESTABLISHED
20 CONNECTED

2.3 Execute the following command to view the net.ipv4.tcp_max_tw_buckets parameter value.

cat /etc/sysctl.conf | grep net.ipv4.tcp_max_tw_buckets
The display is as follows, indicating that the net.ipv4.tcp_max_tw_buckets parameter value is 20000.

2.4 Modify the net.ipv4.tcp_max_tw_buckets parameter value.

a. Run the following command to open the /etc/sysctl.conf file.

vi /etc/sysctl.conf

b. Press i key to enter edit mode.

c. Modify the net.ipv4.tcp_max_tw_buckets parameter value.

For example, change the net.ipv4.tcp_max_tw_buckets parameter value to 65535.

net.ipv4.tcp_max_tw_buckets = 65535

d. Press the Esc key, enter :wq, save and exit editing.

2.5 Execute the following command to make the configuration take effect.

sysctl -p