This topic describes the cause of and solutions to the issue that applications on an Elastic Compute Service (ECS) instance occasionally drop packets and the "kernel: nf_conntrack: table full, dropping packet" error messages appear in kernel logs. To query kernel logs, run the dmesg command.
Problem description
Packet loss occasionally occurs when you connect to applications on an ECS instance. The peripheral network of the ECS instance runs as expected. However, when you run the dmesg command to query kernel logs, the "kernel: nf_conntrack: table full, dropping packet
" error messages appear. The ECS instance on which the issue occurs meets the following conditions:
Image:
aliyun-2.1903-x64-20G-alibase-20190327.vhd
or laterKernel:
kernel-4.19.24-9.al7
or later
Cause
nf_conntrack is a NAT module that tracks connection entries in the Linux operating system. The nf_conntrack module uses a hash table to record the established TCP connections. When entries the hash table are exhausted, the establishment of new TCP connections causes the module to report "nf_conntrack: table full, dropping packet
" errors. Take note of the following important parameters of the nf_conntrack module:
nf_conntrack_buckets
: the size of the hash table, which can be specified when the module is loaded or can be modified by running thesysctl
command. When the system memory is greater than or equal to 4 GB, the default value is 65536.nf_conntrack_max
: the maximum number of nodes in the hash table, which is the maximum number of connections supported by the nf_conntrack module. When the system memory is greater than or equal to 4 GB, the default value is 262144. For servers that handle a large number of connections, you can increase the value based on your business requirements.nf_conntrack_tcp_timeout_time_wait
: the period for which the TCP connections can remain in the TIME_WAIT state, which is stored in the nf_conntrack module. The default value is 120 seconds.
Solutions
Use one of the following solutions based on your business scenario.
Solution 1: Use the sysctl interface to change parameter values in the nf_conntrack module
Estimate the nf_conntrack_max value required for applications in advance, and change the parameter values in the nf_conntrack module by using the sysctl interface. The following commands are used for reference.
If your business requires a large number of concurrent connections, which are mainly short-lived connections, we recommend that you increase the values of the nf_conntrack_max
and nf_conntrack_buckets
parameters. This helps prevent excessive connections from exhausting the entries in the nf_conntrack hash table. In most cases, the nf_conntrack_max
parameter value is four times the nf_conntrack_buckets
parameter value.
sudo sysctl -w net.netfilter.nf_conntrack_max=1503232
sudo sysctl -w net.netfilter.nf_conntrack_buckets=375808 # This option cannot be modified during runtime if the kernel version is not 4.19.
sudo sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=60
The parameter values in the commands are provided only for reference. Change the values based on your business requirements. Before you modify the parameter values, we recommend that you create snapshots or back up important files in advance to ensure data security.
We recommend that you change the values of the
nf_conntrack_buckets
andnf_conntrack_max
parameters together. If you change only the value of thenf_conntrack_max
parameter, the linked list on the hash table may be long, and query efficiency becomes low. If you change only the value of thenf_conntrack_buckets
parameter, the preceding packet dropping issue persists.
Scenario 2: Use the Iptables utility to filter connections that do not need to be tracked
Run the following commands to add the "-j notrack" action to Iptables rules to filter connections that do not need to be tracked. This method removes the records of the connections that do not need to be tracked from the hash table, which prevents excessive connections from causing "kernel: nf_conntrack: table full, dropping packet" errors. This is because the "-j notrack" action takes effect directly on the corresponding connections.
sudo iptables -t raw -A PREROUTING -p udp -j NOTRACK
sudo iptables -t raw -A PREROUTING -p tcp --dport 22 -j NOTRACK
The preceding commands are provided only for reference. They are run to prevent the nf_conntrack module from tracking UDP connections and the TCP connections over port 22. You can run commands based on your business requirements.