Packet loss occurs occasionally when connecting applications in ECS instances. After troubleshooting, the perimeter network of the ECS instance is normal, but the Kernel Log (dmesg) contains the following error message.
kernel: nf_conntrack: table full, dropping packet
The faulty ECS instance meets the following requirements:
- Image: all
versionsof images and later.
kernel-4.19.24-9.al7and later versions.
nf_conntrack is a connection entry tracking module for NAT within the Linux. The nf_conntrack module uses a hash table to record the TCP protocol
establishment connection. When the hash table is full, the new connection will cause the
failure. The following table describes the important parameters in the nf_conntrack module.
nf_conntrack_buckets: the size of the hash table. You can specify parameters during module loading or modify them by running the
sysctlcommand. When the system memory is greater than or equal to 4GB, its default value is 65536.
nf_conntrack_max: the maximum number of nodes in the hash table, that is, the maximum number of connections supported by the nf_conntrack module. When the system memory is larger than 4 GB, its default value is "262144". For servers handling a large number of connections, the default value is relatively small.
Duration: the TCP connection time in the time_wait status stored by the nf_conntrack module. The default value is 120s.
Alibaba Cloud reminds you that:
- Before you perform operations that may cause risks, such as modifying instance configurations or data, we recommend that you check the disaster recovery and fault tolerance capabilities of the instances to ensure data security.
- You can modify the configurations and data of instances including but not limited to Elastic Compute Service (ECS) and Relational Database Service (RDS) instances. Before the modification, we recommend that you create snapshots or enable RDS log backup.
- If you have authorized or submitted sensitive information such as the logon account and password in the Alibaba Cloud Management Console, we recommend that you modify such information in a timely manner.
Select the method that best suits your business needs from the following two methods based on the on-site conditions:
Solution 1: Use the sysctl interface to adjust the parameter values of the nf_conntrack module.
The business side should confirm the maximum number of nf_conntrack connections that applications may use in advance, and adjust the parameter values of nf_conntrack through the sysctl interface by referring to the following command.
Note: if your service has a high number of concurrent connections, short-lived connections are the main feature of this service. We recommend that you increase the parameter values of
nf_conntrack_bucketsto ensure that the hash table of nf_conntrack does not become full due to excessive connections. We recommend that you set
nf_conntrack_maxto four times the value of
sudo sysctl -w net.netfilter.nf_conntrack_max=1503232 sudo sysctl -w net.netfilter.nf_conntrack_buckets=375808# If you use a non -4.19 kernel, this option may not be able to modify the sudo sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=60 at runtime.
- The value in the command is for reference only. Adjust the value based on the actual business situation on site. Before resizing, we recommend that you create a snapshot or back up important files to ensure data security.
- We recommend that you adjust the
nf_conntrack_maxparameters together. If you only change the value of
nf_conntrack_max, the linked list in a hash table may be too long, causing low query efficiency. This problem cannot be solved if you only modify the
Solution 2: filter connections that do not need to be tracked through iptables
-jnotrack action to the iptables rule to filter connections that do not need to be tracked (tracked). The advantage of this approach is the root cause. Connections that do not need to be tracked can be directly notrack, which does not occupy the space of the hash table and does not cause errors.
sudo iptables -t raw -A PREROUTING -p udp -j NOTRACK sudo iptables -t raw -A PREROUTING -p tcp --dport 22 -j NOTRACK
Note: The Command here indicates that the TCP connection between UDP and port 22 is not tracked. It is for reference only. The site is subject to the actual situation.
- Elastic Compute Service