All Products
Search

This Product

    :What do I do if applications on an ECS instance occasionally drop packets and the "kernel: nf_conntrack: table full, dropping packet" error messages appear in kernel logs?

    Document Center

    :What do I do if applications on an ECS instance occasionally drop packets and the "kernel: nf_conntrack: table full, dropping packet" error messages appear in kernel logs?

    Last Updated:Jul 26, 2024

    This topic describes the cause of and solutions to the issue that applications on an Elastic Compute Service (ECS) instance occasionally drop packets and the "kernel: nf_conntrack: table full, dropping packet" error messages appear in kernel logs. To query kernel logs, run the dmesg command.

    Problem description

    Packet loss occasionally occurs when you connect to applications on an ECS instance. The peripheral network of the ECS instance runs as expected. However, when you run the dmesg command to query kernel logs, the "kernel: nf_conntrack: table full, dropping packet" error messages appear. The ECS instance on which the issue occurs meets the following conditions:

    • Image: aliyun-2.1903-x64-20G-alibase-20190327.vhd or later

    • Kernel: kernel-4.19.24-9.al7 or later

    Cause

    nf_conntrack is a NAT module that tracks connection entries in the Linux operating system. The nf_conntrack module uses a hash table to record the established TCP connections. When entries the hash table are exhausted, the establishment of new TCP connections causes the module to report "nf_conntrack: table full, dropping packet" errors. Take note of the following important parameters of the nf_conntrack module:

    • nf_conntrack_buckets: the size of the hash table, which can be specified when the module is loaded or can be modified by running the sysctl command. When the system memory is greater than or equal to 4 GB, the default value is 65536.

    • nf_conntrack_max: the maximum number of nodes in the hash table, which is the maximum number of connections supported by the nf_conntrack module. When the system memory is greater than or equal to 4 GB, the default value is 262144. For servers that handle a large number of connections, you can increase the value based on your business requirements.

    • nf_conntrack_tcp_timeout_time_wait: the period for which the TCP connections can remain in the TIME_WAIT state, which is stored in the nf_conntrack module. The default value is 120 seconds.

    Solutions

    Use one of the following solutions based on your business scenario.

    Solution 1: Use the sysctl interface to change parameter values in the nf_conntrack module

    Estimate the nf_conntrack_max value required for applications in advance, and change the parameter values in the nf_conntrack module by using the sysctl interface. The following commands are used for reference.

    Note

    If your business requires a large number of concurrent connections, which are mainly short-lived connections, we recommend that you increase the values of the nf_conntrack_max and nf_conntrack_buckets parameters. This helps prevent excessive connections from exhausting the entries in the nf_conntrack hash table. In most cases, the nf_conntrack_max parameter value is four times the nf_conntrack_buckets parameter value. 

    sudo sysctl -w net.netfilter.nf_conntrack_max=1503232
sudo sysctl -w net.netfilter.nf_conntrack_buckets=375808  # This option cannot be modified during runtime if the kernel version is not 4.19.
sudo sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=60
    Note

    • The parameter values in the commands are provided only for reference. Change the values based on your business requirements. Before you modify the parameter values, we recommend that you create snapshots or back up important files in advance to ensure data security.

    • We recommend that you change the values of the nf_conntrack_buckets and nf_conntrack_max parameters together. If you change only the value of the nf_conntrack_max parameter, the linked list on the hash table may be long, and query efficiency becomes low. If you change only the value of the nf_conntrack_buckets parameter, the preceding packet dropping issue persists.

    Scenario 2: Use the Iptables utility to filter connections that do not need to be tracked

    Run the following commands to add the "-j notrack" action to Iptables rules to filter connections that do not need to be tracked. This method removes the records of the connections that do not need to be tracked from the hash table, which prevents excessive connections from causing "kernel: nf_conntrack: table full, dropping packet" errors. This is because the "-j notrack" action takes effect directly on the corresponding connections. 

    sudo iptables -t raw -A PREROUTING -p udp -j NOTRACK
sudo iptables -t raw -A PREROUTING -p tcp --dport 22 -j NOTRACK
    Note

    The preceding commands are provided only for reference. They are run to prevent the nf_conntrack module from tracking UDP connections and the TCP connections over port 22. You can run commands based on your business requirements.

    References