All Products
Search

This Product

    :The application on the ECS instance occasionally suffers packet loss and the kernel Log (dmesg) contains the error message "kernel: nf_conntrack: table full, dropping packet"

    Document Center

    :The application on the ECS instance occasionally suffers packet loss and the kernel Log (dmesg) contains the error message "kernel: nf_conntrack: table full, dropping packet"

    Last Updated:Apr 12, 2022

    Problem description

    Packet loss occurs occasionally when connecting applications in ECS instances. After troubleshooting, the perimeter network of the ECS instance is normal, but the Kernel Log (dmesg) contains the following error message.

    kernel: nf_conntrack: table full, dropping packet

    The faulty ECS instance meets the following requirements:

    • Image: all versions of images and later.
    • Kernel: kernel-4.19.24-9.al7 and later versions.

    Possible cause

    nf_conntrack is a connection entry tracking module for NAT within the Linux. The nf_conntrack module uses a hash table to record the TCP protocol establishment connection. When the hash table is full, the new connection will cause the failure. The following table describes the important parameters in the nf_conntrack module.

    • nf_conntrack_buckets: the size of the hash table. You can specify parameters during module loading or modify them by running the sysctl command. When the system memory is greater than or equal to 4GB, its default value is 65536.
    • nf_conntrack_max: the maximum number of nodes in the hash table, that is, the maximum number of connections supported by the nf_conntrack module. When the system memory is larger than 4 GB, its default value is "262144". For servers handling a large number of connections, the default value is relatively small.
    • Duration: the TCP connection time in the time_wait status stored by the nf_conntrack module. The default value is 120s.

    Solution

    Alibaba Cloud reminds you that:

    • Before you perform operations that may cause risks, such as modifying instance configurations or data, we recommend that you check the disaster recovery and fault tolerance capabilities of the instances to ensure data security.
    • You can modify the configurations and data of instances including but not limited to Elastic Compute Service (ECS) and Relational Database Service (RDS) instances. Before the modification, we recommend that you create snapshots or enable RDS log backup.
    • If you have authorized or submitted sensitive information such as the logon account and password in the Alibaba Cloud Management Console, we recommend that you modify such information in a timely manner.

    Select the method that best suits your business needs from the following two methods based on the on-site conditions:

    Solution 1: Use the sysctl interface to adjust the parameter values of the nf_conntrack module.

    The business side should confirm the maximum number of nf_conntrack connections that applications may use in advance, and adjust the parameter values of nf_conntrack through the sysctl interface by referring to the following command.

    Note: if your service has a high number of concurrent connections, short-lived connections are the main feature of this service. We recommend that you increase the parameter values of nf_conntrack_max and nf_conntrack_buckets to ensure that the hash table of nf_conntrack does not become full due to excessive connections. We recommend that you set nf_conntrack_max to four times the value of nf_conntrack_buckets.

    sudo sysctl -w net.netfilter.nf_conntrack_max=1503232 sudo sysctl -w net.netfilter.nf_conntrack_buckets=375808# If you use a non -4.19 kernel, this option may not be able to modify the sudo sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=60 at runtime.

    Note:

    • The value in the command is for reference only. Adjust the value based on the actual business situation on site. Before resizing, we recommend that you create a snapshot or back up important files to ensure data security.
    • We recommend that you adjust the nf_conntrack_buckets and nf_conntrack_max parameters together. If you only change the value of nf_conntrack_max, the linked list in a hash table may be too long, causing low query efficiency. This problem cannot be solved if you only modify the nf_conntrack_buckets parameter.

    Solution 2: filter connections that do not need to be tracked through iptables

    Add the -jnotrack action to the iptables rule to filter connections that do not need to be tracked (tracked). The advantage of this approach is the root cause. Connections that do not need to be tracked can be directly notrack, which does not occupy the space of the hash table and does not cause errors.

    sudo iptables -t raw -A PREROUTING -p udp -j NOTRACK sudo iptables -t raw -A PREROUTING -p tcp --dport 22 -j NOTRACK

    Note: The Command here indicates that the TCP connection between UDP and port 22 is not tracked. It is for reference only. The site is subject to the actual situation.

    Reference

    Application scope

    • Elastic Compute Service