Disable the estimation feature of IPVS to prevent network jitters - Alibaba Cloud Linux

IP Virtual Server (IPVS) collects per-service statistics through a built-in timer named estimation_timer. In large-scale environments, this timer introduces latency of tens to hundreds of milliseconds and triggers network jitters. Disable the estimation feature on Alibaba Cloud Linux to eliminate this overhead.

How the estimation feature causes jitters

IPVS

IPVS is a Linux kernel component that directs TCP and UDP requests to real servers, providing kernel-mode Layer 4 Server Load Balancer (SLB) operations. In Kubernetes, IPVS serves as a proxy for the kube-proxy component, forwarding service requests to backend Pods for load balancing.

Note

kube-proxy is a Kubernetes cluster component that provides network proxy and load balancing for services.

The estimation feature of IPVS and the cause of network request latency or jitters

The estimation_timer counts connections and packets sent and received per service, per time slice. It is enabled by default and causes network jitters for two reasons:

Linear scaling with services and CPUs. The kernel stores statistics on a per-CPU basis. As services grow, computation increases linearly. As CPUs grow, the kernel must traverse more CPUs to collect statistics. When both counts are high, each estimation_timer cycle carries significantly more overhead.
Timer priority blocks other tasks. Linux timer execution runs at high priority. While a timer runs, other tasks cannot execute—delaying packet reception, adding tens to hundreds of milliseconds of latency to application requests, and triggering jitters during ping operations.

Supported kernel versions

Disabling the IPVS estimation feature requires one of the following Alibaba Cloud Linux kernel versions:

Distribution	Minimum kernel version
Alibaba Cloud Linux 2	`4.19.91-22`
Alibaba Cloud Linux 3	`5.10.134-14`

Disable the estimation feature

Important

Disabling the estimation feature makes IPVS service statistics unavailable. Commands such as sudo ipvsadm -Ln --stats will no longer report connection count or received packet count changes. Evaluate the impact on your workloads before proceeding.

Connect to your Elastic Compute Service (ECS) instance.

For more information, see Connect to a Linux instance by using a password or key.
Configure the sysctl parameter.
1. Open the sysctl configuration file:
```
sudo vim /etc/sysctl.conf
```
2. Add the following line at the end of the file, then save and close:
```
net.ipv4.vs.run_estimation = 0
```
3. Apply the configuration:
```
sudo sysctl -p
```
Apply additional configuration for container environments.

In container environments, perform the following additional steps.
- IPVS module reloads
  
  Each time the IPVS module reloads, run sysctl -p again for the configuration to take effect. To automate this, create a modprobe configuration file:
  1. Create the file:
```
sudo vim /etc/modprobe.d/ipvs.conf
```
  2. Add the following content, then save and close:
```
install ip_vs /sbin/modprobe --ignore-install ip_vs && /sbin/sysctl -p
```
  This ensures that the sysctl.conf settings apply automatically when the system reloads the IPVS module.
- Network namespace containers
  
  If you configured IPVS rules for containers that use network namespaces, disable the estimation feature in each container separately.
  
  Note
  For Alibaba Cloud Container Service for Kubernetes (ACK), the default kube-proxy mode is IPVS, and IPVS rules are configured on the host side. ACK users do not need to disable estimation for each container. Skip this step if you use ACK.
Verify that the feature is disabled.

Use either of the following methods to confirm the estimation feature is disabled.
- Method 1: Check the sysctl parameter. Run the following command. If the output is 0, the feature is disabled.
```
sudo sysctl net.ipv4.vs.run_estimation
```
- Method 2: Check IPVS statistics. Run the following command multiple times. If the statistical values do not change between runs, the feature is disabled.
  Important
  If ipvsadm is not installed, install it first:
```
sudo yum install ipvsadm
```
```
sudo ipvsadm -Ln --stats
```

Next steps

Verify that network jitter and latency are resolved

After confirming the feature is disabled, verify that jitter and latency are resolved:

Ping test. Ping the IP address or domain name of the ECS instance. If latency remains stable and within a reasonable range, the issue is resolved.
Performance test. Use a network performance testing tool to measure latency and jitter. If long-tail latency is significantly reduced or no longer occurs, the issue is resolved.

Note
If jitter or latency persists after disabling the estimation feature, use the network diagnostics feature in the Container Service for Kubernetes (ACK) console to investigate. For more information, see Network diagnostics.

Re-enable the estimation feature

To re-enable estimation, set run_estimation to 1 in both the sysctl configuration file (/etc/sysctl.conf) and the modprobe configuration file (/etc/modprobe.d/ipvs.conf), then run sudo sysctl -p to apply.