All Products
Search
Document Center

Container Service for Kubernetes:Notice on occasional high CPU utilization on some Linux nodes when using Terway

Last Updated:Mar 20, 2026

A Linux community kernel patch affecting eBPF programs may cause occasional and abnormal spikes in CPU utilization on some nodes. This issue occurs if your cluster uses the Terway container network plugin and has the eBPF-based DataPath V2 feature enabled. DataPath V2 is automatically enabled when you enable NetworkPolicy.

Scope of impact

Nodes that meet all of the following conditions are affected:

  • The node operating system is Alibaba Cloud Linux 3 or ContainerOS, and the kernel version is between 5.10.134-15 and 5.10.134-19.1, inclusive.

  • The cluster uses the Terway container network plugin, and DataPath V2 mode is enabled. DataPath V2 is automatically enabled when NetworkPolicy support is enabled.

Solutions

Check if nodes need to be fixed

First, determine whether the nodes in your cluster require remediation. Use ECS Cloud Assistant to run the following command in batch on the nodes in your cluster:

Check nodes

#!/bin/bash

# Script functionality:
# 1. Check if /sys/fs/bpf/tc/globals/cilium_ct4_global exists
# 2. Only proceed if kernel release contains 'al8' or 'lifsea8'
# 3. Check if kernel version is in the range 5.10.134-15 to 5.10.134-19.1 (inclusive)
# 4. Check if kpatch_22519882 module is loaded
# 5. Behavior differs by kernel type:
#    - al8: allow fix (with -y) or prompt to install
#    - lifsea8: do NOT install; instead, warn user to upgrade to ContainerOS 3.5.1
# 6. Other kernels (e.g., el8) are skipped.

set -euo pipefail

# Default: dry-run mode (no install)
INSTALL=false

# Parse arguments
while getopts "y" opt; do
  case $opt in
    y)
      INSTALL=true
      ;;
    \?)
      echo "Usage: $0 [-y]" >&2
      exit 1
      ;;
  esac
done


# Define critical paths to check (in order of preference)
declare -a CT4_GLOBAL_PATHS=(
    "/sys/fs/bpf/tc/globals/cilium_ct4_global"
    "/.lifsea/rootfs/sys/fs/bpf/tc/globals/cilium_ct4_global"
)

# Variable to store the found path
CT4_GLOBAL_PATH=""

# Check each path in order
for path in "${CT4_GLOBAL_PATHS[@]}"; do
    if [[ -e "$path" ]]; then
        CT4_GLOBAL_PATH="$path"
        break
    fi
done

# If no valid path found, skip check
if [[ -z "$CT4_GLOBAL_PATH" ]]; then
    echo "Warning: None of the expected paths for cilium_ct4_global exist. Skipping check."
    exit 0
fi

echo "Detected $CT4_GLOBAL_PATH, proceeding with kernel version check..."

# Get full kernel release
KERNEL_RELEASE=$(uname -r)
echo "Current kernel release: $KERNEL_RELEASE"

# Extract the base version (e.g., 5.10.134-19.1)
if [[ $KERNEL_RELEASE =~ 5\.10\.134-[0-9]+(\.[0-9]+)? ]]; then
    KERNEL_VERSION="${BASH_REMATCH[0]}"
else
    echo "Error: Unable to extract kernel version from $KERNEL_RELEASE" >&2
    exit 1
fi

# Determine kernel type
if [[ $KERNEL_RELEASE == *"al8"* ]]; then
    KERNEL_TYPE="al8"
elif [[ $KERNEL_RELEASE == *"lifsea8"* ]]; then
    KERNEL_TYPE="lifsea8"
else
    echo "Kernel type not supported (neither al8 nor lifsea8), skipping."
    exit 0
fi

echo "Detected kernel type: $KERNEL_TYPE"

# Define version range (inclusive): 5.10.134-15 <= version <= 5.10.134-19.1
MIN_VERSION="5.10.134-15"
MAX_VERSION="5.10.134-19.1"

# Version comparison functions using natural sort
version_ge() {
    [[ "$1" == "$(printf '%s\n' "$1" "$2" | sort -V | tail -n1)" ]]
}

version_le() {
    [[ "$1" == "$(printf '%s\n' "$1" "$2" | sort -V | head -n1)" ]]
}

# Check version range
if ! version_ge "$KERNEL_VERSION" "$MIN_VERSION"; then
    echo "Kernel version $KERNEL_VERSION is below $MIN_VERSION, skipping."
    exit 0
fi

if ! version_le "$KERNEL_VERSION" "$MAX_VERSION"; then
    echo "Kernel version $KERNEL_VERSION is above $MAX_VERSION, skipping."
    exit 0
fi

echo "Kernel version $KERNEL_VERSION is within range $MIN_VERSION ~ $MAX_VERSION."

# Check if kpatch_22519882 module is loaded
if lsmod | grep "kpatch_22519882" >/dev/null; then
    echo "kpatch_22519882 module is already loaded, no action needed."
    exit 0
fi

# If we reach here, the hotfix is missing
HOTFIX_PKG="kernel-hotfix-22519882-$KERNEL_VERSION"

case "$KERNEL_TYPE" in
    "al8")
        echo "kpatch_22519882 module is not loaded. Hotfix package '$HOTFIX_PKG' needs to be installed."

        if [[ "$INSTALL" == true ]]; then
            echo "Installing $HOTFIX_PKG..."
            if yum install -y "$HOTFIX_PKG"; then
                echo "Installation successful."
                exit 0
            else
                echo "Installation failed. Please check yum repository or permissions." >&2
                exit 1
            fi
        else
            echo "Running in dry-run mode. Use -y to install the hotfix."
            exit 1
        fi
        ;;
    "lifsea8")
        echo "WARNING: This is a lifsea8 kernel ($KERNEL_RELEASE)." >&2
        echo "The issue cannot be fixed by hotpatch. You must upgrade to ContainerOS 3.5.1 or later." >&2
        echo "See official documentation or contact support for upgrade instructions." >&2
        exit 1
        ;;
esac

If a node requires remediation, the following result is returned:

Alibaba Cloud Linux 3 nodes

Detected /sys/fs/bpf/tc/globals/cilium_ct4_global, proceeding with kernel version check...
Current kernel release: 5.10.134-18.al8.x86_64
Detected kernel type: al8
Kernel version 5.10.134-18 is within range 5.10.134-15~5.10.134-19.1.
kpatch_22519882 module is not loaded. Hotfix package 'kernel-hotfix-22519882-5.10.134-18' needs to be installed.
Running in dry-run mode.Use2yto install the hotfix.

ContainerOS nodes

Detected /sys/fs/bpf/tc/globals/cilium_ct4_global, proceeding with kernel version check...
Current kernel release: 5.10.134-18.0.1.lifsea8.x86_64
Detected kernel type: lifsea8
Kernel version 5.10.134-18.0 is within range 5.10.134-15 ~ 5.10.134-19.1.
WARNING: This is a lifsea8 kernel (5.10.134-18.0.1.lifsea8.x86_64).
The issue cannot be fixed by hotpatch. You must upgrade to ContainerOS 3.5.1 or later.
See official documentation or contact support for upgrade instructions.

If one of the following results is returned, the node does not require remediation:

  • The Terway eBPF feature is not enabled on the node. The patch installation is skipped.

    Path /sys/fs/bpf/tc/globals/cilium_ct4_global does not exist, skipping check.
  • The patch is already installed. No reinstallation is needed.

    ...
    kpatch_22519882 module is already loaded, no actions needed.

Apply the fix

Different remediation steps apply to ContainerOS and Alibaba Cloud Linux 3.

Fix for ContainerOS nodes

For ContainerOS nodes, this issue will be automatically fixed in the upcoming ContainerOS 3.5.1 release. Follow the ContainerOS image release notes. After version 3.5.1 is published, upgrade the version. For more information, see Change the operating system.

Fix for Alibaba Cloud Linux 3 nodes

Add custom data for new scale-out nodes

When you create a new node pool or edit an existing one, add the following script to the Pre-defined Custom Data field. This ensures that newly scaled-out nodes automatically install Kernel Live Patching after they start up. For detailed steps, see Create and manage node pools.

image.png

Predefined instance data

#!/bin/bash

# Function: check_and_apply_hotfix
# Purpose: Checks the current kernel version and type, verifies if it's within the supported range,
#          and installs the hotfix package (kpatch_22519882) if not already applied.
check_and_apply_hotfix() {
    local KERNEL_RELEASE
    local KERNEL_VERSION
    local KERNEL_TYPE
    local MIN_VERSION="5.10.134-15"
    local MAX_VERSION="5.10.134-19.1"
    local HOTFIX_PKG

    # Get the full kernel release string
    KERNEL_RELEASE=$(uname -r)
    echo "Current kernel release: $KERNEL_RELEASE"

    # Extract base kernel version (e.g., 5.10.134-19.1)
    if [[ $KERNEL_RELEASE =~ 5\.10\.134-[0-9]+(\.[0-9]+)? ]]; then
        KERNEL_VERSION="${BASH_REMATCH[0]}"
    else
        echo "Error: Unable to extract kernel version from $KERNEL_RELEASE" >&2
        return 1
    fi

    # Detect kernel type (al8 or lifsea8)
    if [[ $KERNEL_RELEASE == *"al8"* ]]; then
        KERNEL_TYPE="al8"
    elif [[ $KERNEL_RELEASE == *"lifsea8"* ]]; then
        KERNEL_TYPE="lifsea8"
    else
        echo "Kernel type not supported (neither al8 nor lifsea8), skipping."
        return 0
    fi

    echo "Detected kernel type: $KERNEL_TYPE"

    # Version comparison functions using natural (version) sort
    version_ge() {
        [[ "$1" == "$(printf '%s\n%s' "$1" "$2" | sort -V | tail -n1)" ]]
    }

    version_le() {
        [[ "$1" == "$(printf '%s\n%s' "$1" "$2" | sort -V | head -n1)" ]]
    }

    # Check if kernel version is >= minimum supported version
    if ! version_ge "$KERNEL_VERSION" "$MIN_VERSION"; then
        echo "Kernel version $KERNEL_VERSION is below $MIN_VERSION, skipping."
        return 0
    fi

    # Check if kernel version is <= maximum supported version
    if ! version_le "$KERNEL_VERSION" "$MAX_VERSION"; then
        echo "Kernel version $KERNEL_VERSION is above $MAX_VERSION, skipping."
        return 0
    fi

    echo "Kernel version $KERNEL_VERSION is within range $MIN_VERSION ~ $MAX_VERSION."

    # Check if the kpatch module is already loaded
    if lsmod | grep "kpatch_22519882" >/dev/null; then
        echo "kpatch_22519882 module is already loaded, no action needed."
        return 0
    fi

    # If module is not loaded, prepare the hotfix package name
    HOTFIX_PKG="kernel-hotfix-22519882-$KERNEL_VERSION"

    # Handle installation based on kernel type
    case "$KERNEL_TYPE" in
        "al8")
            echo "kpatch_22519882 module is not loaded. Hotfix package '$HOTFIX_PKG' needs to be installed."
            echo "Installing $HOTFIX_PKG..."

            if yum install -y "$HOTFIX_PKG"; then
                echo "Installation successful."
                return 0
            else
                echo "Installation failed. Please check yum repository or permissions." >&2
                return 1
            fi
            ;;
        "lifsea8")
            echo "Kernel type 'lifsea8' is recognized but not currently supported for automatic installation."
            return 0
            ;;
        *)
            echo "Unknown kernel type: $KERNEL_TYPE"
            return 1
            ;;
    esac
}

# =====================
# Call the function
# =====================
check_and_apply_hotfix
Install the patch on existing nodes

For existing Alibaba Cloud Linux 3 nodes, use ECS Cloud Assistant to run the following command on the nodes that require remediation:

Fix nodes

#!/bin/bash

# Script functionality:
# 1. Check if /sys/fs/bpf/tc/globals/cilium_ct4_global exists
# 2. Only proceed if kernel release contains 'al8' or 'lifsea8'
# 3. Check if kernel version is in the range 5.10.134-15 to 5.10.134-19.1 (inclusive)
# 4. Check if kpatch_22519882 module is loaded
# 5. Behavior differs by kernel type:
#    - al8: allow fix (with -y) or prompt to install
#    - lifsea8: do NOT install; instead, warn user to upgrade to ContainerOS 3.5.1
# 6. Other kernels (e.g., el8) are skipped.

set -euo pipefail

# Default: dry-run mode (no install)
INSTALL=true

# Parse arguments
while getopts "y" opt; do
  case $opt in
    y)
      INSTALL=true
      ;;
    \?)
      echo "Usage: $0 [-y]" >&2
      exit 1
      ;;
  esac
done


# Define critical paths to check (in order of preference)
declare -a CT4_GLOBAL_PATHS=(
    "/sys/fs/bpf/tc/globals/cilium_ct4_global"
    "/.lifsea/rootfs/sys/fs/bpf/tc/globals/cilium_ct4_global"
)

# Variable to store the found path
CT4_GLOBAL_PATH=""

# Check each path in order
for path in "${CT4_GLOBAL_PATHS[@]}"; do
    if [[ -e "$path" ]]; then
        CT4_GLOBAL_PATH="$path"
        break
    fi
done

# If no valid path found, skip check
if [[ -z "$CT4_GLOBAL_PATH" ]]; then
    echo "Warning: None of the expected paths for cilium_ct4_global exist. Skipping check."
    exit 0
fi

echo "Detected $CT4_GLOBAL_PATH, proceeding with kernel version check..."

# Get full kernel release
KERNEL_RELEASE=$(uname -r)
echo "Current kernel release: $KERNEL_RELEASE"

# Extract the base version (e.g., 5.10.134-19.1)
if [[ $KERNEL_RELEASE =~ 5\.10\.134-[0-9]+(\.[0-9]+)? ]]; then
    KERNEL_VERSION="${BASH_REMATCH[0]}"
else
    echo "Error: Unable to extract kernel version from $KERNEL_RELEASE" >&2
    exit 1
fi

# Determine kernel type
if [[ $KERNEL_RELEASE == *"al8"* ]]; then
    KERNEL_TYPE="al8"
elif [[ $KERNEL_RELEASE == *"lifsea8"* ]]; then
    KERNEL_TYPE="lifsea8"
else
    echo "Kernel type not supported (neither al8 nor lifsea8), skipping."
    exit 0
fi

echo "Detected kernel type: $KERNEL_TYPE"

# Define version range (inclusive): 5.10.134-15 <= version <= 5.10.134-19.1
MIN_VERSION="5.10.134-15"
MAX_VERSION="5.10.134-19.1"

# Version comparison functions using natural sort
version_ge() {
    [[ "$1" == "$(printf '%s\n' "$1" "$2" | sort -V | tail -n1)" ]]
}

version_le() {
    [[ "$1" == "$(printf '%s\n' "$1" "$2" | sort -V | head -n1)" ]]
}

# Check version range
if ! version_ge "$KERNEL_VERSION" "$MIN_VERSION"; then
    echo "Kernel version $KERNEL_VERSION is below $MIN_VERSION, skipping."
    exit 0
fi

if ! version_le "$KERNEL_VERSION" "$MAX_VERSION"; then
    echo "Kernel version $KERNEL_VERSION is above $MAX_VERSION, skipping."
    exit 0
fi

echo "Kernel version $KERNEL_VERSION is within range $MIN_VERSION ~ $MAX_VERSION."

# Check if kpatch_22519882 module is loaded
if lsmod | grep "kpatch_22519882" >/dev/null; then
    echo "kpatch_22519882 module is already loaded, no action needed."
    exit 0
fi

# If we reach here, the hotfix is missing
HOTFIX_PKG="kernel-hotfix-22519882-$KERNEL_VERSION"

case "$KERNEL_TYPE" in
    "al8")
        echo "kpatch_22519882 module is not loaded. Hotfix package '$HOTFIX_PKG' needs to be installed."

        if [[ "$INSTALL" == true ]]; then
            echo "Installing $HOTFIX_PKG..."
            if yum install -y "$HOTFIX_PKG"; then
                echo "Installation successful."
                exit 0
            else
                echo "Installation failed. Please check yum repository or permissions." >&2
                exit 1
            fi
        else
            echo "Running in dry-run mode. Use -y to install the hotfix."
            exit 1
        fi
        ;;
    "lifsea8")
        echo "WARNING: This is a lifsea8 kernel ($KERNEL_RELEASE)." >&2
        echo "The issue cannot be fixed by hotpatch. You must upgrade to ContainerOS 3.5.1 or later." >&2
        echo "See official documentation or contact support for upgrade instructions." >&2
        exit 1
        ;;
esac

The expected output is as follows. This indicates that the patch installation is complete:

......
Total                                                                                                                                                                                                                                                                                    1.0 MB/s |  52 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                                                                                 1/1 
  Installing       : kpatch-dnf-0.9.7_0.4-2.0.1.al8.noarch                                                                                                                                                                                                                                                           1/3 
  Running scriptlet: kpatch-dnf-0.9.7_0.4-2.0.1.al8.noarch                                                                                                                                                                                                                                                           1/3 
To enable automatic kpatch-patch subscription, run:
        $ dnf kpatch auto

  Installing       : kpatch-0.9.7-2.0.1.al8.noarch                                                                                                                                                                                                                                                                   2/3 
  Running scriptlet: kernel-hotfix-22519882-5.10.134-18-1.0-20250804154834.al8.x86_64                                                                                                                                                                                                                                3/3 
  Installing       : kernel-hotfix-22519882-5.10.134-18-1.0-20250804154834.al8.x86_64                                                                                                                                                                                                                                3/3 
  Running scriptlet: kernel-hotfix-22519882-5.10.134-18-1.0-20250804154834.al8.x86_64                                                                                                                                                                                                                                3/3 
Created symlink /etc/systemd/system/multi-user.target.wants/kpatch.service → /usr/lib/systemd/system/kpatch.service.
installing /var/khotfix/5.10.134-18.al8.x86_64/22519882/kpatch-22519882.ko (5.10.134-18.al8.x86_64)
loading patch module: /var/khotfix/5.10.134-18.al8.x86_64/22519882/kpatch-22519882.ko

  Verifying        : kpatch-0.9.7-2.0.1.al8.noarch                                                                                                                                                                                                                                                                   1/3 
  Verifying        : kpatch-dnf-0.9.7_0.4-2.0.1.al8.noarch                                                                                                                                                                                                                                                           2/3 
  Verifying        : kernel-hotfix-22519882-5.10.134-18-1.0-20250804154834.al8.x86_64                                                                                                                                                                                                                                3/3 

Installed:
  kernel-hotfix-22519882-5.10.134-18-1.0-20250804154834.al8.x86_64                                                            kpatch-0.9.7-2.0.1.al8.noarch                                                            kpatch-dnf-0.9.7_0.4-2.0.1.al8.noarch                                                           

Complete!
Installation successful.

References

Troubleshooting high CPU utilization caused by LRU hash in eBPF programs on Alibaba Cloud Linux 3