Container Service for Kubernetes:Terway network plugin

Before you begin

Read this topic to understand how Terway works before you use the Terway container network plugin.

First, read Networking overview and Terway vs. Flannel CNI plugins to understand the basic concepts of container network plugins and choose one.

Before you create a cluster, you must plan its CIDR blocks. For more information, see ACK managed cluster network planning.

Billing

The Terway plugin is free of charge. However, Terway deploys pods to each node, which consume a small amount of node resources. For details on billing for ACK cloud services, see Cloud resource costs.

Important

The Terway configuration file eni-config contains critical system parameters. Modifying or deleting fields not explicitly permitted for user configuration can cause network disruptions or prevent pods from being created. For information about the parameters that you can customize, see Customize Terway parameters.

The Terway component uses CRDs to track the status of resources. Do not manually modify these system resources. Unauthorized changes can cause network disruptions or prevent pods from being created.

Maximum pods per node

When using the Terway network plugin, the maximum number of pods a node can run depends on the number of elastic network interfaces (ENIs) its ECS instance type supports. Terway enforces a minimum pod limit, which each node must meet to be successfully added to a cluster.

Install the Terway network plug-in

You must install the Terway network plug-in when you create a cluster. You cannot change the network plug-in type for an existing cluster.

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click Create Kubernetes Cluster.

3. Configure the key network parameters for the Terway network plug-in. For other cluster creation parameters, see Create an ACK managed cluster.

   Parameter	Description
   IPv6 Dual-stack	Select Enable to create a dual-stack cluster that supports both IPv4 and IPv6 addresses. Supported only for Kubernetes 1.22 or later, only with Terway, and cannot be used together with eRDMA. The cluster supports both IPv4 and IPv6 protocols, but communication between worker nodes and the control plane still uses IPv4 addresses. Ensure the following: The cluster VPC supports IPv6 dual-stack. When using Terway in shared ENI mode, the instance type of the node must support IPv6 and have the same number of assignable IPv4 and IPv6 addresses.
   VPC	The VPC for the cluster.
   Network Plug-in	Select Terway.
   DataPath V2	Select this option to enable the DataPath V2 acceleration mode. In this mode, Terway uses a different traffic forwarding path than the standard shared ENI mode to provide network acceleration. For more information, see Best practices for Terway with Datapath V2. Note For new clusters that run Kubernetes 1.34 or later and use DataPath V2, kube-proxy no longer runs on Terway nodes. This mode includes built-in support for portmap, so you do not need to configure the portmap plug-in. For more information, see Configure a custom CNI chain. DataPath V2 is supported only on the following operating system images and requires Linux kernel 5.10 or later: Alibaba Cloud Linux 4 Alibaba Cloud Linux 3 (all versions) ContainerOS Ubuntu When enabled, the Terway policy container on each worker node is expected to consume an additional 0.5 CPU cores and 512 MB of memory. This resource consumption increases with the cluster size. In the default Terway configuration, the CPU limit for the policy container is 1 core, and there is no memory limit. In DataPath V2 mode, container network connection tracking (conntrack) information is stored in an eBPF map. Similar to the native conntrack mechanism in Linux, eBPF conntrack uses a Least Recently Used (LRU) algorithm. When the map is full, the oldest connection records are evicted to make space for new ones. Configure the relevant parameters based on your workload to prevent exceeding the connection limit. For more information, see Optimize conntrack configurations in Terway mode.
   NetworkPolicy support	Select this option to enable native Kubernetes NetworkPolicy. Note Starting from Terway v1.9.2, NetworkPolicy for new clusters is implemented by eBPF, and the DataPath V2 feature is enabled for the data plane. The feature that allows you to manage NetworkPolicy resources in the console is in public preview. To use this feature, submit a request in Quota Center console.
   Support for ENI Trunking	Select this option to enable Trunk ENI mode. This allows you to assign a static IP address, an independent vSwitch, and a separate security group to each pod. Note You can enable Trunk ENI for ACK managed clusters without submitting a request. If you want to enable Trunk ENI for an ACK dedicated cluster, submit a request in Quota Center console first. For newly created ACK managed clusters that run Kubernetes 1.31 or later, the Trunk ENI feature is automatically enabled. You do not need to manually select it.
   vSwitch	The vSwitches for the cluster nodes. For high availability, we recommend selecting vSwitches from at least three different availability zones.
   Pod vSwitch	The vSwitches for pods. These can be the same as the node vSwitches.
   Service CIDR	The CIDR block for Kubernetes Services. This CIDR block cannot overlap with the node CIDR block or the pod CIDR block.
   IPv6 Service CIDR Block	You can configure this parameter only after you enable IPv6 dual-stack.

Terway working modes

This topic compares the Terway modes and how they work.

Shared ENI mode and exclusive ENI mode

Terway provides two modes for assigning IP addresses to pods: shared ENI mode and exclusive ENI mode.

Important

• In Terway v1.11.0 and later, you can select either shared ENI mode or exclusive ENI mode for a node pool. This option is no longer available at the cluster level during creation.

• The node's operating system (OS) uses the primary ENI. Terway manages the remaining ENIs to configure the pod network. Do not manually configure these ENIs. If you need to manage some ENIs yourself, see Configure a filter for ENIs.

Item		Shared ENI mode	Exclusive ENI mode
Pod IP address management	ENI allocation	Multiple pods share an ENI.	Each pod is assigned a dedicated ENI on its node.
	Pod deployment density	High density, supporting hundreds of pods on a single node.	Low density. A typical node supports only a few pods.
	Network architecture
	Data path	When a pod communicates with other pods or is accessed as a Service backend, traffic passes through the node's network stack.	When a pod accesses a Service, traffic still passes through the node's OS network stack. However, for pod-to-pod communication or when a pod acts as a Service backend, traffic bypasses the node's network stack by directly using the attached ENI, which improves performance.
Scenarios		General-purpose Kubernetes workloads.	Provides network performance comparable to virtual machines, making it ideal for applications that require high network throughput or low latency.
Network acceleration		Supports DataPath V2 for network acceleration. For more information, see Network acceleration.	Network acceleration is not supported. This mode already provides excellent network performance because each pod has a dedicated ENI.
NetworkPolicy support		Supports native Kubernetes NetworkPolicy for policy-based access control. For more information, see NetworkPolicy support.	Does not support NetworkPolicy.
Node-level network configuration		Supported. For more information, see Node-level network configuration.	Supported. For more information, see Node-level network configuration.
Access control		When Trunk ENI is enabled, you can configure a static IP address, a separate vSwitch, and a separate security group for each pod. For more information, see Configure a static IP address, separate vSwitch, and security group for a pod.	Allows you to configure a static IP address, a separate vSwitch, and a separate security group for each pod.

Network acceleration

In shared ENI mode, you can enable network acceleration. This uses a different traffic forwarding path from the standard mode to achieve higher performance. Terway currently supports the DataPath V2 acceleration mode. The following section describes its features.

Important

• DataPath V2 is an upgraded version of the previous IPvlan+eBPF acceleration mode. In Terway V1.8.0 and later, DataPath V2 is the only available acceleration mode when you create a cluster and install the Terway plugin.

• The DataPath V2 and IPvlan+eBPF acceleration modes apply only to node pools in shared ENI mode and do not affect node pools in exclusive ENI mode.

DataPath V2 features	Description
Applicable Terway versions	Clusters created with Terway v1.8.0 or later.
Network architecture
Accelerated data path	When a pod accesses a Service, eBPF resolves the Service IP address to the IP address of a backend pod. When a pod communicates with a pod on a different node, eBPF is used to bypass the network stacks on both nodes. When a pod communicates with a pod on the same node, traffic not only bypasses the node's network stack but is also forwarded directly within the node without leaving it.
Performance optimization	Simplifies the on-host network forwarding path for pods, achieving performance nearly comparable to the host itself and reducing latency by 30% compared to the standard mode. eBPF replaces the traditional kube-proxy implementation for Service networking. This bypasses iptables or IPVS on the node, significantly reducing request latency and improving scalability in large clusters. eBPF also replaces iptables for pod network policies (NetworkPolicy). This avoids generating excessive iptables rules on the host, minimizing the performance impact of network policies.
Usage	When you create a cluster, set Network Plug-in to Terway and then select the DataPath V2 option.
Usage notes	Requires kernel version 5.10 or later. We recommend using Alibaba Cloud Linux OS images. The Sandboxed-Container runtime is not supported. Limitations on NetworkPolicy: The CIDR selector does not support controlling traffic to pod IP ranges. To control traffic to pods, you must use pod selectors. Support for the except keyword in CIDR selectors is limited. We recommend that you do not use this keyword. An egress NetworkPolicy may block access to pods with hostNetwork enabled or to node IP addresses within the cluster. Access from within the cluster to a public-facing Server Load Balancer (SLB) instance for a LoadBalancer Service may fail due to loopback issues. For more information, see Why am I unable to access a load balancer?. IPv6 hairpin access is not supported. Limitations on NodePort: If you access a service by using ExternalTrafficPolicy=Local, traffic may be blocked. To resolve this issue, change the setting to ExternalTrafficPolicy=Cluster. When ExternalTrafficPolicy=Cluster is used, source network address translation (SNAT) is performed on the source IP address. The available port range for SNAT is 32768 to 65535. The eBPF acceleration feature differs from the default Linux implementation. You must adjust component configurations based on your traffic volume. For more information about how to configure monitoring and adjust eBPF map limits, see Best practices for Terway Datapath V2.

The IPvlan+eBPF acceleration mode might be in use on older clusters. Its features are described below.

IPvlan+eBPF acceleration mode

IPvlan+eBPF features	Description
Applicable Terway versions	Clusters created with Terway v1.7.0 or earlier.
Network architecture
Accelerated data path	When a pod accesses a Service, eBPF is used in the pod's network namespace to resolve the Service IP address to the IP address of a backend pod. When a pod communicates with another pod, IPvlan is used to bypass the network stacks of both nodes.
Usage	When you create a cluster, set Network Plug-in to Terway and then select the Pod IPvlan option.
Usage notes	Requires kernel version 4.19 or later. We recommend using Alibaba Cloud Linux OS images. The Sandboxed-Container runtime is not supported. Limitations on NetworkPolicy: The CIDR selector does not support controlling traffic to pod IP ranges. To control traffic to pods, you must use pod selectors. Support for the except keyword in CIDR selectors is limited. We recommend that you do not use this keyword. An egress NetworkPolicy may block access to pods with hostNetwork enabled or to node IP addresses within the cluster. Access from within the cluster to a public-facing Server Load Balancer (SLB) instance for a LoadBalancer Service may fail due to loopback issues. For more information, see Why am I unable to access a load balancer?. IPv6 hairpin access is not supported. Limitations on NodePort: If you access a service by using ExternalTrafficPolicy=Local, traffic may be blocked. To resolve this issue, change the setting to ExternalTrafficPolicy=Cluster. When ExternalTrafficPolicy=Cluster is used, source network address translation (SNAT) is performed on the source IP address. The available port range for SNAT is 32768 to 65535. The eBPF acceleration feature is different from the default Linux implementation, and you must adjust component configurations based on the traffic volume of your services. Be sure to refer to Best practices for Terway Datapath V2 to configure monitoring and adjust the eBPF map limit.

Access control

Terway offers fine-grained traffic management. In shared ENI mode, this is achieved via NetworkPolicy and the Trunk ENI option. Exclusive ENI mode also provides traffic control capabilities.

NetworkPolicy

• Node pools in exclusive ENI mode do not support NetworkPolicy.

• Node pools in shared ENI mode support native Kubernetes NetworkPolicy, which allows you to control network traffic between pods by using user-defined rules.

  When you create a cluster, you can enable NetworkPolicy by selecting Terway as the Network Plug-in and selecting the NetworkPolicy support option. For more information, see Use network policies in ACK clusters.

  Note

  The feature that allows you to manage NetworkPolicy resources in the console is in public preview. To use this feature, submit a request in Quota Center console.

Static IP, vSwitch, and security group

• Node pools in exclusive ENI mode inherently support assigning a static IP address, a separate vSwitch, and a separate security group to each pod. This enables fine-grained traffic management, network isolation, policy configuration, and IP address management.

• For node pools in shared ENI mode, the optional Trunk ENI feature enables you to configure a static IP address, a separate vSwitch, and a separate security group for each pod.

  To enable this feature, set Network Plug-in to Terway and then select the Support for ENI Trunking option when you create your cluster. For more information, see Configure a static IP address, separate vSwitch, and security group for a pod.

  Note

  • You can enable Trunk ENI for ACK managed clusters without submitting a request. If you want to enable Trunk ENI for an ACK dedicated cluster, submit a request in Quota Center console first.

  • For newly created ACK managed clusters that run Kubernetes 1.31 or later, the Trunk ENI feature is automatically enabled. You do not need to manually select it.

  • After you enable Trunk ENI mode, the terway-eniip and terway-controlplane components are installed.

Scaling limits

Terway calls the OpenAPI of cloud products to manage node network interfaces and IP addresses. For the usage limits of these operations, see the documentation for each cloud product.

• shared ENI mode: A maximum of 500 nodes can be allocated concurrently.

• exclusive ENI/TrunkENI mode: A maximum of 100 pods can be allocated concurrently.

These quotas are fixed.

Data plane configuration

The Terway data plane relies on the precise sequence and integrity of its kernel-level configurations. Any uncoordinated modification of IP rule, IP route, or eBPF hooks by external components—including priority adjustments, rule overrides, or program unloading—can cause severe failures such as pod network disruptions, ineffective network policies, and unintended traffic redirection. Strictly validate all third-party components before integration to prevent conflicts.

TC filter rules

IP rules

FAQ