Configure exclusive ENI mode for a node pool - Container Service for Kubernetes

The exclusive ENI mode provides optimal network performance for pods and is ideal for network-intensive applications. This mode delivers high network throughput and ultra-low network latency for tasks like big data analytics, real-time stream processing, and network-sensitive applications such as video streaming, online gaming, or scientific computing. For high-frequency trading, this mode also provides multicast capabilities.

Limitations

Adding an ECS instance requires Terway v1.11.0 or later, and adding a Lingjun instance requires Terway v1.14.3 or later. To upgrade the component version, see Terway.
If you enable dual-stack for a cluster, node additions are subject to the same instance type restrictions as the shared ENI mode. The number of IPv4 addresses (primary and secondary) must be equal to the number of IPv6 addresses. For details about the number of IPv4 and IPv6 addresses supported by an ECS instance, see instance family.
Lingjun instances do not support dual-stack.
Pods assigned an exclusive ENI do not support eBPF network acceleration or Kubernetes NetworkPolicy.
Exclusive ENI mode requires new nodes. If you add existing nodes to the node pool, their attached elastic network interfaces are not used.
Exclusive ENI mode applies only to new nodes. After a node pool is configured with this mode, you cannot change it to shared ENI mode. You also cannot switch existing nodes from shared ENI mode to exclusive ENI mode.
You cannot configure a custom CNI chain.

Configure container networking at the node pool level

Exclusive ENI mode is a node pool mode provided by Terway. For a detailed comparison between shared ENI mode and exclusive ENI mode, see Shared ENI mode and exclusive ENI mode. Follow the procedure below to plan and create a node pool in exclusive ENI mode. After the node pool is created, you can schedule pods to it.

1. Plan the exclusive ENI node pool

In exclusive ENI mode, the maximum number of pods per node is smaller. A worker node must have more than six elastic network interfaces to be added to the cluster. For information about how to calculate the number of elastic network interfaces, see instance family.
Plan the vSwitches and security groups for your pods.

Terway supports multiple configuration methods, listed in descending order of priority:

Configure a fixed IP address, an independent virtual switch, and a security group for a pod
Node-level network configuration
Cluster default configuration: Customize Terway configuration parameters
Important
Ensure that your configuration includes the vSwitches for each node's zone. Otherwise, pod creation will fail.
Lingjun node pools do not support configuring a fixed IP address, an independent vSwitch, and a security group for a pod.

2. Create an exclusive ENI node pool

See Create and manage node pools to create a new node pool, and during the creation process, add the label k8s.aliyun.com/exclusive-mode-eni-type: eniOnly to the nodes.
You should also add a taint to the node pool to prevent pods that do not require an exclusive ENI from being scheduled to it.
Important
You must add the label when you create the node pool. Existing nodes cannot be switched to exclusive ENI mode. If you configure the label incorrectly, delete the node pool and recreate it.

Run the following command to query the allocatable resources on a node and verify that exclusive ENI mode is enabled.

kubectl describe node <node-name>

Expected output:

Capacity:
  aliyun/eni:         7
  cpu:                16
  ephemeral-storage:  123460788Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             31555380Ki
  pods:               213
Allocatable:
  aliyun/eni:         7
  cpu:                15890m
  ephemeral-storage:  113781462033
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             28587828Ki
  pods:               213

The expected output shows aliyun/eni, which indicates that the exclusive ENI mode is enabled successfully.

3. Schedule pods to the ENI node pool

You can use NodeAffinity or a PodNetworking custom resource to schedule pods to the exclusive ENI node pool.

NodeAffinity: Does not support pod-level configurations, such as using a fixed IP address or an independent vSwitch and security group.
PodNetworking: Supports pod-level configurations for vSwitches, security groups, and fixed pod IP addresses. For more information, see Configure a fixed IP address, an independent virtual switch, and a security group for a pod.
You can refer to the following YAML example and set eniType to ENI in the eniOptions field to schedule the pod to a dedicated ENI node pool.
```
apiVersion: network.alibabacloud.com/v1beta1
kind: PodNetworking
metadata:
  name: enionly
spec:
  eniOptions:
    eniType: ENI
  allocationType:
    type: Elastic
  selector:
    podSelector:
      matchLabels:
        network: enionly
```

FAQ

Verifying a pod's exclusive ENI

Terway automatically creates a PodENI resource that has the same name and namespace as the pod. This resource records the pod's network configuration.

Query the resource by running the following command:

kubectl get podeni nginx-9d557694f-rcdzs -oyaml

Expected output:

apiVersion: network.alibabacloud.com/v1beta1
kind: PodENI
metadata:
  annotations:
    k8s.aliyun.com/pod-uid: 05590939-fc51-47ab-a204-3dd187233bca
  creationTimestamp: "2024-09-13T08:09:27Z"
  finalizers:
  - pod-eni
  generation: 1
  labels:
    k8s.aliyun.com/node: cn-hangzhou.172.XX.XX.25
  name: example-9d557694f-rcdzs
  namespace: default
  resourceVersion: "1131123"
spec:
  allocations:
  - allocationType:
      type: Elastic
    eni:
      attachmentOptions: {}
      id: eni-xxxx
      mac: 00:16:3e:37:xx:xx
      securityGroupIDs:
      - sg-xxxx
      vSwitchID: vsw-xxxx
      zone: cn-hangzhou-j
    ipv4: 172.16.0.30
    ipv4CIDR: 172.16.0.0/24
    ipv6: 2408:4005:xxxx:xxxx:xxxx:xxxx:xxxx:9ad4
    ipv6CIDR: 2408:4005:39c:xxxx::/64
  zone: cn-hangzhou-j
status:
  eniInfos:
    eni-xxxx:
      id: eni-xxxx
      status: Bind
      type: Secondary
  instanceID: i-xxxx
  phase: Bind