Container Service for Kubernetes:Configure a fixed IP address, an independent virtual switch, and a security group for a pod

How it works

After you enable the Trunk ENI feature, the terway-controlplane component is deployed in the cluster. When a pod is created, Terway matches its labels against PodNetworking CRDs. If a match is found, Terway allocates a dedicated ENI to the pod from the vSwitch and security groups defined in the matching PodNetworking. Pods without a matching PodNetworking continue to use IP addresses from shared ENIs.

Node configuration modes

Two node modes support per-pod network isolation. Choose based on your cluster type and density requirements.

Limitations

• ACK dedicated clusters: Apply for the Container network supports Terway ENI Trunking mode quota at the Quota Centerconsole before proceeding.

• Pod count per node: The number of pods per node is limited. For details, see Work with Terway.

• Same-node traffic: Pod security group rules do not apply to traffic between pods on the same node, or between pods and their host node. To enforce isolation in these scenarios, configure a network policy.

• ACK managed clusters created before June 2020: This feature may not be supported. Follow Step 1: Check whether the cluster supports the Terway Trunk ENI feature to verify.

• Irreversible: Static IP addresses, vSwitches, and security groups cannot be disabled after they are enabled.

Prerequisites

Before you begin, ensure that you have:

• An ACK managed cluster or ACK dedicated cluster with the terway-eniip network plug-in installed

• (ACK dedicated clusters only) A submitted ticket to apply for Trunk ENI ECS instance permissions

• Nodes with instance types that support Trunk ENIs (EniTrunkSupported=true), if you plan to use the Trunk ENI node mode

• Sufficient available IP addresses in the vSwitches you plan to assign to pods

Step 1: Enable the Terway Trunk ENI feature

kubectl get secret -nkube-system addon.network.token

NAME                  TYPE     DATA   AGE
addon.network.token   Opaque   1      69m

Step 2: Create a PodNetworking

A PodNetworking is a Terway CRD that defines network configurations for a set of pods. Create one PodNetworking per network plane you want to configure.

1. Log on to the ACK console and click Clusters.

2. Click the cluster name. In the left-side navigation pane, choose Workloads > Custom Resources.

3. On the Custom Resources page, click the CRDs tab, and then click Create from YAML.

4. Paste the following template and modify the parameters for your environment:

   Parameter	Description
   allocationType.type	Elastic: IP is released when the pod is deleted. Fixed: pod retains its IP address across restarts. Fixed mode applies only to pods with fixed names (StatefulSets and pods without ownerReferences by default). If you need to configure custom workloads, configure terway-controlplane in component management. When using Fixed mode, pods are always recreated in the same zone.
   releaseStrategy	Applies only when type is Fixed. TTL: release the IP after the delay specified in releaseAfter (minimum 5 minutes). Never: the IP is never released automatically; delete PodENIs manually to free the address.
   releaseAfter	Applies only when releaseStrategy is TTL. Uses Go duration format. Examples: 2h45m, 5m0s.
   podSelector	Matches pod labels. If both podSelector and namespaceSelector are set, a pod must match both to use this PodNetworking. Each pod must match at most one PodNetworking — if a pod matches multiple, the network configuration used is undefined.
   namespaceSelector	Matches namespace labels. Same matching rules as podSelector.
   vSwitchOptions	A list of vSwitches. Terway selects one vSwitch per pod. Pods can only be scheduled to nodes in the same zone as the listed vSwitches. Make sure the vSwitches have enough available IP addresses — pod creation fails if no IPs are available. If node auto scaling is enabled, this zone constraint may prevent the node pool from scaling out.
   vSwitchSelectOptions.vSwitchSelectionPolicy	Supported in Terway v1.11.0 and later. ordered (default): use vSwitches in the order listed. most: use the vSwitch with the most available IPs. random: select randomly.
   securityGroupIDs	One to five security group IDs. All specified security groups take effect simultaneously.
   eniOptions.eniType	Supported in Terway v1.11.0 and later. Default (default): Trunk ENI mode in shared ENI clusters; exclusive ENI mode in exclusive ENI clusters. ENI: exclusive ENI mode. Trunk: Trunk ENI mode.

   apiVersion: network.alibabacloud.com/v1beta1
   kind: PodNetworking
   metadata:
     name: example
   spec:
     allocationType:
       type: Fixed          # IP allocation policy: Elastic or Fixed
       releaseStrategy: TTL # Only for Fixed policy: TTL or Never
       releaseAfter: "1h"   # Only for TTL strategy: delay before releasing the IP
     selector:
       podSelector:
         matchLabels:
           foo: bar          # Pods with this label use this PodNetworking
       namespaceSelector:
         matchLabels:
           foo: bar          # Namespace must also match if namespaceSelector is set
     securityGroupIDs:
     - sg-bpxxxx             # Up to 5 security group IDs; all take effect simultaneously
     vSwitchOptions:
     - vsw-bpxxxx            # One or more vSwitches; Terway selects one per pod
   status:
     status: Ready

   The following table describes the parameters.

5. Click Create.

Verify the PodNetworking is ready:

kubectl describe PodNetworking example  # Replace example with your PodNetworking name

Wait until the status field shows Ready. The PodNetworking does not take effect on pods until it reaches Ready status.

(Optional) Step 3: Add labels to the namespace

If your PodNetworking uses a namespaceSelector, add matching labels to the target namespace.

1. Create a test namespace:

   kubectl create ns example

2. Add the label that matches your namespaceSelector:

   kubectl label namespaces example foo=bar  # Replace example with your namespace name

3. Verify the label was applied:

   kubectl get namespace example --show-labels  # Replace example with your namespace name

   Expected output:

    NAME      STATUS   AGE   LABELS
   example   Active   24s   foo=bar,kubernetes.io/metadata.name=example

(Optional) Step 4: Deploy a pod and verify network configuration

1. Create a my-nginx.yaml file with the following content:

   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: my-nginx        # Application name
     namespace: example    # Target namespace
     labels:
       app: nginx
   spec:
     serviceName: "nginx-service"
     replicas: 1
     selector:
       matchLabels:
         app: nginx
     template:
       metadata:
         labels:
           app: nginx
           foo: bar        # Must match the podSelector in your PodNetworking
       spec:
         containers:
         - name: nginx
           image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
           ports:
           - containerPort: 80
   # If the StatefulSet requires persistent storage, add volumeClaimTemplates:
   # volumeClaimTemplates:
   # - metadata:
   #     name: nginx-storage
   #   spec:
   #     accessModes: ["ReadWriteOnce"]
   #     storageClassName: "my-storage-class"
   #     resources:
   #       requests:
   #         storage: 1Gi

2. Deploy the application:

   kubectl apply -f my-nginx.yaml

3. Verify that the pod is using the PodNetworking configuration. If the pod matched a PodNetworking, the annotation k8s.aliyun.com/pod-networking is added to the pod:

   apiVersion: v1
   kind: Pod
   metadata:
     annotations:
       k8s.aliyun.com/pod-eni: "true"
       k8s.aliyun.com/pod-networking: podnetworking
     labels:
       app: example
       pod-ip: elastic

4. Inspect the PodENI that Terway automatically created for the pod:

   kubectl get podenis.network.alibabacloud.com <my-nginx-0> -n <example> -o yaml
   # Replace <my-nginx-0> with the pod name and <example> with the namespace

   The output confirms the ENI, vSwitch, security groups, and IP address assigned to the pod:

   apiVersion: network.alibabacloud.com/v1beta1
   kind: PodENI
   metadata:
     finalizers:
     - pod-eni
     generation: 1
     name: <my-nginx-0>
     namespace: default
   spec:
     allocations:
     - allocationType:
         type: Elastic
       eni:
         id: eni-bp1xxxx
         mac: 00:16:xx:xx:xx:xx
         securityGroupIDs:
         - sg-bp1xxxx
         vSwitchID: vsw-bp1xxxx
         zone: cn-hangzhou-h
       ipv4: 192.168.x.x
       ipv4CIDR: 192.168.x.x/19
       ipv6: 2408:x:x:x:x:x:x:x
       ipv6CIDR: 2408:x:x:x::/64
     zone: cn-hangzhou-h
   status:
     eniInfos:
       eni-bp1xxxx:
         id: eni-bp1xxxx
         status: Bind
         vid: 1001
     instanceID: i-bp1xxxx
     phase: Bind
     podLastSeen: "2021-xx-xxT00:00:00Z"
     trunkENIID: eni-bp1xxxx

Stop terway-controlplane before cluster migration

If you have enabled pod custom configurations on an ACK dedicated cluster, you cannot directly migrate workloads to an ACK managed Pro cluster. Stop terway-controlplane before migrating and re-enable it afterward.

Before migration:

1. Stop terway-controlplane:

   kubectl scale deploy -nkube-system terway-controlplane --replicas 0

2. Back up the webhook configurations, then delete them:

   # Back up
   kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io terway-controlplane -oyaml > terway-controlplane.mutatingwebhookconfigurations.yaml
   kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io terway-controlplane -oyaml > terway-controlplane.validatingwebhookconfigurations.yaml
   # Delete
   kubectl delete -f terway-controlplane.mutatingwebhookconfigurations.yaml
   kubectl delete -f terway-controlplane.validatingwebhookconfigurations.yaml

After migration:

For migration instructions, see Hot migration from ACK dedicated clusters to ACK managed Pro clusters.

• If migration fails: Restore the webhook and terway-controlplane:

  kubectl apply -f terway-controlplane.mutatingwebhookconfigurations.yaml
  kubectl apply -f terway-controlplane.validatingwebhookconfigurations.yaml
  kubectl scale deploy -nkube-system terway-controlplane --replicas 1

• If migration succeeds: Delete terway-controlplane and reinstall it from the Add-ons page:

  kubectl delete deploy -nkube-system terway-controlplane

  For reinstallation instructions, see Manage system components.

FAQ

What's next

• Configure multiple security groups per ENI for more granular pod-level firewall policies: Configure multiple security groups for an ENI.

• Troubleshoot container network issues: FAQs about container networks.