Expand Pod IP Capacity by Adding vSwitches in ACK Edge - Container Service for Kubernetes

Prerequisites

Before you begin, ensure that you have:

An ACK Edge cluster with the Terway Edge network plug-in deployed
Edge nodes provisioned through Edge Node Service (ENS)

Considerations

Do not remove any existing vSwitch IDs from the ConfigMap.
Pod IP addresses are allocated from the vSwitch associated with the provisioned Elastic Network Interface (ENI). New pod vSwitches may be in different vSwitches from those hosting ENS instances and ENI instances.

Diagnose insufficient IP resources

If pods are stuck in ContainerCreating status, run the following commands to confirm that the pod vSwitch has run out of IP addresses.

Find the node where the affected pods are scheduled.

kubectl get pod -o wide

Expected output:

NAME      READY   STATUS              RESTARTS   AGE     IP             NODE
cube-1    0/1     ContainerCreating   0          41s     <none>         node-1
cube-2    0/1     ContainerCreating   0          41s     <none>         node-1

Check the events on the ENS node.

kubectl describe ensnodes node-1

Expected output:

Name:         node-1
Namespace:
Labels:       name=node-1
API Version:  network.alibabacloud.com/v1beta1
....
Events:
  Type     Reason         Age      From               Message
  ----     ------         ----     ----               -------
  Warning  AllocIPFailed  1m       ens-multi-ip-node  [SDKError] API: AssignPrivateIpAddresses, ErrorCode: InvalidVSwitchId.IpNotEnough, RequestId: xxxxxxx, Message: Don't have enough private IPs in this switch.
  Warning  AllocIPFailed  1m       ens-multi-ip-node  [SDKError] API: AssignPrivateIpAddresses, ErrorCode: InvalidVSwitchId.IpNotEnough, RequestId: xxxxxxx, Message: Don't have enough private IPs in this switch.

If the Events section contains ErrorCode: InvalidVSwitchId.IpNotEnough, the pod vSwitch has run out of private IP addresses. Proceed to add a new pod vSwitch.

Add a pod vSwitch

Step 1: Create a vSwitch on the ENS network

Log on to the ENS console and choose Network Management > vSwitches in the left-side navigation pane.

On the vSwitches page, click Create vSwitch, fill in the required fields, and click Create.

Field	Description
Node	Select the ENS edge node
Network	Select the network associated with ENS
Name	Enter a name for the new vSwitch
IPv4 CIDR Block	Select a network range within the CIDR block of the ENS network

Step 2: Register the vSwitch in the Terway Edge ConfigMap

Open the privateip-config ConfigMap for editing.

kubectl edit cm privateip-config -n kube-system

Add the new vSwitch ID under vswitches. Multiple vSwitches are supported. n-xxx is the ENS network ID and ["vsw-xxx", "vsw-xxxx"] lists the vSwitch IDs within that network.

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    helm.sh/hook: pre-install
  name: privateip-config
  namespace: kube-system
data:
  privateip_conf: |
    {
      "vswitches": {
         "n-xxx": ["vsw-xxx", "vsw-xxxx"],
         "n-xxx": ["vsw-xxx", "vsw-xxxx"]
      },
      "vswitch_selection_policy": "most"
    }

Container Service for Kubernetes:Add a pod vSwitch