how to configure independent virtual switches and security groups for pods - Container Service for Kubernetes

In an ACK serverless cluster, each pod on a virtual node uses shared network configurations by default. To assign separate vSwitches and security groups to individual pods, create a PodNetworking ConfigMap. A PodNetworking ConfigMap defines a network plane with specific vSwitches, security groups, and label-based selectors that determine which pods use the configuration.

Prerequisites

Before you begin, make sure that you have:

An ACK serverless cluster
At least one vSwitch in the same Virtual Private Cloud (VPC) as your cluster
At least one security group in the same VPC as your cluster

Step 1: Create a PodNetworking ConfigMap

Create one or more PodNetworking ConfigMaps to define separate network planes for your pods. Each PodNetworking ConfigMap specifies the vSwitches, security groups, and label selectors for a group of pods.

The following YAML shows a sample PodNetworking ConfigMap:

apiVersion: network.alibabacloud.com/v1beta1
kind: PodNetworking
metadata:
  name: example
spec:
  allocationType:
    type: Elastic
  selector:
    podSelector:
      matchLabels:
        foo: bar
    namespaceSelector:
      matchLabels:
        foo: bar
  securityGroupIDs:
  - sg-bpxxxx
  vSwitchOptions:
  - vsw-bpxxxx

Parameter reference

Parameter	Type	Description
allocationType	type	The IP address allocation method. Valid values: `Elastic` -- Elastic IP addresses (EIPs) are allocated to pods and released after the pods are deleted. `Fixed` -- Static IP addresses are allocated to pods. Pods in ACK serverless clusters do not support static IP addresses. See the note below this table.
selector	podSelector	A label selector that matches pod labels. If a pod's labels match this selector, the pod uses the network configurations in this PodNetworking ConfigMap. If both `podSelector` and `namespaceSelector` are set, a pod must match both selectors. Make sure each pod matches only one PodNetworking ConfigMap. If a pod matches multiple PodNetworking ConfigMaps, the pod uses the network configurations from an arbitrary one.
selector	namespaceSelector	A label selector that matches namespace labels. If a namespace's labels match this selector, all pods in that namespace use the network configurations in this PodNetworking ConfigMap. When no `namespaceSelector` is set, the PodNetworking ConfigMap applies to all pods that match the `podSelector` in any namespace. If both `podSelector` and `namespaceSelector` are set, a pod must match both selectors.
securityGroupIDs	N/A	A list of security group IDs. You can specify up to five security groups. In an ACK serverless cluster, only the first security group ID takes effect.
vSwitchOptions	N/A	A list of vSwitches for pods to use. The relationship among multiple vSwitches is OR -- each pod uses only one vSwitch, and virtual-kubelet selects the vSwitch automatically. See the usage notes below this table.

Important

Pods in ACK serverless clusters do not support static IP addresses. When type is set to Fixed, the PodNetworking ConfigMap takes effect on all matching pods, including pods created by StatefulSets. In this case, the PodNetworking ConfigMap does not support the releaseStrategy or releaseAfter parameter.

vSwitchOptions usage notes

Setting vSwitchOptions constrains the zones to which the pod can be scheduled. The pod is scheduled only to zones that match the zones of the specified vSwitches.
When the zones of the vSwitches in vSwitchOptions do not support ACK Serverless clusters, virtual nodes are automatically created in those zones, and pods are then scheduled to these virtual nodes.
Make sure the zones of the vSwitches in vSwitchOptions match the zones of the nodes to which pods are scheduled. Also make sure the vSwitches have enough idle IP addresses. Otherwise, ACK fails to create pods.

Step 2: Create a pod

When ACK creates a pod, it matches the pod's labels against all PodNetworking ConfigMaps:

No match: The pod uses an IP address allocated from a shared Elastic Network Interface (ENI) by default.
Match found: An IP address is allocated from an ENI based on the network configurations in the matching PodNetworking ConfigMap.

For more information about pod labels, see Labels.

Example

The following example creates a PodNetworking ConfigMap and a Deployment. The Deployment creates three pods with the label foo: bar, which matches the PodNetworking ConfigMap's pod selector. ACK then allocates network resources to these pods based on the PodNetworking ConfigMap.

---
apiVersion: network.alibabacloud.com/v1beta1
kind: PodNetworking
metadata:
  name: example
spec:
  allocationType:
    type: Elastic
  selector:
    podSelector:
      matchLabels:
        foo: bar
# Security groups that are created in the same virtual private cloud (VPC).
  securityGroupIDs:
  - sg-2zee386juyk226k1cb5h
# vSwitches that reside in the same VPC.
  vSwitchOptions:
  - vsw-2zewh0vanjx3vmjisna6z
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: elastic-podnetworking
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
# Match the pod selector of a PodNetworking ConfigMap.
        foo: bar
    spec:
      containers:
      - name: my-container
        image: nginx

Verify the configuration

After the pod is created, check whether the pod uses the network configurations from a PodNetworking ConfigMap. If the PodNetworking ConfigMap is applied, the pod has the following annotations:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.aliyun.com/pod-eni: "true"
    k8s.aliyun.com/pod-networking: podnetworking
  labels:
    app: example
    pod-ip: elastic

The k8s.aliyun.com/pod-networking annotation confirms that the pod uses the network configurations from a PodNetworking ConfigMap.

FAQ

Why does a pod not use the expected PodNetworking ConfigMap?

This usually happens when the pod's labels match more than one PodNetworking ConfigMap. When a pod matches multiple PodNetworking ConfigMaps, ACK picks one arbitrarily. Check your PodNetworking ConfigMaps and make sure each pod's labels match only one PodNetworking ConfigMap.