All Products
Search
Document Center

Container Service for Kubernetes:Configure a static IP address, a separate vSwitch, and a separate security group for each pod

Last Updated:Apr 02, 2024

If you want to manage and isolate user traffic, configure network policies, and manage IP addresses in a fine-grained manner, you can enable the Terway Trunk elastic network interface (ENI) feature. This feature allows you to specify a static IP address, a separate vSwitch, and a separate security group for each pod.

Background information

Trunk elastic network interfaces (ENIs) are new virtual network interfaces. Trunk ENIs support all features provided by ENIs. In addition, when the maximum number of ENIs that you can use is reached and you want to use additional ENIs, you can associate these ENIs with a Trunk ENI. Each of these ENIs can be separately allocated to a pod.

After you enable the Trunk ENI feature for a cluster, you can customize the configuration of specific pods to use the additional ENIs that are associated with a Trunk ENI in exclusive mode and set the other pods to share the original ENIs. The pod custom configurations feature is optional. By default, all pods use IP addresses allocated from shared ENIs. To allow a pod to use custom configurations, you must first create a claim to enable the custom pod configurations feature for the pod.

After you enable custom pod configurations, the terway-controlplane component is deployed in the cluster. The following figure shows how the component works.

image

Limits

  • This feature is in public preview. To use this feature, go to the Quota Center page and submit an application.

  • To enable the Terway Trunk ENI feature, you need to purchase Elastic Compute Service (ECS) instances that support Trunk ENIs. If the EniTrunkSupported=true parameter is returned after you call the ECS API to query an instance type, the instance type supports Trunk ENIs. For more information about how to call the ECS API to query instance types, see DescribeInstanceTypes.

  • The number of pods that can be created on a node is limited. For more information, see Work with Terway.

  • Pod security group rules do not apply to the traffic between pods on the same node and the traffic between pods and the host. If you want to apply pod security group rules in these scenarios, you can configure a NetworkPolicy.

Step 1: Enable the Terway Trunk ENI feature in a cluster

If you use ACK dedicated clusters, you must submit a ticket to apply for permissions to use ECS instances that support Trunk ENIs. For ACK managed clusters, you do not need to apply for permissions to use ECS instances that support Trunk ENIs.

Method 1: Enable the Terway Trunk ENI feature in a new cluster

Create an ACK cluster, set Network Plug-in to Terway, and set Terway Mode to Support for ENI Trunking. In this mode, the type of the network plug-in is terway-eniip. For more information, see Create an ACK dedicated cluster and Create an ACK managed cluster.

Important

The Trunk ENI feature cannot be disabled after it is enabled for a new cluster.

Method 2: Enable the Trunk ENI feature in an existing cluster

Prerequisites

The type of the network plug-in used by the cluster is terway-eniip. For more information about the Terway network plug-in, see Work with Terway.

Note

You can query the network plug-in that you have installed on the Add-ons page of the cluster.

Limits

  • This feature is not supported by ACK managed clusters that were created before June 2020.

  • This feature cannot be disabled after it is enabled.

Step 1: Check whether the cluster supports the Terway Trunk ENI feature

Important
  • For ACK dedicated clusters, you must apply for permissions to use ECS instances that support Trunk ENIs. You can submit a ticket to apply for permissions to use ECS instances that support Trunk ENIs.

  • For existing ACK managed clusters or ACK managed clusters that are upgraded from ACK dedicated clusters, you need to check whether the cluster supports the Terway Trunk ENI feature and modify the configuration. You do not need to apply for permissions to use ECS instances that support Trunk ENIs.

Run the following command to check the token configuration:

kubectl get secret -nkube-system addon.network.token

Expected output:

NAME                  TYPE     DATA   AGE
addon.network.token   Opaque   1      69m

If the token configuration exists, proceed to the next step. If the token configuration does not exist, the Terway Trunk ENI feature is not supported by the cluster. In this case, you can create a new cluster with the Terway Trunk ENI feature enabled.

Step 2: Enable terway-eniip and enable the Terway Trunk ENI feature for the cluster

  1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

  2. On the Clusters page, click the name of the cluster that you want to manage and choose Operations > Add-ons in the left-side navigation pane.

  3. On the Add-ons page, click the Networking tab and find the terway-eniip component.

  4. In the terway-eniip card, click Upgrade to update terway-eniip to the latest version.

    If the Upgrade button is not displayed, the latest version of terway-eniip is installed. You can skip this step.

  5. Enable terway-enniip.

    1. Run the following command to modify the eni-config ConfigMap:

      kubectl edit cm -nkube-system eni-config
    2. Modify the eni-config parameter in the YAML file. The value of eni-config must be in a valid JSON string. You must configure the following parameters:

      • "enable_eni_trunking": true: Enable the Terway Trunk ENI feature.

      • "credential_path": "/var/addon/token-config": For ACK managed clusters, add this parameter to the YAML file if the parameter does not exist.

      Sample code:

      apiVersion: v1
      data:
        10-terway.conf: |
          {
            "cniVersion": "0.3.1", # The version number automatically changes when the version is updated. You do not need to configure the version number. 
            "name": "terway",
            "eniip_virtual_type": "IPVlan",
            "type": "terway"
          }
        disable_network_policy: "false"
        eni_conf: |
          {
            "min_pool_size": 0,
            "enable_eni_trunking": true, # Enable the Terway Trunk ENI feature. 
            "credential_path": "/var/addon/token-config", # The token configuration of the ACK managed cluster. Add this parameter to the YAML file if the parameter does not exist. 
            ...
          }
      kind: ConfigMap
    3. Run the following command to restart Terway pods for the modified ConfigMap to take effect:

      kubectl delete pod -n kube-system -l app=terway-eniip
  6. Install terway-controlplane from the Networking tab of the Add-ons page after you enable terway-eniip.

    After terway-controlplane is installed, Installed is displayed in the terway-controlplane card.

Step 2: Create a PodNetworking

  1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

  2. On the Clusters page, click the name of the cluster that you want to manage and choose Workloads > Custom Resources in the left-side navigation pane.

  3. On the Custom Resources page, click Create from YAML to create at least one PodNetworking to describe network configurations. The following code block shows a sample PodNetworking:

    Terway allows you to use custom resource definitions (CRDs) named PodNetworking to describe network configurations. You can create multiple PodNetworkings to design different network planes.

    apiVersion: network.alibabacloud.com/v1beta1
    kind: PodNetworking
    metadata:
      name: example
    spec:
      allocationType:
        type: Fixed # The policy that describes how IP addresses are allocated to pods. Valid values: Elastic and Fixed. 
        releaseStrategy: TTL # This parameter takes effect only if the PodNetworking uses the Fixed policy. If the PodNetworking uses the Elastic policy, you do not need to configure the releaseStrategy and releaseAfter parameters. 
        releaseAfter: "1h" # This parameter takes effect only if releaseStrategy is set to TTL. 
      selector:
        podSelector:
          matchLabels:
            foo: bar
        namespaceSelector:
          matchLabels:
            foo: bar
      securityGroupIDs:
      - sg-bpxxxx
      vSwitchOptions:
      - vsw-bpxxxx
    status:
      status: Ready
                    

    The following table describes the parameters in the preceding command.

    Parameter

    Description

    allocationType

    (The policy that describes how IP addresses are allocated to pods)

    type

    Valid values:

    • Elastic: Allocate elastic IP addresses (EIPs) to pods. The IP addresses are released after the pods are deleted.

    • Fixed: Allocate static IP addresses to pods.

      • A PodNetworking that uses the Fixed policy takes effect only on pods that are created by StatefulSets. For more information, see StatefulSets.

      • If you use the Fixed policy, constraints are added to ensure that the zone where a pod is recreated is the same as the original zone.

    releaseStrategy

    The policy that describes how IP addresses are released. This parameter takes effect only if type is set to Fixed. Valid values:

    • TTL: IP addresses are released with a delay. After a pod is deleted, the IP address of the pod is not released until the specified period of time ends. The minimum value is 5 minutes.

    • Never: IP addresses are not released. If IP addresses are no longer used, you must manually delete PodENIs.

    releaseAfter

    The delay for releasing pod IP addresses. This parameter takes effect only if releaseStrategy is set to TTL. The value must be of the Go time type. Example: 2h45m or 5m0s. For more information, see Go time type.

    selector

    (Label selectors that are used to select pods to use the current PodNetworking)

    podSelector

    • A selector that is used to match pod labels. Pods that match this selector use the current PodNetworking.

    • If both podSelector and namespaceSelector are configured, only pods that match both selectors use the current PodNetworking.

    • Make sure that the labels of a pod match the selectors of only one PodNetworking ConfigMap. If a pod matches multiple PodNetworking ConfigMaps, the pod uses the network configurations in an arbitrary PodNetworking ConfigMap.

    namespaceSelector

    • A selector that is used to match namespace labels. Pods that match this selector use the current PodNetworking.

    • If both podSelector and namespaceSelector are configured, only pods that match both selectors use the current PodNetworking.

    • Make sure that the labels of a pod match the selectors of only one PodNetworking ConfigMap. If a pod matches multiple PodNetworking ConfigMaps, the pod uses the network configurations in an arbitrary PodNetworking ConfigMap.

    vSwitchOptions

    -

    • Specifies the vSwitches used by pods. The logical relationship among multiple vSwitches is OR. Each pod can use only one vSwitch. Terway automatically selects a proper vSwitch for each pod.

    • Constraints are added to ensure that the zones to which your pods can be scheduled are the same as the zones of the vSwitches that you specified in the vSwitchOptions parameter.

    • Make sure that the zones of the vSwitches in vSwitchOptions are the same as those of the nodes to which pods are scheduled. In addition, make sure that these vSwitches can provide sufficient idle IP addresses. Otherwise, ACK fails to create pods.

    securityGroupIDs

    -

    Specifies the IDs of security groups. Multiple security groups take effect at the same time. You can specify at most five security groups.

  4. Click Create.

    After the PodNetworking is created, Terway automatically synchronizes the network configurations. The PodNetworking takes effect on pods only after the status of the PodNetworking changes to Ready.

    You can run the following command to check whether the resource status is ready:

    kubectl describe PodNetworking example # Replace example with an actual custom resource name.

Step 3: Create a pod

When ACK creates a pod, the labels of the pod are matched against PodNetworkings. If a pod does not match a PodNetworking, the pod is assigned an IP address from a shared ENI by default. If a pod matches a PodNetworking, an IP address is assigned to the pod based on the configurations specified in the PodNetworking. For more information, see Labels and Selectors.

Terway automatically creates CRDs named PodENI for pods that match PodNetworkings. PodENIs are used to track the resource usage of pods. PodENIs are managed by Terway and cannot be modified.

Stop terway-controlplane before cluster migration

After you enable the pod custom configurations feature for an ACK dedicated cluster, you cannot directly migrate workloads from the ACK dedicated cluster to an ACK Pro cluster. You must first stop terway-controlplane. You can restart terway-controlplane after the migration is complete.

  1. Perform the following steps before you start the migration:

    1. Run the following command to stop terway-controlplane:

      kubectl scale deploy -nkube-system terway-controlplane --replicas 0
    2. Run the following command to configure a webhook:

      # Back up the webhook configuration. 
      kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io terway-controlplane -oyaml > terway-controlplane.mutatingwebhookconfigurations.yaml
      kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io terway-controlplane -oyaml > terway-controlplane.validatingwebhookconfigurations.yaml
      # Delete the webhook configuration. 
      kubectl delete -f terway-controlplane.mutatingwebhookconfigurations.yaml
      kubectl delete -f terway-controlplane.validatingwebhookconfigurations.yaml
  2. Verify that the migration is successful.

    Note

    For more information about the migration procedure, see Hot migration from ACK dedicated clusters to ACK Pro clusters.

    • If the migration fails, run the following command to restore the webhook and terway-controlplane:

      # Restore the webhook configuration. 
      kubectl apply -f terway-controlplane.mutatingwebhookconfigurations.yaml
      kubectl apply -f terway-controlplane.validatingwebhookconfigurations.yaml
      # Restore terway-controlplane. 
      kubectl scale deploy -nkube-system terway-controlplane --replicas 1
    • If the migration succeeds, run the following command to delete the relevant resources:

      kubectl delete deploy -nkube-system terway-controlplane
  3. Install the terway-controlplane component from the Add-ons page. For more information, see Manage system components.

FAQ

How do I check whether a pod uses the network configurations specified in a PodNetworking ConfigMap?

  1. If a pod uses the network configurations specified in a PodNetworking, an annotation whose value is k8s.aliyun.com/pod-networking is added to the pod after the pod is created.

    apiVersion: v1
    kind: Pod
    metadata:
      annotations:
        k8s.aliyun.com/pod-eni: "true"
        k8s.aliyun.com/pod-networking: podnetworking
      labels:
        app: example
        pod-ip: elastic
  2. Terway automatically creates a PodENI with the same pod name and namespace to record the network configurations used by the pod. You can use the following method to query the PodENI:

    kubectl get podenis.network.alibabacloud.com your-pod-name -n default -oyaml
    apiVersion: network.alibabacloud.com/v1beta1
    kind: PodENI
    metadata:
      finalizers:
      - pod-eni
      generation: 1
      name: your-pod-name
      namespace: default
    spec:
      allocations:
      - allocationType:
          type: Elastic
        eni:
          id: eni-bp1xxxx
          mac: 00:16:xx:xx:xx:xx
          securityGroupIDs:
          - sg-bp1xxxx
          vSwitchID: vsw-bp1xxxx
          zone: cn-hangzhou-h
        ipv4: 192.168.x.x
        ipv4CIDR: 192.168.x.x/19
        ipv6: 2408:x:x:x:x:x:x:x
        ipv6CIDR: 2408:x:x:x::/64
      zone: cn-hangzhou-h
    status:
      eniInfos:
        eni-bp1xxxx:
          id: eni-bp1xxxx
          status: Bind
          vid: 1001
      instanceID: i-bp1xxxx
      phase: Bind
      podLastSeen: "2021-xx-xxT00:00:00Z"
      trunkENIID: eni-bp1xxxx

Why does a pod not use the network configurations specified in the desired PodNetworking ConfigMap after the pod is created?

  1. Make sure that the status of the PodNetworking is Ready.

  2. Make sure that the pod labels match only the labels in the desired PodNetworking.

  3. If the PodNetworking uses the Fixed policy, pods that are not created by StatefulSets do not match the PodNetworking.

References