All Products
Search
Document Center

Container Service for Kubernetes:Associate an EIP with a pod

Last Updated:Oct 08, 2023

In most cases, a pod uses a private IP address. In a Container Service for Kubernetes (ACK) cluster that uses Terway, a pod may require a separate public IP address in some cases. This topic describes how to associate an elastic IP address (EIP) with a pod in a cluster that uses Terway.

Prerequisites

One or more of the following clusters are created. For more information about how to create the following clusters, see Create an ACK managed cluster, Create an ACK dedicated cluster, and Create an ACK Serverless cluster.

  • ACK managed clusters

    Note

    The network plug-in of the clusters must be terway-eniip or terway-eni.

  • ACK dedicated clusters

    Note

    The network plug-in of the clusters must be terway-eniip or terway-eni.

  • ACK Serverless clusters

Background information

In most cases, to enable a pod to access the Internet, the IP address of the pod is converted to an EIP that is associated with the host or the NAT gateway of the virtual private cloud (VPC) based on SNAT rules. The inbound Internet traffic to the pod is routed through the related LoadBalancer type Services. A pod may require a separate public IP address in the following scenarios.

  • The pod exposes random ports to the Internet. For example, game servers and audio conferences that use the UDP. For example, if the Real Time Streaming Protocol (RTSP) is used, different ports are used for different clients.

  • The pod competes with other pods for a public IP address to access the Internet. In this case, the pod requires a separate public IP address to access the Internet.

For more information about how to use the Terway network plug-in in an ACK cluster, see Work with Terway.

Plug-in comparison

Both the ack-extend-network-controller and Terway plug-ins support EIPs. The following table compares the ack-extend-network-controller and Terway plug-ins.

Warning

You cannot enable the EIP feature for the ack-extend-network-controller and Terway plug-ins at the same time.

-

ack-extend-network-controller

terway

Supported cluster types

  • ACK managed clusters

  • ACK dedicated clusters

  • ACK Serverless clusters

  • ACK managed clusters

  • ACK dedicated clusters

Fixed EIPs

Supported

Unsupported

Phase

The controller associates EIPs with pods during the pod IP allocation phase.

The Container Network Interface (CNI) plug-in associates EIPs with pods when the CNI plug-in applies configurations.

How to enable

You can install ack-extend-network-controller from the marketplace in the ACK console and then enable the EIP controller. For more information, see App Marketplace.

For more information about how to enable the EIP feature for Terway, see Enable the EIP feature for Terway.

Supported versions

V0.2.0 and later.

V1.0.10.280-gdc2cb6c-aliyun and later.

Limits

  • ACK edge clusters do not support EIPs.

  • Before you use EIPs, make sure that you understand the limits on EIPs. For more information, see Limits.

  • If you want the system to automatically create and associate EIPs with pods, EIPs may be repeatedly created and released. As a result, specific limits on EIPs may be reached. To avoid this issue, you can add the annotation k8s.aliyun.com/pod-eip-instanceid to a pod to specify an existing EIP for the pod.

Pod configurations

Specify an EIP ID

You can use this method to associate an EIP with a specific pod. The method does not change the configuration of the EIP that you want to use. Controllers that manage multiple replicated pods do not support this method. You must ensure that the EIP is used by only one pod.

Pod annotation

Value

k8s.aliyun.com/pod-eip-instanceid

Specifies the ID of an EIP. Example: epi-bp14qxxxxxxx.

k8s.aliyun.com/eci-eip-instanceid (compatible)

Automatically create and associate an EIP with a pod

Pod annotation

Value

k8s.aliyun.com/pod-with-eip

Specifies whether to automatically create an EIP and associate the EIP with a pod. Valid values:

  • true: automatically creates an EIP and associates the EIP with a pod.

  • false: does not automatically create an EIP or associate the EIP with a pod.

k8s.aliyun.com/eci-with-eip (compatible)

k8s.aliyun.com/eip-bandwidth

Specifies the maximum bandwidth of the EIP. Unit: Mbit/s. For more information, see Apply for an EIP.

k8s.aliyun.com/eip-internet-charge-type

Specifies the metering method of the EIP. Valid values:

  • PayByTraffic: pay-by-data-transfer

  • PayByBandwidth: pay-by-bandwidth

For more information, see Apply for an EIP.

Note
  • In Terway mode, the default value is PayByTraffic.

  • In ack-extend-network-controller mode, the default value is PayByBandwidth.

k8s.aliyun.com/eip-charge-type (compatible)

k8s.aliyun.com/eip-common-bandwidth-package-id

Specifies the EIP bandwidth plan that you want to use.

Note

To use this annotation, the Terway version must be V1.2.3 or later. This annotation is available for all ack-extend-network-controller versions.

k8s.aliyun.com/eip-isp

Specifies the line type of the EIP. Valid values:

  • BGP: BGP (Multi-ISP) lines.

  • BGP_PRO: BGP (Multi-ISP) Pro lines.

For more information, see Apply for an EIP.

Note

To use this annotation, the Terway version must be V1.2.3 or later. This annotation is available for all ack-extend-network-controller versions.

k8s.aliyun.com/eip-public-ip-address-pool-id

Specifies the EIP address pool. For more information, see Apply for an EIP.

Note

Only ack-extend-network-controller supports this annotation.

k8s.aliyun.com/eip-resource-group-id

Specifies the resource group to which the EIP belongs. For more information, see Apply for an EIP.

Note

Only ack-extend-network-controller 0.4.0 and later support this annotation.

k8s.aliyun.com/eip-name

The name of the EIP. For more information, see Apply for an EIP.

Note

Only ack-extend-network-controller 0.6.0 and later support this annotation.

k8s.aliyun.com/eip-description

The description of the EIP. For more information, see Apply for an EIP.

Note

Only ack-extend-network-controller 0.6.0 and later support this annotation.

Use a fixed EIP

If you associate a fixed EIP with a pod, the pod still uses the fixed EIP after the pod is recreated. Only ack-extend-network-controller supports fixed EIPs. You can use this method together with automatic EIP allocation to associate fixed EIPs with the pods of stateful applications.

Note
  • This method is applicable to controllers that manage the replicated pods of stateful applications. Replicated pods of stateless applications do not support fixed EIPs.

  • You can specify an EIP by its ID. The specified EIP will not be released.

Pod annotation

Value

k8s.aliyun.com/pod-eip-release-strategy

Specifies the EIP release policy. Valid values:

  • Follow: follows the lifecycle of the pod that is associated with the EIP. This is the default value.

  • Never: does not release the EIP. You need to manually release the EIP when you no longer need the EIP.

  • You can also specify the timeout period of the EIP. For example, if you set the time period to 5m30s, the EIP is released 5.5 minutes after the pod is deleted. Time expressions written in Go are supported.

Control the Ready state of a pod

A pod may change the Ready state before the pod is associated with an EIP. You can use one of the following methods to control the Ready state of the pod.

Configure readiness gates for the pod

Note

Only ack-extend-network-controller supports pod readiness gates.

After you configure readiness gates for the pod and associate an EIP with the pod, the controller automatically configures pod conditions. The pod does not change to the Ready state before the pod is associated with an EIP.

kind: Pod
...
spec:
  readinessGates:
  - conditionType: "k8s.aliyun.com/eip"
status:
  conditions:
  - lastProbeTime: "2022-12-12T03:45:48Z"
    lastTransitionTime: "2022-12-12T03:45:48Z"
    reason: Associate eip succeed
    status: "True"
    type: k8s.aliyun.com/eip
...

Configure an init container for the pod

You can configure an init container for the pod and then use the init container to check whether the pod is associated with an EIP. The following code block shows the sample init container configuration:

apiVersion: v1
kind: Pod
metadata:
  name: example
  annotations:
    k8s.aliyun.com/pod-with-eip: "true"
spec:
  containers:
  - name: example
    image: busybox:1.28
    command: ['sh', '-c', 'echo The app is running! && sleep 3600']
  initContainers:
  - name: init
    image: busybox:1.28
    command: ['timeout', '-t' ,'60', 'sh','-c', "until grep -E '^k8s.aliyun.com\\/pod-ips=\\S?[0-9]+\\S?' /etc/podinfo/annotations; do echo waiting for annotations; sleep 2; done"]
    volumeMounts:
      - name: podinfo
        mountPath: /etc/podinfo
  volumes:
    - name: podinfo
      downwardAPI:
        items:
          - path: "labels"
            fieldRef:
              fieldPath: metadata.labels
          - path: "annotations"
            fieldRef:
              fieldPath: metadata.annotations

Enable the EIP feature for ack-extend-network-controller

ack-extend-network-controller needs to access OpenAPI Explorer to create resources. You need to first grant the required permissions to the worker role of the cluster in the Resource Access Management (RAM) console. Then, you need to install ack-extend-network-controller from the marketplace of the ACK console and add annotations to create and associate an EIP with a specific pod.

Step 1: Grant EIP permissions to the worker role of the cluster in the RAM console

  • Perform the following steps if you use an ACK cluster:

    1. Log on to the ACK console and click Clusters in the left-side navigation pane.

    2. On the Clusters page, click the name of the cluster that you want to manage.

    3. On the Cluster Information page, click the Cluster Resources tab, and then click the hyperlink next to Worker RAM Role.

    4. On the Permissions tab, click the name of the policy that is attached to the worker role.

    5. Click Modify Policy Document, add the following content to the policy, and then click OK.

           {
            "Effect": "Allow",
            "Action": [
              "vpc:DescribeVSwitches",
              "vpc:AllocateEipAddress",
              "vpc:AllocateEipAddressPro",
              "vpc:DescribeEipAddresses",
              "vpc:AssociateEipAddress",
              "vpc:UnassociateEipAddress",
              "vpc:ReleaseEipAddress",
              "vpc:AddCommonBandwidthPackageIp",
              "vpc:RemoveCommonBandwidthPackageIp",
              "vpc:TagResources",
              "ecs:DescribeNetworkInterfaces"
            ],
            "Resource": [
              "*"
            ],
            "Condition": {}
          }
      授权
  • Configure an AccessKey pair for the corresponding RAM user if you use an ACK Serverless cluster. For more information, see Create a RAM user and Create a custom policy.

Step 2: Install ack-extend-network-controller from the marketplace

Install ack-extend-network-controller from the marketplace of the ACK console. For more information, see App Marketplace.

Parameters:

clusterID: "c11ba338192xxxxxxx"          # Replace with the actual value. 
regionID: "cn-hangzhou"                  # Replace with the actual value. 
vpcID: "vpc-bp1rkq0zxxxxxx"              # Replace with the actual value. 
enableControllers:
  - eip                                  # Enable the EIP feature. 

networkController:
  eipController:
    maxConcurrentReconciles: 1            # Specify the maximum number of concurrent reconciles. 
    garbageCollectionPeriodInMinutes: 1   # Specify the fixed EIP release cycle. 
  customStatefulWorkloadKinds:            # Specify a stateful controller. The default controller is StatefulSet. 
  - foo

credential:                               # Specify the AccessKey pair. For ACK clusters, we recommend that you use the RamRole mode. 
  accessKey: ""
  accessSecret: ""

Step 3: Add annotations to create and associate an EIP with a pod

You can add annotations to a pod to create or associate an EIP with the pod. For more information about the annotations, see Pod configurations.

Examples

  1. Use the following sample code to create a Deployment. The Deployment allocates an EIP from the ppippool-bp1bbpq1jjncxxxx EIP address pool to each pod. The maximum bandwidth of each EIP is 5 Mbit/s.

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: example
      labels:
        app: example
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: example
      template:
        metadata:
          labels:
            app: example
          annotations:
            k8s.aliyun.com/pod-with-eip: "true"
            k8s.aliyun.com/eip-bandwidth: "5"
            k8s.aliyun.com/eip-public-ip-address-pool-id: "pippool-bp1bbpq1jjncxxxx"
        spec:
          readinessGates:
          - conditionType: "k8s.aliyun.com/eip"
          containers:
          - name: example
            image: nginx
  2. After the pod is created, access the podeips.alibabacloud.com resource that has the same name as the pod to check the EIP allocation information.

    apiVersion: alibabacloud.com/v1beta1
    kind: PodEIP
    metadata:
      annotations:
        k8s.aliyun.com/eip-controller: ack-extend-network-controller
      creationTimestamp: "2022-11-18T07:46:18Z"
      finalizers:
      - podeip-controller.alibabacloud.com/finalizer
      generation: 1
      name: example-xxxx-xxxx
      namespace: default
      resourceVersion: "597256"
      uid: 2fd39250-7cf0-4b6e-a581-xxxxxx
    spec:
      allocationID: eip-bp1rcs9ilupxxxxx
      allocationType:
        releaseStrategy: Follow
        type: Auto
    status:
      bandwidth: "5"
      eipAddress: 112.xx.xx.xx
      internetChargeType: PayByBandwidth
      isp: BGP
      networkInterfaceID: eni-bp14qrdskvxxxxx
      podLastSeen: "2022-11-18T08:42:29Z"
      privateIPAddress: 172.18.XX.XX
      publicIpAddressPoolID: pippool-bp1bbpq1jjncxxxx
      resourceGroupID: rg-acfm2omxxxxx
      status: InUse
  3. Use the following sample code to create a StatefulSet. The StatefulSet creates two pods and allocates an EIP to each pod. The EIP custom resources are automatically deleted 10 minutes after the pods are deleted.

    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: example
      labels:
        app: example
    spec:
      serviceName: "example"
      replicas: 2
      selector:
        matchLabels:
          app: example
      template:
        metadata:
          labels:
            app: example
          annotations:
            k8s.aliyun.com/pod-with-eip: "true"
            k8s.aliyun.com/pod-eip-release-strategy: "10m"
        spec:
          containers:
          - name: example
            image: nginx
  4. After the pod is created, access the podeips.alibabacloud.com resource that has the same name as the pod to check the EIP allocation information.

    apiVersion: v1
    items:
    - apiVersion: alibabacloud.com/v1beta1
      kind: PodEIP
      metadata:
        annotations:
          k8s.aliyun.com/eip-controller: ack-extend-network-controller
        creationTimestamp: "2022-12-01T02:56:27Z"
        finalizers:
        - podeip-controller.alibabacloud.com/finalizer
        generation: 1
        name: example-0
        namespace: default
        resourceVersion: "393589"
        uid: c4a69543-ef01-4596-a1ff-d3199624****
      spec:
        allocationID: eip-bp1tqnbtwxxxxx
        allocationType:
          releaseAfter: 10m
          releaseStrategy: TTL
          type: Auto
      status:
        eipAddress: 121.40.XX.XX
        internetChargeType: PayByBandwidth
        isp: BGP
        networkInterfaceID: eni-bp1d9uh1tphxxxx
        podLastSeen: "2022-12-01T02:57:05Z"
        privateIPAddress: 172.16.XX.XX
        resourceGroupID: rg-acfm2om7xxxxx
        status: InUse
    - apiVersion: alibabacloud.com/v1beta1
      kind: PodEIP
      metadata:
        annotations:
          k8s.aliyun.com/eip-controller: ack-extend-network-controller
        creationTimestamp: "2022-12-01T02:56:42Z"
        finalizers:
        - podeip-controller.alibabacloud.com/finalizer
        generation: 1
        name: example-1
        namespace: default
        resourceVersion: "393590"
        uid: 3a4fbc70-1bed-4e32-8961-da84768e****
      spec:
        allocationID: eip-bp1sdp3axxxx
        allocationType:
          releaseAfter: 10m
          releaseStrategy: TTL
          type: Auto
      status:
        eipAddress: 121.199.XX.XX
        internetChargeType: PayByBandwidth
        isp: BGP
        networkInterfaceID: eni-bp1d9uh1tphxxxx
        podLastSeen: "2022-12-01T02:57:05Z"
        privateIPAddress: 172.16.1.145
        resourceGroupID: rg-acfm2omxxxxxx
        status: InUse
    kind: List
    metadata:
      resourceVersion: ""
  5. The EIP custom resource is deleted 10 minutes after the pod is deleted. If a pod that has the same name is created within the 10-minute time period, the pod takes over the EIP.

Enable the EIP feature for Terway

Step 1: Update Terway to the latest version

Update Terway in your ACK cluster to a version that supports EIPs. For more information about how to upgrade Terway, see Manage components.

Note

Terway V1.0.10.280-gdc2cb6c-aliyun and later support EIP. We recommend that you update Terway to the latest version.

Step 2: Configure and grant EIP permissions

You must configure and grant EIP-related permissions to the cluster that uses Terway.

  1. Grant EIP-related permissions to the RAM role of the cluster that uses Terway.

    • For ACK managed clusters that were created before June 2020 or ACK dedicated clusters, you must grant permissions to the worker role of the cluster.

      1. Log on to the ACK console and click Clusters in the left-side navigation pane.

      2. On the Clusters page, click the name of the cluster that you want to manage. Click the Cluster Resources tab and click the hyperlink to the right of Worker RAM Role.

      3. On the Permissions tab, click the name of the policy that is attached to the RAM role. The details page of the policy appears.授权策略详情

      4. Click Modify Policy Document. In the Modify Policy Document panel that appears on the right side of the page, add the following content to the Action field in the Policy Document section. Click OK.修改策略内容

        "vpc:DescribeVSwitches",
        "vpc:AllocateEipAddress",
        "vpc:DescribeEipAddresses",
        "vpc:AssociateEipAddress",
        "vpc:UnassociateEipAddress",
        "vpc:ReleaseEipAddress",
        "vpc:AddCommonBandwidthPackageIp",
        "vpc:RemoveCommonBandwidthPackageIp"
        Note

        Before you add the content, add a comma (,) to the end of the bottom line in the Action field.

    • For ACK Pro clusters or ACK Basic clusters that are created in June 2020 or later, the permissions of Terway are provided by the RAM role AliyunCSManagedNetworkRole. Grant EIP permissions to the AliyunCSManagedNetworkRole RAM role in the RAM console.

      1. On the RAM role AliyunCSManagedNetworkRole page, find the policy and click Add Permissions on the right side of the page.

      2. In the Add Permissions panel, click + Create Policy in the Select Policy section to create a custom policy.

        For more information about how to create a custom policy, see Create a custom policy.

      3. Add the following policy content to the custom policy.

        For more information about how to modify a custom policy, see Modify the document and description of a custom policy.

        "vpc:DescribeVSwitches",
        "vpc:AllocateEipAddress",
        "vpc:DescribeEipAddresses",
        "vpc:AssociateEipAddress",
        "vpc:UnassociateEipAddress",
        "vpc:ReleaseEipAddress",
        "vpc:AddCommonBandwidthPackageIp",
        "vpc:RemoveCommonBandwidthPackageIp"
  2. Configure Terway to support EIPs.

    1. Run the following command to modify the eni_conf ConfigMap of Terway:

      kubectl edit cm eni-config -n kube-system
    2. Add the following content to the eni_conf ConfigMap:

      "enable_eip_pool": "true"
      Note

      If you want to disassociate the existing EIP before you associate another EIP, add "allow_eip_rob": "true" to the eni_conf ConfigMap.

    3. After you modify the ConfigMap, press Esc. Then, enter :wq! and press Enter to save the modified file and exit the edit mode.

    4. Run the following command to redeploy Terway:

      kubectl delete pod -n kube-system -l app=terway-eniip

Step 3: Add annotations to create and associate an EIP with a pod

You can add annotations to a pod to create or associate an EIP with the pod. For more information about the annotations, see Pod configurations. For more information about the limits on EIP, see Limits.

Add annotations as described in the following scenarios:

  • Automatically create and associate an EIP with a pod

    • Add the following annotation to create and associate an EIP with a pod:

      k8s.aliyun.com/pod-with-eip: "true"

      Example:

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: nginx-deployment-basic
        labels:
          app: nginx
      spec:
        replicas: 2
        selector:
          matchLabels:
            app: nginx
        template:
          metadata:
            annotations:
              k8s.aliyun.com/pod-with-eip: "true" # Specifies that an EIP is automatically created and associated with each NGINX pod. 
            labels:
              app: nginx
          spec:
            containers:
            - name: nginx
              image: nginx
    • Add the following annotation to specify the maximum bandwidth of the EIP. The default value is 5 Mbit/s.

      k8s.aliyun.com/eip-bandwidth: "5"
  • Specify an existing EIP

    Add the following annotation to specify the ID of an existing EIP and associate it with a pod:

    k8s.aliyun.com/pod-eip-instanceid: "<youreipInstanceId>"
    Note
    • An EIP cannot be associated with multiple pods. Therefore, you cannot use this annotation for pods that are created by Deployments and StatefulSets.

    • By default, if an EIP is already associated with a pod, you fail to create another EIP for the pod. If you want to disassociate the existing EIP and create another EIP, add the "allow_eip_rob": "true" setting to the eni_conf ConfigMap as described in the preceding step.

    • You can specify an existing EIP only when the application runs on one replicated pod, such as a Deployment that provisions only one replicated pod.

Verify the configurations

When the pod changes to the Running state, you can check whether the value of k8s.aliyun.com/allocated-eipAddress in the pod annotations indicates the associated EIP. If yes, you can access the pod by using the EIP.