In most cases, a pod uses a private IP address. In a Container Service for Kubernetes (ACK) cluster that uses Terway, a pod may require a separate public IP address in some cases. This topic describes how to associate an elastic IP address (EIP) with a pod in a cluster that uses Terway.
Prerequisites
One or more of the following clusters are created. For more information about how to create the following clusters, see Create an ACK managed cluster, Create an ACK dedicated cluster, and Create an ACK Serverless cluster.
ACK managed clusters
NoteThe network plug-in of the clusters must be terway-eniip or terway-eni.
ACK dedicated clusters
NoteThe network plug-in of the clusters must be terway-eniip or terway-eni.
ACK Serverless clusters
Background information
In most cases, to enable a pod to access the Internet, the IP address of the pod is converted to an EIP that is associated with the host or the NAT gateway of the virtual private cloud (VPC) based on SNAT rules. The inbound Internet traffic to the pod is routed through the related LoadBalancer type Services. A pod may require a separate public IP address in the following scenarios.
The pod exposes random ports to the Internet. For example, game servers and audio conferences that use the UDP. For example, if the Real Time Streaming Protocol (RTSP) is used, different ports are used for different clients.
The pod competes with other pods for a public IP address to access the Internet. In this case, the pod requires a separate public IP address to access the Internet.
For more information about how to use the Terway network plug-in in an ACK cluster, see Work with Terway.
Plug-in comparison
Both the ack-extend-network-controller and Terway plug-ins support EIPs. The following table compares the ack-extend-network-controller and Terway plug-ins.
You cannot enable the EIP feature for the ack-extend-network-controller and Terway plug-ins at the same time.
- | ack-extend-network-controller | terway |
Supported cluster types |
|
|
Fixed EIPs | Supported | Unsupported |
Phase | The controller associates EIPs with pods during the pod IP allocation phase. | The Container Network Interface (CNI) plug-in associates EIPs with pods when the CNI plug-in applies configurations. |
How to enable | You can install ack-extend-network-controller from the marketplace in the ACK console and then enable the EIP controller. For more information, see App Marketplace. | For more information about how to enable the EIP feature for Terway, see Enable the EIP feature for Terway. |
Supported versions | V0.2.0 and later. | V1.0.10.280-gdc2cb6c-aliyun and later. |
Limits
ACK edge clusters do not support EIPs.
Before you use EIPs, make sure that you understand the limits on EIPs. For more information, see Limits.
If you want the system to automatically create and associate EIPs with pods, EIPs may be repeatedly created and released. As a result, specific limits on EIPs may be reached. To avoid this issue, you can add the annotation
k8s.aliyun.com/pod-eip-instanceid
to a pod to specify an existing EIP for the pod.
Pod configurations
Specify an EIP ID
You can use this method to associate an EIP with a specific pod. The method does not change the configuration of the EIP that you want to use. Controllers that manage multiple replicated pods do not support this method. You must ensure that the EIP is used by only one pod.
Pod annotation | Value |
k8s.aliyun.com/pod-eip-instanceid | Specifies the ID of an EIP. Example: epi-bp14qxxxxxxx. |
k8s.aliyun.com/eci-eip-instanceid (compatible) |
Automatically create and associate an EIP with a pod
Pod annotation | Value |
k8s.aliyun.com/pod-with-eip | Specifies whether to automatically create an EIP and associate the EIP with a pod. Valid values:
|
k8s.aliyun.com/eci-with-eip (compatible) | |
k8s.aliyun.com/eip-bandwidth | Specifies the maximum bandwidth of the EIP. Unit: Mbit/s. For more information, see Apply for an EIP. |
k8s.aliyun.com/eip-internet-charge-type | Specifies the metering method of the EIP. Valid values:
For more information, see Apply for an EIP. Note
|
k8s.aliyun.com/eip-charge-type (compatible) | |
k8s.aliyun.com/eip-common-bandwidth-package-id | Specifies the EIP bandwidth plan that you want to use. Note To use this annotation, the Terway version must be V1.2.3 or later. This annotation is available for all ack-extend-network-controller versions. |
k8s.aliyun.com/eip-isp | Specifies the line type of the EIP. Valid values:
For more information, see Apply for an EIP. Note To use this annotation, the Terway version must be V1.2.3 or later. This annotation is available for all ack-extend-network-controller versions. |
k8s.aliyun.com/eip-public-ip-address-pool-id | Specifies the EIP address pool. For more information, see Apply for an EIP. Note Only ack-extend-network-controller supports this annotation. |
k8s.aliyun.com/eip-resource-group-id | Specifies the resource group to which the EIP belongs. For more information, see Apply for an EIP. Note Only ack-extend-network-controller 0.4.0 and later support this annotation. |
k8s.aliyun.com/eip-name | The name of the EIP. For more information, see Apply for an EIP. Note Only ack-extend-network-controller 0.6.0 and later support this annotation. |
k8s.aliyun.com/eip-description | The description of the EIP. For more information, see Apply for an EIP. Note Only ack-extend-network-controller 0.6.0 and later support this annotation. |
Use a fixed EIP
If you associate a fixed EIP with a pod, the pod still uses the fixed EIP after the pod is recreated. Only ack-extend-network-controller supports fixed EIPs. You can use this method together with automatic EIP allocation to associate fixed EIPs with the pods of stateful applications.
This method is applicable to controllers that manage the replicated pods of stateful applications. Replicated pods of stateless applications do not support fixed EIPs.
You can specify an EIP by its ID. The specified EIP will not be released.
Pod annotation | Value |
k8s.aliyun.com/pod-eip-release-strategy | Specifies the EIP release policy. Valid values:
|
Control the Ready state of a pod
A pod may change the Ready state before the pod is associated with an EIP. You can use one of the following methods to control the Ready state of the pod.
Configure readiness gates for the pod
Only ack-extend-network-controller supports pod readiness gates.
After you configure readiness gates for the pod and associate an EIP with the pod, the controller automatically configures pod conditions. The pod does not change to the Ready state before the pod is associated with an EIP.
kind: Pod
...
spec:
readinessGates:
- conditionType: "k8s.aliyun.com/eip"
status:
conditions:
- lastProbeTime: "2022-12-12T03:45:48Z"
lastTransitionTime: "2022-12-12T03:45:48Z"
reason: Associate eip succeed
status: "True"
type: k8s.aliyun.com/eip
...
Configure an init container for the pod
You can configure an init container for the pod and then use the init container to check whether the pod is associated with an EIP. The following code block shows the sample init container configuration:
apiVersion: v1
kind: Pod
metadata:
name: example
annotations:
k8s.aliyun.com/pod-with-eip: "true"
spec:
containers:
- name: example
image: busybox:1.28
command: ['sh', '-c', 'echo The app is running! && sleep 3600']
initContainers:
- name: init
image: busybox:1.28
command: ['timeout', '-t' ,'60', 'sh','-c', "until grep -E '^k8s.aliyun.com\\/pod-ips=\\S?[0-9]+\\S?' /etc/podinfo/annotations; do echo waiting for annotations; sleep 2; done"]
volumeMounts:
- name: podinfo
mountPath: /etc/podinfo
volumes:
- name: podinfo
downwardAPI:
items:
- path: "labels"
fieldRef:
fieldPath: metadata.labels
- path: "annotations"
fieldRef:
fieldPath: metadata.annotations
Enable the EIP feature for ack-extend-network-controller
ack-extend-network-controller needs to access OpenAPI Explorer to create resources. You need to first grant the required permissions to the worker role of the cluster in the Resource Access Management (RAM) console. Then, you need to install ack-extend-network-controller from the marketplace of the ACK console and add annotations to create and associate an EIP with a specific pod.
Step 1: Grant EIP permissions to the worker role of the cluster in the RAM console
Perform the following steps if you use an ACK cluster:
Log on to the ACK console and click Clusters in the left-side navigation pane.
On the Clusters page, click the name of the cluster that you want to manage.
On the Cluster Information page, click the Cluster Resources tab, and then click the hyperlink next to Worker RAM Role.
On the Permissions tab, click the name of the policy that is attached to the worker role.
Click Modify Policy Document, add the following content to the policy, and then click OK.
{ "Effect": "Allow", "Action": [ "vpc:DescribeVSwitches", "vpc:AllocateEipAddress", "vpc:AllocateEipAddressPro", "vpc:DescribeEipAddresses", "vpc:AssociateEipAddress", "vpc:UnassociateEipAddress", "vpc:ReleaseEipAddress", "vpc:AddCommonBandwidthPackageIp", "vpc:RemoveCommonBandwidthPackageIp", "vpc:TagResources", "ecs:DescribeNetworkInterfaces" ], "Resource": [ "*" ], "Condition": {} }
Configure an AccessKey pair for the corresponding RAM user if you use an ACK Serverless cluster. For more information, see Create a RAM user and Create a custom policy.
Step 2: Install ack-extend-network-controller from the marketplace
Install ack-extend-network-controller from the marketplace of the ACK console. For more information, see App Marketplace.
Parameters:
clusterID: "c11ba338192xxxxxxx" # Replace with the actual value.
regionID: "cn-hangzhou" # Replace with the actual value.
vpcID: "vpc-bp1rkq0zxxxxxx" # Replace with the actual value.
enableControllers:
- eip # Enable the EIP feature.
networkController:
eipController:
maxConcurrentReconciles: 1 # Specify the maximum number of concurrent reconciles.
garbageCollectionPeriodInMinutes: 1 # Specify the fixed EIP release cycle.
customStatefulWorkloadKinds: # Specify a stateful controller. The default controller is StatefulSet.
- foo
credential: # Specify the AccessKey pair. For ACK clusters, we recommend that you use the RamRole mode.
accessKey: ""
accessSecret: ""
Step 3: Add annotations to create and associate an EIP with a pod
You can add annotations to a pod to create or associate an EIP with the pod. For more information about the annotations, see Pod configurations.
Examples
Use the following sample code to create a Deployment. The Deployment allocates an EIP from the
ppippool-bp1bbpq1jjncxxxx
EIP address pool to each pod. The maximum bandwidth of each EIP is 5 Mbit/s.apiVersion: apps/v1 kind: Deployment metadata: name: example labels: app: example spec: replicas: 1 selector: matchLabels: app: example template: metadata: labels: app: example annotations: k8s.aliyun.com/pod-with-eip: "true" k8s.aliyun.com/eip-bandwidth: "5" k8s.aliyun.com/eip-public-ip-address-pool-id: "pippool-bp1bbpq1jjncxxxx" spec: readinessGates: - conditionType: "k8s.aliyun.com/eip" containers: - name: example image: nginx
After the pod is created, access the
podeips.alibabacloud.com
resource that has the same name as the pod to check the EIP allocation information.apiVersion: alibabacloud.com/v1beta1 kind: PodEIP metadata: annotations: k8s.aliyun.com/eip-controller: ack-extend-network-controller creationTimestamp: "2022-11-18T07:46:18Z" finalizers: - podeip-controller.alibabacloud.com/finalizer generation: 1 name: example-xxxx-xxxx namespace: default resourceVersion: "597256" uid: 2fd39250-7cf0-4b6e-a581-xxxxxx spec: allocationID: eip-bp1rcs9ilupxxxxx allocationType: releaseStrategy: Follow type: Auto status: bandwidth: "5" eipAddress: 112.xx.xx.xx internetChargeType: PayByBandwidth isp: BGP networkInterfaceID: eni-bp14qrdskvxxxxx podLastSeen: "2022-11-18T08:42:29Z" privateIPAddress: 172.18.XX.XX publicIpAddressPoolID: pippool-bp1bbpq1jjncxxxx resourceGroupID: rg-acfm2omxxxxx status: InUse
Use the following sample code to create a StatefulSet. The StatefulSet creates two pods and allocates an EIP to each pod. The EIP custom resources are automatically deleted 10 minutes after the pods are deleted.
apiVersion: apps/v1 kind: StatefulSet metadata: name: example labels: app: example spec: serviceName: "example" replicas: 2 selector: matchLabels: app: example template: metadata: labels: app: example annotations: k8s.aliyun.com/pod-with-eip: "true" k8s.aliyun.com/pod-eip-release-strategy: "10m" spec: containers: - name: example image: nginx
After the pod is created, access the
podeips.alibabacloud.com
resource that has the same name as the pod to check the EIP allocation information.apiVersion: v1 items: - apiVersion: alibabacloud.com/v1beta1 kind: PodEIP metadata: annotations: k8s.aliyun.com/eip-controller: ack-extend-network-controller creationTimestamp: "2022-12-01T02:56:27Z" finalizers: - podeip-controller.alibabacloud.com/finalizer generation: 1 name: example-0 namespace: default resourceVersion: "393589" uid: c4a69543-ef01-4596-a1ff-d3199624**** spec: allocationID: eip-bp1tqnbtwxxxxx allocationType: releaseAfter: 10m releaseStrategy: TTL type: Auto status: eipAddress: 121.40.XX.XX internetChargeType: PayByBandwidth isp: BGP networkInterfaceID: eni-bp1d9uh1tphxxxx podLastSeen: "2022-12-01T02:57:05Z" privateIPAddress: 172.16.XX.XX resourceGroupID: rg-acfm2om7xxxxx status: InUse - apiVersion: alibabacloud.com/v1beta1 kind: PodEIP metadata: annotations: k8s.aliyun.com/eip-controller: ack-extend-network-controller creationTimestamp: "2022-12-01T02:56:42Z" finalizers: - podeip-controller.alibabacloud.com/finalizer generation: 1 name: example-1 namespace: default resourceVersion: "393590" uid: 3a4fbc70-1bed-4e32-8961-da84768e**** spec: allocationID: eip-bp1sdp3axxxx allocationType: releaseAfter: 10m releaseStrategy: TTL type: Auto status: eipAddress: 121.199.XX.XX internetChargeType: PayByBandwidth isp: BGP networkInterfaceID: eni-bp1d9uh1tphxxxx podLastSeen: "2022-12-01T02:57:05Z" privateIPAddress: 172.16.1.145 resourceGroupID: rg-acfm2omxxxxxx status: InUse kind: List metadata: resourceVersion: ""
The EIP custom resource is deleted 10 minutes after the pod is deleted. If a pod that has the same name is created within the 10-minute time period, the pod takes over the EIP.
Enable the EIP feature for Terway
Step 1: Update Terway to the latest version
Update Terway in your ACK cluster to a version that supports EIPs. For more information about how to upgrade Terway, see Manage components.
Terway V1.0.10.280-gdc2cb6c-aliyun
and later support EIP. We recommend that you update Terway to the latest version.
Step 2: Configure and grant EIP permissions
You must configure and grant EIP-related permissions to the cluster that uses Terway.
Grant EIP-related permissions to the RAM role of the cluster that uses Terway.
For ACK managed clusters that were created before June 2020 or ACK dedicated clusters, you must grant permissions to the worker role of the cluster.
Log on to the ACK console and click Clusters in the left-side navigation pane.
On the Clusters page, click the name of the cluster that you want to manage. Click the Cluster Resources tab and click the hyperlink to the right of Worker RAM Role.
On the Permissions tab, click the name of the policy that is attached to the RAM role. The details page of the policy appears.
Click Modify Policy Document. In the Modify Policy Document panel that appears on the right side of the page, add the following content to the
Action
field in the Policy Document section. Click OK."vpc:DescribeVSwitches", "vpc:AllocateEipAddress", "vpc:DescribeEipAddresses", "vpc:AssociateEipAddress", "vpc:UnassociateEipAddress", "vpc:ReleaseEipAddress", "vpc:AddCommonBandwidthPackageIp", "vpc:RemoveCommonBandwidthPackageIp"
NoteBefore you add the content, add a comma (,) to the end of the bottom line in the
Action
field.
For ACK Pro clusters or ACK Basic clusters that are created in June 2020 or later, the permissions of Terway are provided by the RAM role AliyunCSManagedNetworkRole. Grant EIP permissions to the AliyunCSManagedNetworkRole RAM role in the RAM console.
On the RAM role AliyunCSManagedNetworkRole page, find the policy and click Add Permissions on the right side of the page.
In the Add Permissions panel, click + Create Policy in the Select Policy section to create a custom policy.
For more information about how to create a custom policy, see Create a custom policy.
Add the following policy content to the custom policy.
For more information about how to modify a custom policy, see Modify the document and description of a custom policy.
"vpc:DescribeVSwitches", "vpc:AllocateEipAddress", "vpc:DescribeEipAddresses", "vpc:AssociateEipAddress", "vpc:UnassociateEipAddress", "vpc:ReleaseEipAddress", "vpc:AddCommonBandwidthPackageIp", "vpc:RemoveCommonBandwidthPackageIp"
Configure Terway to support EIPs.
Run the following command to modify the eni_conf ConfigMap of Terway:
kubectl edit cm eni-config -n kube-system
Add the following content to the eni_conf ConfigMap:
"enable_eip_pool": "true"
NoteIf you want to disassociate the existing EIP before you associate another EIP, add "allow_eip_rob": "true" to the eni_conf ConfigMap.
After you modify the ConfigMap, press Esc. Then, enter :wq! and press Enter to save the modified file and exit the edit mode.
Run the following command to redeploy Terway:
kubectl delete pod -n kube-system -l app=terway-eniip
Step 3: Add annotations to create and associate an EIP with a pod
You can add annotations to a pod to create or associate an EIP with the pod. For more information about the annotations, see Pod configurations. For more information about the limits on EIP, see Limits.
Add annotations as described in the following scenarios:
Automatically create and associate an EIP with a pod
Add the following annotation to create and associate an EIP with a pod:
k8s.aliyun.com/pod-with-eip: "true"
Example:
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment-basic labels: app: nginx spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: annotations: k8s.aliyun.com/pod-with-eip: "true" # Specifies that an EIP is automatically created and associated with each NGINX pod. labels: app: nginx spec: containers: - name: nginx image: nginx
Add the following annotation to specify the maximum bandwidth of the EIP. The default value is 5 Mbit/s.
k8s.aliyun.com/eip-bandwidth: "5"
Specify an existing EIP
Add the following annotation to specify the ID of an existing EIP and associate it with a pod:
k8s.aliyun.com/pod-eip-instanceid: "<youreipInstanceId>"
NoteAn EIP cannot be associated with multiple pods. Therefore, you cannot use this annotation for pods that are created by Deployments and StatefulSets.
By default, if an EIP is already associated with a pod, you fail to create another EIP for the pod. If you want to disassociate the existing EIP and create another EIP, add the "allow_eip_rob": "true" setting to the eni_conf ConfigMap as described in the preceding step.
You can specify an existing EIP only when the application runs on one replicated pod, such as a Deployment that provisions only one replicated pod.
Verify the configurations
When the pod changes to the Running state, you can check whether the value of k8s.aliyun.com/allocated-eipAddress in the pod annotations indicates the associated EIP. If yes, you can access the pod by using the EIP.