If you want to manage and isolate user traffic, configure network policies, and manage IP addresses in a fine-grained manner, you can enable the Terway Trunk elastic network interface (ENI) feature. This feature allows you to specify a static IP address, a separate vSwitch, and a separate security group for each pod.
Background information
Trunk elastic network interfaces (ENIs) are new virtual network interfaces. Trunk ENIs support all features provided by ENIs. In addition, when the maximum number of ENIs that you can use is reached and you want to use additional ENIs, you can associate these ENIs with a Trunk ENI. Each of these ENIs can be separately allocated to a pod.
After you enable the Trunk ENI feature for a cluster, you can customize the configuration of specific pods to use the additional ENIs that are associated with a Trunk ENI in exclusive mode and set the other pods to share the original ENIs. The pod custom configurations feature is optional. By default, all pods use IP addresses allocated from shared ENIs. To allow a pod to use custom configurations, you must first create a claim to enable the custom pod configurations feature for the pod.
After you enable custom pod configurations, the terway-controlplane component is deployed in the cluster. The following figure shows how the component works.
Limits
This feature is in public preview. To use this feature, go to the Quota Center page and submit an application.
To enable the Terway Trunk ENI feature, you need to purchase Elastic Compute Service (ECS) instances that support Trunk ENIs. If the
EniTrunkSupported=true
parameter is returned after you call the ECS API to query an instance type, the instance type supports Trunk ENIs. For more information about how to call the ECS API to query instance types, see DescribeInstanceTypes.The number of pods that can be created on a node is limited. For more information, see Work with Terway.
Pod security group rules do not apply to the traffic between pods on the same node and the traffic between pods and the host. If you want to apply pod security group rules in these scenarios, you can configure a NetworkPolicy.
Step 1: Enable the Terway Trunk ENI feature in a cluster
If you use ACK dedicated clusters, you must submit a ticket to apply for permissions to use ECS instances that support Trunk ENIs. For ACK managed clusters, you do not need to apply for permissions to use ECS instances that support Trunk ENIs.
Method 1: Enable the Terway Trunk ENI feature in a new cluster
Create an ACK cluster, set Network Plug-in to Terway, and set Terway Mode to Support for ENI Trunking. In this mode, the type of the network plug-in is terway-eniip. For more information, see Create an ACK dedicated cluster and Create an ACK managed cluster.
The Trunk ENI feature cannot be disabled after it is enabled for a new cluster.
Method 2: Enable the Trunk ENI feature in an existing cluster
Prerequisites
The type of the network plug-in used by the cluster is terway-eniip. For more information about the Terway network plug-in, see Work with Terway.
You can query the network plug-in that you have installed on the Add-ons page of the cluster.
Limits
This feature is not supported by ACK managed clusters that were created before June 2020.
This feature cannot be disabled after it is enabled.
Step 1: Check whether the cluster supports the Terway Trunk ENI feature
For ACK dedicated clusters, you must apply for permissions to use ECS instances that support Trunk ENIs. You can submit a ticket to apply for permissions to use ECS instances that support Trunk ENIs.
For existing ACK managed clusters or ACK managed clusters that are upgraded from ACK dedicated clusters, you need to check whether the cluster supports the Terway Trunk ENI feature and modify the configuration. You do not need to apply for permissions to use ECS instances that support Trunk ENIs.
Run the following command to check the token configuration:
kubectl get secret -nkube-system addon.network.token
Expected output:
NAME TYPE DATA AGE
addon.network.token Opaque 1 69m
If the token configuration exists, proceed to the next step. If the token configuration does not exist, the Terway Trunk ENI feature is not supported by the cluster. In this case, you can create a new cluster with the Terway Trunk ENI feature enabled.
Step 2: Enable terway-eniip and enable the Terway Trunk ENI feature for the cluster
Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, click the name of the cluster that you want to manage and choose in the left-side navigation pane.
On the Add-ons page, click the Networking tab and find the terway-eniip component.
In the terway-eniip card, click Upgrade to update terway-eniip to the latest version.
If the Upgrade button is not displayed, the latest version of terway-eniip is installed. You can skip this step.
Enable terway-enniip.
Run the following command to modify the eni-config ConfigMap:
kubectl edit cm -nkube-system eni-config
Modify the eni-config parameter in the YAML file. The value of eni-config must be in a valid JSON string. You must configure the following parameters:
"enable_eni_trunking": true
: Enable the Terway Trunk ENI feature."credential_path": "/var/addon/token-config"
: For ACK managed clusters, add this parameter to the YAML file if the parameter does not exist.
Sample code:
apiVersion: v1 data: 10-terway.conf: | { "cniVersion": "0.3.1", # The version number automatically changes when the version is updated. You do not need to configure the version number. "name": "terway", "eniip_virtual_type": "IPVlan", "type": "terway" } disable_network_policy: "false" eni_conf: | { "min_pool_size": 0, "enable_eni_trunking": true, # Enable the Terway Trunk ENI feature. "credential_path": "/var/addon/token-config", # The token configuration of the ACK managed cluster. Add this parameter to the YAML file if the parameter does not exist. ... } kind: ConfigMap
Run the following command to restart Terway pods for the modified ConfigMap to take effect:
kubectl delete pod -n kube-system -l app=terway-eniip
Install terway-controlplane from the Networking tab of the Add-ons page after you enable terway-eniip.
After terway-controlplane is installed, Installed is displayed in the terway-controlplane card.
Step 2: Create a PodNetworking
Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, click the name of the cluster that you want to manage and choose in the left-side navigation pane.
On the Custom Resources page, click Create from YAML to create at least one PodNetworking to describe network configurations. The following code block shows a sample PodNetworking:
Terway allows you to use custom resource definitions (CRDs) named PodNetworking to describe network configurations. You can create multiple PodNetworkings to design different network planes.
apiVersion: network.alibabacloud.com/v1beta1 kind: PodNetworking metadata: name: example spec: allocationType: type: Fixed # The policy that describes how IP addresses are allocated to pods. Valid values: Elastic and Fixed. releaseStrategy: TTL # This parameter takes effect only if the PodNetworking uses the Fixed policy. If the PodNetworking uses the Elastic policy, you do not need to configure the releaseStrategy and releaseAfter parameters. releaseAfter: "1h" # This parameter takes effect only if releaseStrategy is set to TTL. selector: podSelector: matchLabels: foo: bar namespaceSelector: matchLabels: foo: bar securityGroupIDs: - sg-bpxxxx vSwitchOptions: - vsw-bpxxxx status: status: Ready
The following table describes the parameters in the preceding command.
Parameter
Description
allocationType
(The policy that describes how IP addresses are allocated to pods)
type
Valid values:
Elastic: Allocate elastic IP addresses (EIPs) to pods. The IP addresses are released after the pods are deleted.
Fixed: Allocate static IP addresses to pods.
A PodNetworking that uses the Fixed policy takes effect only on pods that are created by StatefulSets. For more information, see StatefulSets.
If you use the Fixed policy, constraints are added to ensure that the zone where a pod is recreated is the same as the original zone.
releaseStrategy
The policy that describes how IP addresses are released. This parameter takes effect only if
type
is set toFixed
. Valid values:TTL
: IP addresses are released with a delay. After a pod is deleted, the IP address of the pod is not released until the specified period of time ends. The minimum value is 5 minutes.Never
: IP addresses are not released. If IP addresses are no longer used, you must manually delete PodENIs.
releaseAfter
The delay for releasing pod IP addresses. This parameter takes effect only if
releaseStrategy
is set toTTL
. The value must be of the Go time type. Example:2h45m
or5m0s
. For more information, see Go time type.selector
(Label selectors that are used to select pods to use the current PodNetworking)
podSelector
A selector that is used to match pod labels. Pods that match this selector use the current PodNetworking.
If both
podSelector
andnamespaceSelector
are configured, only pods that match both selectors use the current PodNetworking.Make sure that the labels of a pod match the selectors of only one PodNetworking ConfigMap. If a pod matches multiple PodNetworking ConfigMaps, the pod uses the network configurations in an arbitrary PodNetworking ConfigMap.
namespaceSelector
A selector that is used to match namespace labels. Pods that match this selector use the current PodNetworking.
If both
podSelector
andnamespaceSelector
are configured, only pods that match both selectors use the current PodNetworking.Make sure that the labels of a pod match the selectors of only one PodNetworking ConfigMap. If a pod matches multiple PodNetworking ConfigMaps, the pod uses the network configurations in an arbitrary PodNetworking ConfigMap.
vSwitchOptions
-
Specifies the vSwitches used by pods. The logical relationship among multiple vSwitches is OR. Each pod can use only one vSwitch. Terway automatically selects a proper vSwitch for each pod.
Constraints are added to ensure that the zones to which your pods can be scheduled are the same as the zones of the vSwitches that you specified in the vSwitchOptions parameter.
Make sure that the zones of the vSwitches in vSwitchOptions are the same as those of the nodes to which pods are scheduled. In addition, make sure that these vSwitches can provide sufficient idle IP addresses. Otherwise, ACK fails to create pods.
securityGroupIDs
-
Specifies the IDs of security groups. Multiple security groups take effect at the same time. You can specify at most five security groups.
Click Create.
After the PodNetworking is created, Terway automatically synchronizes the network configurations. The PodNetworking takes effect on pods only after the
status
of the PodNetworking changes toReady
.You can run the following command to check whether the resource status is
ready
:kubectl describe PodNetworking example # Replace example with an actual custom resource name.
Step 3: Create a pod
When ACK creates a pod, the labels of the pod are matched against PodNetworkings. If a pod does not match a PodNetworking, the pod is assigned an IP address from a shared ENI by default. If a pod matches a PodNetworking, an IP address is assigned to the pod based on the configurations specified in the PodNetworking. For more information, see Labels and Selectors.
Terway automatically creates CRDs named PodENI for pods that match PodNetworkings. PodENIs are used to track the resource usage of pods. PodENIs are managed by Terway and cannot be modified.
Stop terway-controlplane before cluster migration
After you enable the pod custom configurations feature for an ACK dedicated cluster, you cannot directly migrate workloads from the ACK dedicated cluster to an ACK Pro cluster. You must first stop terway-controlplane. You can restart terway-controlplane after the migration is complete.
Perform the following steps before you start the migration:
Run the following command to stop terway-controlplane:
kubectl scale deploy -nkube-system terway-controlplane --replicas 0
Run the following command to configure a webhook:
# Back up the webhook configuration. kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io terway-controlplane -oyaml > terway-controlplane.mutatingwebhookconfigurations.yaml kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io terway-controlplane -oyaml > terway-controlplane.validatingwebhookconfigurations.yaml # Delete the webhook configuration. kubectl delete -f terway-controlplane.mutatingwebhookconfigurations.yaml kubectl delete -f terway-controlplane.validatingwebhookconfigurations.yaml
Verify that the migration is successful.
NoteFor more information about the migration procedure, see Hot migration from ACK dedicated clusters to ACK Pro clusters.
If the migration fails, run the following command to restore the webhook and terway-controlplane:
# Restore the webhook configuration. kubectl apply -f terway-controlplane.mutatingwebhookconfigurations.yaml kubectl apply -f terway-controlplane.validatingwebhookconfigurations.yaml # Restore terway-controlplane. kubectl scale deploy -nkube-system terway-controlplane --replicas 1
If the migration succeeds, run the following command to delete the relevant resources:
kubectl delete deploy -nkube-system terway-controlplane
Install the terway-controlplane component from the Add-ons page. For more information, see Manage system components.
FAQ
How do I check whether a pod uses the network configurations specified in a PodNetworking ConfigMap?
If a pod uses the network configurations specified in a PodNetworking, an
annotation
whose value isk8s.aliyun.com/pod-networking
is added to the pod after the pod is created.apiVersion: v1 kind: Pod metadata: annotations: k8s.aliyun.com/pod-eni: "true" k8s.aliyun.com/pod-networking: podnetworking labels: app: example pod-ip: elastic
Terway automatically creates a PodENI with the same pod name and namespace to record the network configurations used by the pod. You can use the following method to query the PodENI:
kubectl get podenis.network.alibabacloud.com your-pod-name -n default -oyaml apiVersion: network.alibabacloud.com/v1beta1 kind: PodENI metadata: finalizers: - pod-eni generation: 1 name: your-pod-name namespace: default spec: allocations: - allocationType: type: Elastic eni: id: eni-bp1xxxx mac: 00:16:xx:xx:xx:xx securityGroupIDs: - sg-bp1xxxx vSwitchID: vsw-bp1xxxx zone: cn-hangzhou-h ipv4: 192.168.x.x ipv4CIDR: 192.168.x.x/19 ipv6: 2408:x:x:x:x:x:x:x ipv6CIDR: 2408:x:x:x::/64 zone: cn-hangzhou-h status: eniInfos: eni-bp1xxxx: id: eni-bp1xxxx status: Bind vid: 1001 instanceID: i-bp1xxxx phase: Bind podLastSeen: "2021-xx-xxT00:00:00Z" trunkENIID: eni-bp1xxxx
Why does a pod not use the network configurations specified in the desired PodNetworking ConfigMap after the pod is created?
Make sure that the status of the PodNetworking is
Ready
.Make sure that the pod labels match only the labels in the desired PodNetworking.
If the PodNetworking uses the Fixed policy, pods that are not created by StatefulSets do not match the PodNetworking.
References
You can configure multiple security groups for an Elastic Network Interface (ENI) to apply pod network firewall policies in a more flexible manner. For more information, see Configure multiple security groups for an ENI.
For more information about how to troubleshoot container network issues, see FAQ about container networks.