Increase the number of pod vSwitches in a cluster that uses the Terway plug-in - Container Service for Kubernetes

In a Container Service for Kubernetes (ACK) cluster that uses the Terway plug-in, if the IP addresses of vSwitches are insufficient or you want to expand the pod CIDR block, you need to increase the number of vSwitches in the cluster. This topic describes how to create a new vSwitch to provide more IP addresses for an ACK cluster.

Limits

Make sure that the zone in which the node resides is included in the zones of the vSwitches that you want to add. If the zone in which the node resides is not included in the zones of the vSwitches that you want to add, the vSwitch to which the primary elastic network interface (ENI) of the node belongs is used.
You cannot modify the configurations of the vSwitches to which existing ENIs belong. You can add new nodes after you increase the number of pod vSwitches and then modify the configurations of the vSwitches that are used by the newly added nodes.

Symptoms of insufficient IP addresses

In a cluster that uses the Terway plug-in, the following symptoms indicate that the IP addresses provided by the vSwitch in the cluster are exhausted:

Attempts to create a new pod failed. The pod that you want to create remains in the ContainerCreating state. In this case, run the following command to query the log of Terway on the node where the pod is deployed:
```
kubectl logs --tail=100 -f terway-eniip-zwjwx -n kube-system -c terway
```
If an error message similar to the following content is returned, it indicates that the IP addresses of the vSwitch are exhausted. The pod cannot be created and remains in the ContainerCreating state because no IP address is available.
```
time="2020-03-17T07:03:40Z" level=warning msg="Assign private ip address failed: Aliyun API Error: RequestId: 2095E971-E473-4BA0-853F-0C41CF52651D Status Code: 403 Code: InvalidVSwitchId.IpNotEnough Message: The specified VSwitch \"vsw-AAA\" has not enough IpAddress., retrying"
```
Log on to the Virtual Private Cloud (VPC) console. In the left-side navigation pane, click vSwitch. On the vSwitch page, find the vSwitch of your cluster and verify that 0 is displayed in the Available IP Addresses column.

Add a vSwitch

You can perform a few steps in the ACK console to add pod vSwitches. Before you add pod vSwitches for your cluster, we recommend that you update Terway to the latest version.

If your cluster uses Terway 1.4.4 or later, you can add pod vSwitches by using the ACK console or kubectl. If your cluster uses a Terway version earlier than 1.4.4, you can add pod vSwitches only by using kubectl.

Method 1: Add a vSwitch by using the ACK console

Log on to the VPC console and create a vSwitch. You must create the vSwitch in the same region as the existing vSwitch whose IP addresses are exhausted. For more information about how to create a vSwitch, see Create and manage a vSwitch.
Note To provide sufficient IP addresses for an increasing number of pods in the cluster, we recommend that the CIDR block of the vSwitch contains at least 8,192 IP addresses. This means that the prefix length of the CIDR block must be no greater than 19 bits.
Log on to the ACK console and click Clusters in the left-side navigation pane.
On the Clusters page, click the name of a cluster and choose Operations > Add-ons in the left-side navigation pane.
On the Add-ons page, click the Networking tab. Find Terway and click Upgrade. After Terway is updated to the latest version, click Configuration.
If the Upgrade button is not displayed, the Terway plug-in is of the latest version.
Note If you reinstall the component, only the configurations that are specified by using this method are retained. The system resets configurations that are specified by using other methods.

In the terway-eniip Parameters dialog box, select the vSwitch that you created in the PodVswitchId section and use the default values for other parameters. The following table describes the parameters.


Parameter	Description	Value
TerwayMemoryRequest	The memory request of the Terway container.	100Mi
TerwayMemoryLimit	The memory limit of the Terway container.	256Mi
TerwayCpuLimit	The CPU limit of the Terway container.	100m
TerwayCpuRequest	The CPU request of the Terway container.	100m
PolicyMemoryRequest	The memory request of the Policy container.	250m
PolicyCpuRequest	The CPU request of the Policy container.	100Mi
PolicyMemoryLimit	The memory limit of the Policy container.	Unlimited
PolicyCpuLimit	The CPU limit of the Policy container.	1
NetworkPolicy	Specifies whether to enable NetworkPolicies.	Enable NetworkPolicies.
PodVswitchId	The pod vSwitches that are managed by Terway.	Make sure that the zones in which the pod vSwitches reside include the zones in which the cluster nodes reside.

Click OK.

Method 2: Add a vSwitch by using kubectl

Log on to the VPC console and create a vSwitch. You must create the vSwitch in the same region as the existing vSwitch whose IP addresses are exhausted. For more information about how to create a vSwitch, see Create and manage a vSwitch.
Note To provide sufficient IP addresses for an increasing number of pods in the cluster, we recommend that the CIDR block of the vSwitch contains at least 8,192 IP addresses. This means that the prefix length of the CIDR block must be no greater than 19 bits.
Run the following command to add the newly created vSwitch to the ConfigMap of Terway:
```
kubectl edit cm eni-config -n kube-system
```
Example:
```
eni_conf: |
{
"version": "1",
"max_pool_size": 25,
"min_pool_size": 10,
"vswitches": {"cn-shanghai-f":["vsw-AAA", "vsw-BBB"]},
"service_cidr": "172.21.0.0/20",
"security_group": "sg-CCC"
}
```
In this example, vsw-BBB is added to the value of the vswitches parameter. vsw-AAA represents the existing vSwitch that has insufficient IP addresses.
Run the following commands to delete all pods that are created for Terway. ACK automatically recreates the pods.
- If an ENI is shared among multiple pods, run the following command:
```
kubectl delete -n kube-system pod -l app=terway-eniip
```
- If an ENI is exclusive to one pod, run the following command:
```
kubectl delete -n kube-system pod -l app=terway-eni
```
Run the following command to check whether all pods are recreated:
```
kubectl get pod -n kube-system  | grep terway
```
Create a pod to check whether the pod is assigned an IP address from the newly added vSwitch.
Note After you modify the configurations of the vSwitch, the configurations apply only to newly created ENIs. Existing ENIs use previous configurations. To make the configurations take effect, you can restart the node.

If exceptions occur in the preceding steps, Submit a ticket.

FAQ

Why is the cluster unable to access the Internet after a new vSwitch is created in the Terway network?

Symptom: In the Terway network, after a new vSwitch is created to provide more IP addresses for pods, the cluster cannot access the Internet.

Cause: The new vSwitch that assigns IP addresses to pods does not have access to the Internet.

Solution: You can create a NAT gateway and configure SNAT rules to enable the new vSwitch to access the Internet. For more information, see Enable an existing ACK cluster to access the Internet.

What do I do if the IP address of a newly created pod does not fall within the vSwitch CIDR block?

Symptom: In Terway mode, the IP address of a newly created pod does not fall within the vSwitch CIDR block.

Cause: After the pod is created, the ENI of the ECS instance assigns an IP address from the VPC CIDR block to the pod. You can configure vSwitches only if a newly created ENI is attached to the node. If an ENI is attached to the node before you add the node to the cluster or modify vSwitches used by Terway, the ENI assigns IP addresses to newly created pods on the node from the vSwitch to which the ENI belongs. This issue may occur in the following scenarios:

You add a node that is removed from another cluster to your cluster. The node was not drained when it was removed. In this case, the node uses the ENI that was attached by the cluster to which the node previously belonged.
You manually add or modify vSwitches used by Terway. However, a node may still be attached to the original ENI. In this case, the ENI assigns IP addresses to newly created pods on the node from the vSwitch to which the ENI belongs.

Solution: Create new nodes or create pods on other nodes.