Accelerate pod IP allocation in Terway mode - Container Service for Kubernetes

Terway manages the elastic network interfaces (ENIs) of Elastic Compute Service (ECS) instances and assigns IP addresses to pods. Due to the limits of ENIs, new containers cannot be launched within seconds if they use newly created ENIs. Terway optimizes ENI management to meet requirements such as fast IP allocation and IP revocation. You can use Terway to meet these requirements without additional configurations. If you require fast IP allocation, refer to the suggestions in this topic to adjust the watermark and API throlling threshold of the IP address pool.

CNI execution time

Creating a pod often takes a long time, because it involves the creation of components, such as kubelet, the runtime (containerd), and the CNI plug-in. After these components complete their respective tasks, kubelet updates the IP address and running status of pods to the kube-apiserver. The following figure illustrates the process:

For more information about how to optimize the kubelet configuration, see Customize the kubelet parameters of a node pool.
For more information about how to accelerate storage mounting, see Enable disk parallel mounting.

ENI management model

Before you use Terway, we recommend that you learn about how Terway manages ENIs. The following figure shows the ENIs of a node and how Terway manages ENIs.

An ENI is a resource provided by ECS and has multiple IP addresses. ENIs are classified into primary ENIs and secondary ENIs. Terway does not manage the primary ENI of a node. Terway dynamically adjusts the number of ENIs and the number of IP addresses of ENIs based on the number of pods.
IP address pool: Terway uses IP address pools to manage IP addresses. After ENIs are assigned from ECS instances, the IP addresses of the ENIs are added to the IP address pool.
- When a pod is created, an IP address is preferentially allocated from the address pool. This process does not require API operations, which improves pod creation efficiency.
- When a pod is deleted, the IP address of the pod is preferentially reclaimed to the IP address pool for subsequent use. When the number of idle IP addresses exceeds the specified threshold, an API operation is called to return the IP addresses.

Default configurations

By default, IP address prefetching is disabled and five idle IP addresses are cached. The default configurations provide an adequate number of cached IP addresses to meet the requirements for fast IP allocation. The default configurations are suitable for most scenarios.

Optimization suggestions

Terway allows you to adjust the watermark for the IP address pool and set the API throttling threshold. These configurations can be tailored to meet your specific business requirements. For more information, see Customize the eni_conf parameters in the Terway ConfigMap.

Parameter	Type	Description	Adjustable	Configurable in the ACK console
`max_pool_size`	int	The maximum number of idle IP addresses that can be retained in the IP address pool. Default value: `5`.	Yes	Yes
`min_pool_size`	int	The minimum number of idle IP addresses that must be retained in the IP address pool. Default value: `0`. The value of `min_pool_size` cannot be greater than the value of `max_pool_size`.	Yes	Yes
`kube_client_qps`	float32	The queries per second (QPS) setting of the Kubernetes client. This parameter must be configured together with `kube_client_burst`.	Yes	No
`kube_client_burst`	int	The burst setting of the Kubernetes client. This parameter must be configured together with `kube_client_qps` and must be greater than the value of `kube_client_qps`.	Yes	No

Important

If you decrease the value of max_pool_size and increase the value of min_pool_size, Terway modifies the watermark of the IP address pool until the configurations meet your business requirements. However, this may increase the number of API calls, which may trigger API throttling if the number of API calls reaches the quota limit. Proceed with caution.

For more information about API quotas, see Throttling and quota management. For more information about the API operations that are used by Terway, see ACK role AliyunCSManagedNetworkRole.

Reduce IP waste

If your cluster uses a relatively small CIDR block and contains more than 2,000 nodes, the default number of idle IP addresses is 10,000.

In this case, we recommend that you decrease the value of max_pool_size to reduce the number of idle IP addresses.

Accelerate pod IP allocation

Increase the value of max_pool_size to retain more idle IP addresses on the node. This accelerates pod IP allocation. In this case, Terway does not cache IP addresses to nodes.
Increase the value of min_pool_size to enable IP address prefetching and cache idle IP addresses to the node.
Modify the throttling limits of kubelet, such as kubeAPIQPS and kubeAPIBurst. For more information, see Customize the kubelet parameters of a node pool.
If you have strict requirements for the IP allocation speed, we recommend that you configure the pod to use the host network to bypass the CNI plug-in during IP allocation.

Resolve traffic throttling in Datapath V2 mode

In Datapath V2 or IPVLAN mode, when you create a large number of pods in parallel on one node, pod creation is throttled on the agent side. The default creation rate limit is 0.5 per second. In this case, an event that contains the following information is generated. Example:

unable to create endpoint: [PUT /endpoint/{id}][429]

You can modify the rate limit on the node to address this issue.

You must connect to the ACK cluster by using kubectl first. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

Warning

Using kubectl to configure Terway is a high-risk operation because kubectl cannot verify the input content. Incorrect input parameters may cause errors such as Terway suspension. Make sure that you understand the configuration before you make changes.
If you increase the rate limit, the CPU and memory usage of Terway increases, which may increase the loads of kube-apiserver.

Run the following command to configure advanced parameters:
```
kubectl edit cm -n kube-system  eni-config 
```

In the 10-terway.conf section, add the cilium_args field. The following code block is an example, which sets the creation rate limit to 2 per second and the maximum burst creation rate limit to 5 per second.

10-terway.conf: |
    {
        "cniVersion": "0.4.0",
        "name": "terway",
        "capabilities": {"bandwidth": true},
        "eniip_virtual_type": "IPvlan",
        "cilium_args": "--api-rate-limit=endpoint-create=rate-limit:2/s,rate-burst:5",
        "type": "terway"
    }

After the configuration is complete, save the file and exit. Run the following command to update the configuration:
```
 kubectl rollout restart -n kube-system daemonset.apps/terway-eniip
```