This topic describes virtual nodes and elastic container instances (ECIs). It also describes how to deploy a virtual node add-on in a cluster of Container Service for Kubernetes (ACK). You can schedule a pod to a virtual node by using ack-virtual-node.
Virtual nodes and ECI
ECI is a serverless compute service that is provided by Alibaba Cloud for containerized business. You can use ECI to quickly set up an operations and maintenance (O&M)-free and isolated runtime environment for your application containers. ECI allows you to focus on container-based applications without the need to purchase or manage Elastic Compute Service (ECS) instances. This saves you the hassle of infrastructure maintenance. You can create ECIs based on your business requirements. You are charged for resource usage on a per second basis.
Based on Virtual Kubelet, virtual nodes enable seamless integration between Kubernetes and ECI. This significantly improves the elasticity of Kubernetes clusters and eliminates the constraint of the computing capacity that is provided by cluster nodes. For more information about how Virtual Kubelet works and its architecture, see Virtual Kubelet.
- Online business with periodic traffic patterns, such as online education and e-commerce. Virtual nodes significantly reduce computing costs by optimizing resource pool maintenance.
- Virtual nodes effectively reduce costs in computing scenarios where Spark or Presto is used to process data.
- CI/CD Pipeline: Jenkins and Gitlab-Runner.
- Jobs: Jobs in Artificial Intelligence (AI) scenarios and CronJobs.
Deploy ack-virtual-node in an ACK cluster
- A managed Kubernetes cluster or dedicated Kubernetes cluster is created. For more information, see Create a managed kubernetes cluster.
- ECI is activated. You can log on to the ECI console to activate ECI.
- The region where the created cluster is deployed must be supported by ECI. To view the supported regions and zones, go to the ECI console.
- Log on to the ACK console.
- In the left-side navigation pane, choose .
- On the App Catalog page, click the Alibaba Cloud Apps tab and find and click ack-virtual-node.In the upper-right corner of the App Catalog page, you can enter ack-virtual-node into the Name search bar and click the search icon. You can also enter a keyword to perform a fuzzy match.
- On the App Catalog - ack-virtual-node page, select the created cluster in the Deploy section to deploy ack-virtual-node.Namespace is automatically set to kube-system. Release Name is automatically set to ack-virtual-node.
- On the App Catalog - ack-virtual-node page, click the Parameters tab and set the parameters. Then, click Create in the Deploy section.
Parameter Description How to obtain the value ALIYUN_CLUSTERID The ID of the cluster where you want to deploy ack-virtual-node On the details page of the cluster, click the Basic Information tab. In the Basic Information section, you can obtain the ID of the cluster. ALIYUN_RESOURCEGROUP_ID The ID of the resource group to which the cluster belongs If you do not specify the parameter, the default resource group is used. To specify a resource group, log on to the Resource Management console to obtain the ID of the resource group. ECI_REGION The name of the region where the cluster is deployed On the details page of the cluster, click the Basic Information tab. In the Basic Information section, you can check the region where the cluster is deployed.Note For example, cn-hangzhou indicates the China (Hangzhou) region. ECI_VSWITCH vSwitches On the Nodes page, click the ID of a node. On the instance details page, you can obtain the value of VSwtich in the Configuration Information section.Note
Make sure that the zone of the vSwitch is supported by ECI.
vSwitches support multiple zones. You can specify multiple vSwitches in the format of
ECI_VSWITCH: "vsw-xxxxxxx1, vsw-xxxxxxx2, vsw-xxxxxxx3".
ECI_SECURITY_GROUP The ID of the security group to which nodes in the cluster belong On the Nodes page, click the ID of a node. In the left-side navigation pane, click Security Groups. Click the Security Groups tab to obtain the ID of the security group. ECI_ACCESS_KEY The AccessKey ID of your Alibaba Cloud account For more information, see Obtain an AccessKey pair.
You must attach the AliyunECIReadOnlyAccess policy to your account in the Resource Access Management (RAM) console. For more information, see Grant permissions to a RAM user.
ECI_SECRET_KEY The AccessKey secret of your Alibaba Cloud account For more information, see Obtain an AccessKey pair.
You must attach the AliyunECIReadOnlyAccess policy to your account in the RAM console. For more information, see Grant permissions to a RAM user.
- After ack-virtual-node is deployed, you can view information about the newly added
node virtual-node-eci by performing the following steps:
- In the left-side navigation pane, click Clusters.
- On the Clusters page, click the name of a cluster or click Applications in the Actions column.
- In the left-side navigation pane, click Nodes. On the Nodes page, you can find the newly added node virtual-node-eci.
- Run the following commands to query the deployment result of virtual-node-controller and virtual-node-admission-controller: For more information, see Use kubectl on Cloud Shell to manage ACK clusters.Run the following command:
kubectl -n kube-system get statefulset virtual-node-eciThe following output is returned:
NAME READY AGE virtual-node-eci 1/1 1mRun the following command:
kubectl -n kube-system get deploy ack-virtual-node-affinity-admission-controllerThe following output is returned:
NAME READY UP-TO-DATE AVAILABLE AGE ack-virtual-node-affinity-admission-controller 1/1 1 1 1mRun the following command:
kubectl -n kube-system get pod|grep virtual-node-eciThe following output is returned:
virtual-node-eci-0 1/1 Running 0 1mRun the following command:
kubectl get no|grep virtual-node-eciThe following output is returned:
virtual-node-eci-0 Ready agent 1m v1.11.2-aliyun-1.0.207
Schedule a pod to a virtual node
- Set node selectors and tolerations for the pod.
Virtual nodes have specific taints. You must set node selectors and tolerations for a pod before you can schedule the pod to a virtual node. The following code block is a YAML template that is used to create a pod:
apiVersion: v1 kind: Pod metadata: name: nginx spec: containers: - image: nginx imagePullPolicy: Always name: nginx nodeSelector: type: virtual-kubelet tolerations: - key: virtual-kubelet.io/provider operator: Exists
- Attach labels to the pod .
The virtual-node-affinity-admission-controller webhook automatically schedules pods with specified labels to a virtual node. The eci=true label is used in the following example. Example:Run the following commands:
kubectl run nginx --image nginx -l eci=true
kubectl get pod -o wide|grep virtual-node-eciThe following output is returned:
nginx-7fc9f746b6-r4xgx 0/1 ContainerCreating 0 20s 192.168. *.* virtual-node-eci-0 <none> <none>
- Add a label to the namespace where the pod is deployed.
The virtual-node-affinity-admission-controller webhook automatically schedules pods that are created in a namespace with specified labels to a virtual node. The virtual-node-affinity-injection=enabled label is used in the following example. Example:Run the following commands:
kubectl create ns vk
kubectl label namespace vk virtual-node-affinity-injection=enabled
kubectl -n vk run nginx --image nginx
kubectl -n vk get pod -o wide|grep virtual-node-eciThe following output is returned:
nginx-6f489b847d-vgj4d 1/1 Running 0 1m 192.168. *.* virtual-node-eci-0 <none> <none>
Modify the configurations of the virtual node controller
The configurations of the virtual node controller determine how pods are scheduled to a virtual node and specify the pod runtime environment such as vSwitches and security group settings. You can modify the configurations of the controller based on your business requirements. Modified configurations apply to only pods that are scheduled after modifications and do not apply to existing pods that run on the node.
kubectl -n kube-system edit statefulset virtual-node-eci
- Upgrade the virtual node controller version.
To use the latest features of virtual nodes, you must upgrade the virtual node controller to the latest version. For example, to enable pods to access ClusterIP type Services, the virtual node controller version must be later than v22.214.171.124-aliyun.
- Modify security group settings.
You can modify the ECI_SECURITY_GROUP environment variable to change the security group that is associated with pods that are scheduled to the virtual node.
- Modify vSwitch settings.
You can modify the ECI_VSWITCH environment variable to change the vSwitch to which pods that are scheduled to the virtual node belong. We recommend that you configure multiple vSwitches that are deployed in different zones to ensure high availability. When ECI resources are insufficient in a zone, the controller can create pods in another zone.
- Modify kube-proxy settings.
By default, the ECI_KUBE_PROXY environment variable is set to true. This indicates that pods can access ClusterIP type Services. If the pods no longer need to access ClusterIP type Services, you can set the environment variable to false to disable kube-proxy. In some large-scale scenarios, a cluster may need to start a large number of pods. This significantly increases the number of concurrent connections between kube-proxy and the Kubernetes API server. In this case, you can disable kube-proxy to reduce the heavy loads on the API server. However, you can also configure PrivateZone to enable pods to access Services in the cluster.
- Create multiple virtual nodes.
We recommend that you deploy up to 3,000 pods on a virtual node. To create more pods, you can create more virtual nodes. You can modify the number of replicas in the StatefulSet configurations to create more virtual nodes. The number of replicas represents the number of virtual nodes in the cluster. Each virtual node is related to a virtual node controller that manages the pods on the virtual node. The controllers do not interfere with each other. Perform the following steps to modify the StatefulSet configurations:Run the following command:
kubectl -n kube-system scale statefulset virtual-node-eci --replicas=4The following output is returned:
statefulset.apps/virtual-node-eci scaledRun the following command:
kubectl get noThe following output is returned:
NAME STATUS ROLES AGE VERSION cn-hangzhou.192.168.1.1 Ready <none> 63d v1.12.6-aliyun.1 cn-hangzhou.192.168.1.2 Ready <none> 63d v1.12.6-aliyun.1 virtual-node-eci-0 Ready agent 6m v1.11.2-aliyun-1.0.207 virtual-node-eci-1 Ready agent 1m v1.11.2-aliyun-1.0.207 virtual-node-eci-2 Ready agent 1m v1.11.2-aliyun-1.0.207 virtual-node-eci-3 Ready agent 1m v1.11.2-aliyun-1.0.207
Delete a virtual node
In most cases, you do not need to delete virtual nodes because they do not occupy the computing resources of the cluster. To delete a virtual node, we recommend that you manually evict or delete the pods that run on the node before you delete the virtual node controller and the node. If you delete the virtual node controller before pods are deleted, some ECIs may fail to be deleted.
kubectl drain virtual-node-eci-0 ...
kubectl -n kube-system delete statefulset virtual-node-eci
kubectl delete no virtual-node-eci-0 ...