This topic describes virtual nodes and introduces how to deploy a virtual node and schedule a pod to a virtual node.

Virtual nodes and ECI

Elastic Container Instance (ECI) is a container-oriented serverless computing service. It provides maintenance-free container runtimes that support strong isolation and quick startup. ECI enables you to focus on container-based applications without the need to purchase or manage ECS instances, saving you the hassle of infrastructure maintenance. You can create ECIs based on the actual needs. Fees are charged based on the resource usage during container execution time.

Based on Virtual Kubelet, virtual nodes enable seamless integration between Kubernetes and ECI and drastically enhance the elasticity of Kubernetes clusters by eliminating the constraint of computing power provided by cluster nodes. For more information about how Virtual Kubelet works and its architecture, see Virtual Kubelet.

Virtual nodes allow you to dynamically create pods based on the actual needs and provide increased flexibility without the trouble of cluster capacity planning. You can use virtual nodes in various scenarios to significantly lower computing costs and improve computing efficiency.
  • Online businesses with periodic traffic patterns, such as online education and e-commerce websites. Virtual nodes can remarkably reduce computing costs by optimizing resource pool maintenance.
  • Virtual nodes can effectively lower costs in computing scenarios where Spark or Presto is used for data processing.
  • CI/CD Pipeline: Jenkins and Gitlab-Runner.
  • Jobs: Scheduled jobs and AI jobs.
Based on virtual nodes and ECI, Container Service provides multiple serverless container products, such as serverless Kubernetes (ASK) and ACK on ECI, which meet your needs in various elastic computing scenarios.virtual node

Install the ack-virtual-node add-on

Note In serverless Kubernetes clusters, you can directly create ECIs without the need to install the ack-virtual-node add-on. In managed or dedicated Kubernetes clusters, you must install the ack-virtual-node add-on before you can create ECIs.
Prerequisites
  • You have created a managed or dedicated Kubernetes cluster. For more information, see Create a managed Kubernetes cluster.
  • You have activated Elastic Container Instance. To activate the service, go to the ECI console.
  • The region where the cluster is deployed must be supported by ECI. To view supported regions and zones, go to the ECI console.
  1. Log on to the Container Service console.
  2. In the left-side navigation pane, choose Marketplace > App Catalog. On the page that appears, click ack-virtual-node.
  3. On the App Catalog - ack-virtual-node page, click the Parameters tab and set the parameters.
    Parameter Description How to obtain the value
    ECI_REGION The name of the region. Click the target cluster name on the Clusters page. On the Basic Information page that appears, copy the value of Region in the Basic Information section.
    Note For example, cn-hangzhou represents the China (Hangzhou) region.
    ECI_VPC The VPC network of the cluster. Click the target cluster name on the Clusters page. On the Basic Information page that appears, copy the value of VPC in the Cluster Resources section.
    ECI_VSWITCH VSwitches. Click a node ID on the Nodes page. On the Instance Details page that appears, copy the value of VSwitch in the Configuration Information section.
    Note

    Make sure that the zone of the VSwitch is supported by ECI.

    VSwitches support multiple zones. You can specify multiple VSwitches in the format of ECI_VSWITCH: "vsw-xxxxxxx1, vsw-xxxxxxx2, vsw-xxxxxxx3".

    ECI_SECURITY_GROUP The security group ID. Click a node ID on the Nodes page. Click Security Groups in the left-side navigation pane. Copy the value of Security Group ID on the Security Groups tab.
    ECI_ACCESS_KEY Your AccessKey ID. For more information, see How can I obtain an AccessKey pair?.
    ECI_SECRET_KEY Your AccessKey secret. For more information, see How can I obtain an AccessKey pair?.
    ALIYUN_CLUSTERID The ID of the target cluster. Click the target cluster name on the Clusters page. On the Basic Information page that appears, copy the value of Cluster ID in the Basic Information section.
  4. On the Deploy page at the right, select the target cluster and verify that Namespace is set to kube-system and Release Name is set to ack-virtual-node. Then click Create.Create an add-on
  5. In the left-side navigation pane, choose Clusters > Nodes. On the Nodes page that appears, virtual node virtual-node-eci is now displayed.Add a node
  6. Run the following commands to query the statuses of virtual-node-controller and virtual-node-admission-controller. For more information, see Use kubectl on Cloud Shell to manage Kubernetes clusters. 
    # kubectl -n kube-system get statefulset virtual-node-eci
NAME              READY   AGE
virtual-node-eci  1/1     1m

# kubectl -n kube-system get deploy ack-virtual-node-affinity-admission-controller
NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE
ack-virtual-node-affinity-admission-controller   1/1     1            1           1m

# kubectl -n kube-system get pod|grep virtual-node-eci
virtual-node-eci-0                                                1/1     Running   0          1m

# kubectl get no|grep virtual-node-eci
virtual-node-eci-0        Ready    agent    1m   v1.11.2-aliyun-1.0.207

Schedule a pod to a virtual node

Note The following operations are not applicable to serverless Kubernetes clusters.
When you schedule a pod to a virtual node, Virtual Node Controller creates a corresponding pod on the virtual node. You can use the following methods to schedule a pod to a virtual node.
  • Set nodeSelector and tolerations.
    Virtual nodes have specific taints. You must set nodeSelector and tolerations for a pod before you can schedule the pod to a virtual node. The sample template is as follows: 
    apiVersion: v1
kind: Pod metadata:  name: nginx spec:
  containers:
  - image: nginx
    imagePullPolicy: Always
    name: nginx
  nodeSelector:
    type: virtual-kubelet
  tolerations:
  - key: virtual-kubelet.io/provider
    operator: Exists
  • Add a label to the pod .
    The virtual-node-affinity-admission-controller webhook automatically schedules correctly labeled pods to a virtual node. The following example uses label eci=true. The sample template is as follows: 
    # kubectl run nginx --image nginx -l eci=true

# kubectl get pod -o wide|grep virtual-node-eci
nginx-7fc9f746b6-r4xgx     0/1     ContainerCreating   0          20s   192.168.1.38   virtual-node-eci-0        <none>           <none>
  • Add a label to the namespace.
    The virtual-node-affinity-admission-controller webhook automatically schedules pods in a correctly labeled namespace to a virtual node. The following example uses label virtual-node-affinity-injection=enabled. The sample template is as follows: 
    # kubectl create ns vk
# kubectl label namespace vk virtual-node-affinity-injection=enabled
# kubectl -n vk run nginx --image nginx

# kubectl -n vk get pod -o wide|grep virtual-node-eci
nginx-6f489b847d-vgj4d      1/1     Running             0          1m   192.168.1.37   virtual-node-eci-0        <none>           <none>

Modify the configurations of the virtual node controller

Note The following operations are not applicable to serverless Kubernetes clusters.

The configurations of the virtual node controller determine how pods are scheduled to the node and pod runtime environment such as VSwitch and security group settings. You can modify the configurations of the controller based on your needs. New configurations apply to new pods only, and do not affect existing pods running on the node.

You can run the following command to modify the configurations of the controller: 
# kubectl -n kube-system edit statefulset virtual-node-eci
Common modifications are as follows:
  • Upgrade the virtual node controller version.

    To use the latest features of virtual nodes, you need to upgrade the virtual node controller to the latest version. For example, to enable pods to access ClusterIP services, the virtual node controller version must be later than v1.0.0.2-aliyun.

  • Modify security group (ECI_SECURITY_GROUP) settings.

    You can modify this environment variable to change the security group associated with pods.

  • Modify VSwitch (ECI_VSWITCH) settings.

    You can modify this environment variable to change the VSwitch where pods belong. We recommend that you configure multiple VSwitches across zones to ensure high availability. When ECI resources are insufficient in one zone, the controller can create pods in another zone.

  • Modify kube-proxy settings (ECI_KUBE_PROXY).

    By default, this environment variable is set to true, indicating that pods can access ClusterIP services. If pods no longer need to access ClusterIP services, you can set the environment variable to false to disable kube-proxy. In large scale scenarios, for example, the cluster needs to start a large number of pods, which dramatically increases the number of concurrent connections between kube-proxy and Kubernetes API server, you can disable kube-proxy to lift pressure on the API server. Instead, you can configure PrivateZone to enable pods to access services in the cluster.

  • Create multiple virtual nodes.
    We recommend that you deploy a maximum of 3,000 pods on a virtual node. To create more pods, you can create more virtual nodes. You can modify the number of replicas in the statefulset configuration to create more virtual nodes. The number of replicas represents that of virtual nodes in the cluster. Each virtual node corresponds to a virtual node controller, which manages the pods on the virtual node. The controllers do not interfere with each other. A sample configuration is as follows: 
    # kubectl -n kube-system scale statefulset virtual-node-eci --replicas=4
statefulset.apps/virtual-node-eci scaled

# kubectl get no
NAME                            STATUS     ROLES    AGE     VERSION
cn-hangzhou.192.168.1.1         Ready      <none>   63d     v1.12.6-aliyun.1
cn-hangzhou.192.168.1.2         Ready      <none>   63d     v1.12.6-aliyun.1
virtual-node-eci-0              Ready      agent     6m     v1.11.2-aliyun-1.0.207
virtual-node-eci-1              Ready      agent     1m     v1.11.2-aliyun-1.0.207
virtual-node-eci-2              Ready      agent     1m     v1.11.2-aliyun-1.0.207
virtual-node-eci-3              Ready      agent     1m     v1.11.2-aliyun-1.0.207

Delete a virtual node

Note The following operations are not applicable to serverless Kubernetes clusters.
Generally, you do not need to delete virtual nodes because they do not occupy computing resources. If you want to delete a virtual node, we recommend that you manually evict or delete the pods running on the node, and then delete the virtual node controller and the node. If you delete the virtual node controller before pods are deleted, some pods may be left over. 
# kubectl drain virtual-node-eci-0 ...

# kubectl -n kube-system delete statefulset virtual-node-eci

# kubectl delete no virtual-node-eci-0 ...