If you have Kubernetes clusters in your self-managed data centers or other offline environments, you must deploy virtual nodes (Virtual Kubelet) in the clusters before you can use Elastic Container Instance in the clusters. This topic describes how to connect offline self-managed clusters in to Elastic Container Instance, including how to register self-managed clusters in the Container Service for Kubernetes (ACK) console and deploy virtual nodes, or deploy Virtual Kubelet in self-managed clusters.
An Alibaba Cloud account is created. For more information, see Sign up with Alibaba Cloud.
Elastic Container Instance is seamlessly connected to Kubernetes by using virtual nodes based on Virtual Kubelet provided by the Kubernetes community. In this scenario, Kubernetes clusters can obtain high elasticity without being limited by the computing capacities of cluster nodes. If you have offline self-managed Kubernetes clusters, you can use one of the following methods to use Elastic Container Instance:
We recommend that you use Method 1. This method allows you to upgrade the Virtual Kubelet version (the ack-virtual-node version) to use new features.
The following limits apply to hybrid cloud environments:
Annotations update is not supported.
Labels update is not supported.
Spec.ActiveDeadlineSeconds update is not supported.
ConfigMap and Secret update is not supported.
The kubectl logs-f command is not supported. The kubectl logs command is supported.
The kubectl attach command is not supported.
The kubectl port-forward command is not supported.
Some parameters of Downward API such as status.hostIP are unavailable.
The following table describes the parameters that you must obtain before you can deploy Virtual Kubelet.
ECI_ACCESS_KEY and ECI_SECRET_KEY
The AccessKey ID and corresponding AccessKey secret. They serve as the credentials to manage Elastic Container Instance in virtual nodes.
For more information, see Obtain an AccessKey pair.
The ID of the cluster, which is the unique identifier of the cluster.
If you register a cluster, the cluster ID is generated by the system. If you deploy Virtual Kubelet in a cluster, you must customize the cluster ID.
The ID of the region. The elastic container instance is deployed within this region.
The ID of the virtual private cloud (VPC). The elastic container instance is deployed within this VPC.
You can create and view VPCs on the VPCs page in the VPC console.
The ID of the vSwitch. The elastic container instance is associate with this vSwitch.
You can create and view vSwitches on the vSwitch page in the VPC console and select the corresponding vSwitch based on the selected VPC.
The ID of the security group. The elastic container instance is added to this security group.
You can create and view security groups on the Security Groups page in the ECS console and select the corresponding vSwitch based on the selected VPC.
Register self-managed clusters and deploy virtual nodes
You can register self-managed clusters in the ACK console and then deploy virtual nodes to use Elastic Container Instance. Perform the following steps:
Log on to the ACK console.
In the left-side navigation pane, click Clusters.
Create a registered cluster.
On the Clusters page, click Create Kubernetes Cluster.
Click the Register Cluster tab. Specify the parameters for the cluster and click Create Cluster.
The following table describes the parameters of which you must take note. For more information, see Register an external Kubernetes cluster.
Region, VPC, and VSwitch
Select the required region, VPC, and vSwitch.
Access to API Server
By default, an internal-facing Server Load Balancer (SLB) instance is created for the API server. You can select the specifications of the SLB instance to suit your needs.
Specify whether to create and associate an elastic IP address (EIP) to connect to the cluster.
Automatically create a security group to divide security domains and control network traffic.
Specify whether to activate Log Service to collect log data from containers.
Specify whether to enable deletion protection for the cluster. If you enable deletion protection, the cluster cannot be deleted by using the ACK console or by calling API operations.
Register the cluster.
On the Clusters page, find the cluster that you created and click the cluster name.
On the Cluster Information page, click the Connection Information tab.
Create a ConfigMap in the cluster.
Click the Public Network or Internal Network tab based on your network. Copy the content to a YAML configuration file such as agent.yaml and run the
kubectl apply -f agent.yamlcommand in the cluster to create a ConfigMap.
Run the following command in the cluster to check the connection status:
kubectl -n kube-system get pod |grep ack-cluster-agent
The following command output is returned:
ack-cluster-agent-5f7d568f6-6fc4k 1/1 Running 0 9s ack-cluster-agent-5f7d568f6-tf6fp 1/1 Running 0 9s
Deploy virtual nodes.
In the left-side navigation pane, choose Marketplace > App Catalog.
On the Alibaba Cloud Apps tab, find ack-virtual-node and click it.
Set the parameters and select the created cluster for installation.
On the Parameters tab, enter the vSwitch ID, security group ID, AccessKey ID, and AccessKey secret that you obtained.Note
If you register the cluster in the Internet in Step 4, remove vpc from the value of the repository parameter. Example: registry.cn-hangzhou.aliyuncs.com/acs/virtual-nodes-eci.
For more information, see Add a virtual node to an external cluster.
Deploy Virtual Kubelet
You can deploy Virtual Kubelet in self-managed clusters to use Elastic Container Instance. You must obtain the latest Virtual Kubelet version before you can deploy Virtual Kubelet. For more information, see ack-virtual-node.
Perform the following steps:
Create a service account for Virtual Kubelet and bind a cluster role to the account to create pods.
Run the following command to create a service account named vk-admin:
kubectl create serviceaccount vk-admin -n kube-system
Run the following command to bind the cluster-admin cluster role to the vk-admin account:
kubectl create clusterrolebinding vk-admin-binding --clusterrole=cluster-admin --serviceaccount=kube-system:vk-admin
Prepare the vk.yaml configuration file required to deploy Virtual Kubelet.
The following code provides an example of the YAML configuration file content. You must replace the parameter values with your own information.Notice
The version of Virtual Kubelet must be v188.8.131.528-0b919e1d2-aliyun or later.
apiVersion: apps/v1 kind: Deployment metadata: labels: app: virtual-node-controller name: virtual-node-controller namespace: kube-system spec: replicas: 1 selector: matchLabels: app: virtual-node-controller template: metadata: labels: app: virtual-node-controller spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: type operator: NotIn values: - virtual-kubelet podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - virtual-node-controller topologyKey: kubernetes.io/hostname weight: 100 containers: - name: virtual-node-controller # The image and its version required to deploy Virtual Kubelet. image: registry.cn-beijing.aliyuncs.com/acs/virtual-nodes-eci:v184.108.40.206-252556a33-aliyun imagePullPolicy: Always args: - --provider - alibabacloud - --nodename - $(VN_INSTANCE) env: - name: VN_INSTANCE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: KUBELET_PORT value: "10250" - name: VKUBELET_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: VKUBELET_TAINT_KEY value: "virtual-kubelet.io/provider" - name: VKUBELET_TAINT_VALUE value: "alibabacloud" - name: VKUBELET_TAINT_EFFECT value: "NoSchedule" - name: ECI_QUOTA_CPU value: "1000000" - name: ECI_QUOTA_MEMORY value: 6400Ti - name: ECI_QUOTA_POD value: "3000" - name: ECI_KUBE_PROXY value: "true" # The ID of the region in which to create the elastic container instance. - name: ECI_REGION value: <region-id> # The AccessKey ID used to create the elastic container instance. - name: ECI_ACCESS_KEY value: <access-key> # The AccessKey secret used to create the elastic container instance. - name: ECI_SECRET_KEY value: <secret-kty> # The ID of the virtual private cloud (VPC) in which to create the elastic container instance. - name: ECI_VPC value: <vpc-id> # The ID of the vSwitch to associate with the elastic container instance. - name: ECI_VSWITCH value: <vsw-id> # The ID of the security group to which the elastic container instance belongs. - name: ECI_SECURITY_GROUP value: <security-group-id> # The customized ID of the cluster to which the elastic container instance belongs. - name: ALIYUN_CLUSTERID value: <cluster-id> - name: ALIYUN_PRIVATE_ZONE value: "false" # The mode of the hybrid cloud environments that is required to create the elastic container instance. - name: ECI_HYBRID_MODE value: "true" dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler serviceAccount: vk-admin serviceAccountName: vk-admin
Deploy Virtual Kubelet.
kubectl apply -f vk.yaml
View the deployment status.
kubectl get deploy/virtual-node-controller -n kube-system
The following code provides an example of the command output if Virtual Kubelet is deployed:
NAME READY UP-TO-DATE AVAILABLE AGE virtual-node-controller 1/1 1 1 161m
View the node information after Virtual Kubelet is deployed.
kubectl get nodes
After Virtual Kubelet is deployed, the generated virtual node is named virtual-kubelet. If the virtual-kubelet node is displayed in the Ready state, Virtual Kubelet is deployed. Example:
NAME STATUS ROLES AGE VERSION master-1 Ready <none> 19d v1.18.8-aliyun.1 master-2 Ready <none> 19d v1.18.8-aliyun.1 virtual-kubelet Ready agent 18d v1.18.8-aliyun.1
If you want to use new features, you may need to upgrade the Virtual Kubelet version. You can run the
kubectl edit deployment -n kube-system virtual-node-controller command to edit resources and modify the image tag to the desired version.
Schedule pods to the virtual nodes
When a cluster contains virtual nodes, you can schedule pods to the virtual nodes to use Elastic Container Instance to run the pods. Take note of the following items:
Virtual nodes have specific taints. You must configure node selectors and tolerations for a pod before you can schedule the pod to a virtual node. Example:
apiVersion: v1 kind: Pod metadata: name: nginx spec: containers: - image: nginx imagePullPolicy: Always name: nginx nodeSelector: type: virtual-kubelet tolerations: - key: virtual-kubelet.io/provider operator: Exists
DaemonSets cannot be deployed because Elastic Container Instance is connected to Kubernetes clusters by using Virtual Kubelet. Virtual nodes are used, instead of real compute nodes. When you create a DaemonSet, you must configure the anti-affinity scheduling policy to prevent the DaemonSet from being scheduled to a virtual node. You must add the following information to the spec.template.spec.affinity field of the DaemonSet to be deployed:
affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: type operator: NotIn values: - virtual-kubelet