All Products
Search
Document Center

Use Elastic Container Instance in offline Kubernetes clusters

Last Updated: Jun 01, 2021

If you have Kubernetes clusters in your self-managed data centers or other offline environments, you must deploy virtual nodes (Virtual Kubelet) in the clusters before you can use Elastic Container Instance in the clusters. This topic describes how to connect offline self-managed clusters in to Elastic Container Instance, including how to register self-managed clusters in the Container Service for Kubernetes (ACK) console and deploy virtual nodes, or deploy Virtual Kubelet in self-managed clusters.

Prerequisites

An Alibaba Cloud account is created. For more information, see Sign up with Alibaba Cloud.

Background information

Elastic Container Instance is seamlessly connected to Kubernetes by using virtual nodes based on Virtual Kubelet provided by the Kubernetes community. In this scenario, Kubernetes clusters can obtain high elasticity without being limited by the computing capacities of cluster nodes. If you have offline self-managed Kubernetes clusters, you can use one of the following methods to use Elastic Container Instance:

Note

We recommend that you use Method 1. This method allows you to upgrade the Virtual Kubelet version (the ack-virtual-node version) to use new features.

Limits

The following limits apply to hybrid cloud environments:

  • Annotations update is not supported.

  • Labels update is not supported.

  • Spec.ActiveDeadlineSeconds update is not supported.

  • ConfigMap and Secret update is not supported.

  • The kubectl logs-f command is not supported. The kubectl logs command is supported.

  • The kubectl attach command is not supported.

  • The kubectl port-forward command is not supported.

  • Some parameters of Downward API such as status.hostIP are unavailable.

  • ClusterIP

    You must establish connections between offline clusters and Alibaba Cloud networks by using Express Connect or Smart Access Gateway. For more information, see Express Connect or Smart Access Gateway.

Preparations

The following table describes the parameters that you must obtain before you can deploy Virtual Kubelet.

Parameter

Description

Obtainment method

ECI_ACCESS_KEY and ECI_SECRET_KEY

The AccessKey ID and corresponding AccessKey secret. They serve as the credentials to manage Elastic Container Instance in virtual nodes.

For more information, see Obtain an AccessKey pair.

ALIYUN_CLUSTERID

The ID of the cluster, which is the unique identifier of the cluster.

If you register a cluster, the cluster ID is generated by the system. If you deploy Virtual Kubelet in a cluster, you must customize the cluster ID.

ECI_REGION

The ID of the region. The elastic container instance is deployed within this region.

You can query supported regions by using the Elastic Container Instance console or by calling the DescribeRegions operation.

ECI_VPC

The ID of the virtual private cloud (VPC). The elastic container instance is deployed within this VPC.

You can create and view VPCs on the VPCs page in the VPC console.

ECI_VSWITCH

The ID of the vSwitch. The elastic container instance is associate with this vSwitch.

You can create and view vSwitches on the vSwitch page in the VPC console and select the corresponding vSwitch based on the selected VPC.

ECI_SECURITY_GROUP

The ID of the security group. The elastic container instance is added to this security group.

You can create and view security groups on the Security Groups page in the ECS console and select the corresponding vSwitch based on the selected VPC.

Register self-managed clusters and deploy virtual nodes

You can register self-managed clusters in the ACK console and then deploy virtual nodes to use Elastic Container Instance. Perform the following steps:

  1. Log on to the ACK console.

  2. In the left-side navigation pane, click Clusters.

  3. Create a registered cluster.

    1. On the Clusters page, click Create Kubernetes Cluster.

    2. Click the Register Cluster tab. Specify the parameters for the cluster and click Create Cluster.

      The following table describes the parameters of which you must take note. For more information, see Register an external Kubernetes cluster.

      Parameter

      Description

      Region, VPC, and VSwitch

      Select the required region, VPC, and vSwitch.

      Access to API Server

      By default, an internal-facing Server Load Balancer (SLB) instance is created for the API server. You can select the specifications of the SLB instance to suit your needs.

      EIP

      Specify whether to create and associate an elastic IP address (EIP) to connect to the cluster.

      Security Group

      Automatically create a security group to divide security domains and control network traffic.

      Log Service

      Specify whether to activate Log Service to collect log data from containers.

      Deletion Protection

      Specify whether to enable deletion protection for the cluster. If you enable deletion protection, the cluster cannot be deleted by using the ACK console or by calling API operations.

  4. Register the cluster.

    1. On the Clusters page, find the cluster that you created and click the cluster name.

    2. On the Cluster Information page, click the Connection Information tab.

    3. Create a ConfigMap in the cluster.

      Click the Public Network or Internal Network tab based on your network. Copy the content to a YAML configuration file such as agent.yaml and run the kubectl apply -f agent.yaml command in the cluster to create a ConfigMap.

    4. Run the following command in the cluster to check the connection status:

      kubectl -n kube-system get pod |grep ack-cluster-agent

      The following command output is returned:

      ack-cluster-agent-5f7d568f6-6fc4k              1/1     Running   0          9s
      ack-cluster-agent-5f7d568f6-tf6fp              1/1     Running   0          9s
  5. Deploy virtual nodes.

    1. In the left-side navigation pane, choose Marketplace > App Catalog.

    2. On the Alibaba Cloud Apps tab, find ack-virtual-node and click it.

    3. Set the parameters and select the created cluster for installation.

      On the Parameters tab, enter the vSwitch ID, security group ID, AccessKey ID, and AccessKey secret that you obtained.

      Note

      If you register the cluster in the Internet in Step 4, remove vpc from the value of the repository parameter. Example: registry.cn-hangzhou.aliyuncs.com/acs/virtual-nodes-eci.

      For more information, see Add a virtual node to an external cluster.

Deploy Virtual Kubelet

You can deploy Virtual Kubelet in self-managed clusters to use Elastic Container Instance. You must obtain the latest Virtual Kubelet version before you can deploy Virtual Kubelet. For more information, see ack-virtual-node.

Perform the following steps:

  1. Create a service account for Virtual Kubelet and bind a cluster role to the account to create pods.

    1. Run the following command to create a service account named vk-admin:

      kubectl create serviceaccount vk-admin -n kube-system
    2. Run the following command to bind the cluster-admin cluster role to the vk-admin account:

      kubectl create clusterrolebinding vk-admin-binding --clusterrole=cluster-admin --serviceaccount=kube-system:vk-admin
  2. Prepare the vk.yaml configuration file required to deploy Virtual Kubelet.

    The following code provides an example of the YAML configuration file content. You must replace the parameter values with your own information.

    Notice

    The version of Virtual Kubelet must be v2.0.0.608-0b919e1d2-aliyun or later.

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: virtual-node-controller
      name: virtual-node-controller
      namespace: kube-system
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: virtual-node-controller
      template:
        metadata:
          labels:
            app: virtual-node-controller
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: type
                    operator: NotIn
                    values:
                    - virtual-kubelet
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchExpressions:
                    - key: app
                      operator: In
                      values:
                      - virtual-node-controller
                  topologyKey: kubernetes.io/hostname
                weight: 100
          containers:
          - name: virtual-node-controller
            # The image and its version required to deploy Virtual Kubelet.
            image: registry.cn-beijing.aliyuncs.com/acs/virtual-nodes-eci:v2.0.0.34-252556a33-aliyun
            imagePullPolicy: Always
            args:
            - --provider
            - alibabacloud
            - --nodename
            - $(VN_INSTANCE)
            env:
            - name: VN_INSTANCE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: KUBELET_PORT
              value: "10250"
            - name: VKUBELET_POD_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
            - name: VKUBELET_TAINT_KEY
              value: "virtual-kubelet.io/provider"
            - name: VKUBELET_TAINT_VALUE
              value: "alibabacloud"
            - name: VKUBELET_TAINT_EFFECT
              value: "NoSchedule"
            - name: ECI_QUOTA_CPU
              value: "1000000"
            - name: ECI_QUOTA_MEMORY
              value: 6400Ti
            - name: ECI_QUOTA_POD
              value: "3000"
            - name: ECI_KUBE_PROXY
              value: "true"
            # The ID of the region in which to create the elastic container instance.
            - name: ECI_REGION
              value: <region-id>
            # The AccessKey ID used to create the elastic container instance.
            - name: ECI_ACCESS_KEY
              value: <access-key>
            # The AccessKey secret used to create the elastic container instance.
            - name: ECI_SECRET_KEY
              value: <secret-kty>
            # The ID of the virtual private cloud (VPC) in which to create the elastic container instance.
            - name: ECI_VPC
              value: <vpc-id>
            # The ID of the vSwitch to associate with the elastic container instance.
            - name: ECI_VSWITCH
              value: <vsw-id>
            # The ID of the security group to which the elastic container instance belongs.
            - name: ECI_SECURITY_GROUP
              value: <security-group-id>
            # The customized ID of the cluster to which the elastic container instance belongs.
            - name: ALIYUN_CLUSTERID
              value: <cluster-id>
            - name: ALIYUN_PRIVATE_ZONE
              value: "false"
            # The mode of the hybrid cloud environments that is required to create the elastic container instance.
            - name: ECI_HYBRID_MODE
              value: "true"
          dnsPolicy: ClusterFirst
          restartPolicy: Always
          schedulerName: default-scheduler
          serviceAccount: vk-admin
          serviceAccountName: vk-admin
  3. Deploy Virtual Kubelet.

    kubectl apply -f vk.yaml
  4. View the deployment status.

    kubectl get deploy/virtual-node-controller -n kube-system

    The following code provides an example of the command output if Virtual Kubelet is deployed:

    NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
    virtual-node-controller            1/1         1                      1                  161m
  5. View the node information after Virtual Kubelet is deployed.

    kubectl get nodes

    After Virtual Kubelet is deployed, the generated virtual node is named virtual-kubelet. If the virtual-kubelet node is displayed in the Ready state, Virtual Kubelet is deployed. Example:

    NAME                STATUS   ROLES    AGE   VERSION
    master-1            Ready    <none>   19d   v1.18.8-aliyun.1
    master-2           Ready    <none>   19d   v1.18.8-aliyun.1
    virtual-kubelet    Ready    agent     18d   v1.18.8-aliyun.1
Note

If you want to use new features, you may need to upgrade the Virtual Kubelet version. You can run the kubectl edit deployment -n kube-system virtual-node-controller command to edit resources and modify the image tag to the desired version.

Schedule pods to the virtual nodes

When a cluster contains virtual nodes, you can schedule pods to the virtual nodes to use Elastic Container Instance to run the pods. Take note of the following items:

  • Virtual nodes have specific taints. You must configure node selectors and tolerations for a pod before you can schedule the pod to a virtual node. Example:

    apiVersion: v1
    kind: Pod
    metadata:
      name: nginx
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: nginx
      nodeSelector:
        type: virtual-kubelet
      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Exists
  • DaemonSets cannot be deployed because Elastic Container Instance is connected to Kubernetes clusters by using Virtual Kubelet. Virtual nodes are used, instead of real compute nodes. When you create a DaemonSet, you must configure the anti-affinity scheduling policy to prevent the DaemonSet from being scheduled to a virtual node. You must add the following information to the spec.template.spec.affinity field of the DaemonSet to be deployed:

    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
             - key: type
                operator: NotIn
                values:
                - virtual-kubelet