All Products
Search
Document Center

Use Elastic Container Instance in offline Kubernetes clusters

Last Updated: Sep 08, 2021

If you have self-managed Kubernetes clusters in your on-premises data centers or other offline environments, you must deploy virtual nodes (Virtual Kubelet) in the clusters before you can use Elastic Container Instance in the clusters. This topic describes how to connect offline self-managed clusters to Elastic Container Instance, including how to register self-managed clusters in the Container Service for Kubernetes (ACK) console and deploy virtual nodes, or deploy Virtual Kubelet in self-managed clusters.

Prerequisites

An Alibaba Cloud account is created. For more information, see Sign up with Alibaba Cloud.

Background information

Elastic Container Instance can seamlessly connect to Kubernetes by using virtual nodes based on Virtual Kubelet provided by the Kubernetes community. Kubernetes clusters can obtain high elasticity without being limited by the computing capacities of cluster nodes. To connect Elastic Container Instance to your Kubernetes clusters, use the following methods:

  • Method 1: Register your clusters and deploy virtual nodes

  • Method 2: Deploy Virtual Kubelet in your clusters

Note

We recommend that you use Method 1. This method allows you to update the Virtual Kubelet version (the ack-virtual-node version) to use new features. For more information, see Update Virtual Kubelet.

Limits

The following limits apply to hybrid cloud environments:

  • Annotations update is not supported.

  • Labels update is not supported.

  • Spec.ActiveDeadlineSeconds update is not supported.

  • ConfigMap and Secret update is not supported.

  • The kubectl logs-f command is not supported. The kubectl logs command is supported.

  • The kubectl attach command is not supported.

  • The kubectl port-forward command is not supported.

  • Some parameters of Downward API such as status.hostIP are unavailable.

  • ClusterIP

    You must establish connections between offline clusters and Alibaba Cloud networks by using Express Connect or Smart Access Gateway. For more information, see Express Connect or Smart Access Gateway.

Preparations

The following table describes the parameters that you must obtain before you can deploy Virtual Kubelet.

Parameter

Description

Method to obtain

ECI_ACCESS_KEY and ECI_SECRET_KEY

The AccessKey ID and corresponding AccessKey secret. These values of these parameters are the credentials used to manage Elastic Container Instance in virtual nodes.

For more information, see Obtain an AccessKey pair.

ALIYUN_CLUSTERID

The ID of the cluster, which is the unique identifier of the cluster.

If you register a cluster, the cluster ID is generated by the system. If you deploy Virtual Kubelet in a cluster, you must customize the cluster ID.

ECI_REGION

The region ID of the elastic container instance. The elastic container instance is deployed within this region.

You can query supported regions by using the Elastic Container Instance console or by calling the DescribeRegions operation.

ECI_VPC

The ID of the virtual private cloud (VPC). The elastic container instance is deployed within this VPC.

You can create and view VPCs on the VPCs page in the VPC console.

ECI_VSWITCH

The ID of the vSwitch. The elastic container instance is associate with this vSwitch.

You can create and view vSwitches on the vSwitch page in the VPC console and select the corresponding vSwitch based on the selected VPC.

ECI_SECURITY_GROUP

The ID of the security group. The elastic container instance is added to this security group.

You can create and view security groups on the Security Groups page in the ECS console and select the corresponding vSwitch based on the selected VPC.

Method 1: Register your clusters and deploy virtual nodes

You can register your clusters in the ACK console and then deploy virtual nodes to use Elastic Container Instance. Perform the following steps:

  1. Log on to the ACK console.

  2. In the left-side navigation pane, click Clusters.

  3. Create a registered cluster.

    1. On the Clusters page, click Create Kubernetes Cluster.

    2. Click the Register Cluster tab. Specify the parameters for the cluster and click Create Cluster.

      The following table describes the parameters required to create a Kubernetes cluster. For more information, see Register an external Kubernetes cluster.

      Parameter

      Description

      Region, VPC, and VSwitch

      Select a region, virtual private cloud (VPC), and vSwitch for the cluster.

      Access to API Server

      Select the specifications of the SLB instance to use for the API Server. By default, an internal-facing Server Load Balancer (SLB) instance is created for the API server. You can select custom specifications based on your business requirements.

      EIP

      Specify whether to create and associate an elastic IP address (EIP) to connect to the cluster.

      Security group

      Configure the security group that is automatically created to isolate security domains and control network traffic.

      Log Service

      Specify whether to activate Log Service to collect log data from containers.

      Deletion Protection

      Specify whether to enable deletion protection for the cluster. If you enable deletion protection, the cluster cannot be deleted by using the ACK console or by calling API operations.

  4. Register the cluster.

    1. On the Clusters page, find the cluster that you created and click the cluster name.

    2. On the Cluster Information page, click the Connection Information tab.

    3. Create a ConfigMap in the cluster.

      Click the Public Network or Internal Network tab based on your network. Copy the content to a YAML configuration file such as agent.yaml and run the kubectl apply -f agent.yaml command in the cluster to create a ConfigMap.

    4. Run the following command in the cluster to check the connection status:

      kubectl -n kube-system get pod |grep ack-cluster-agent

      The following command output is returned:

      ack-cluster-agent-5f7d568f6-6fc4k              1/1     Running   0          9s
      ack-cluster-agent-5f7d568f6-tf6fp              1/1     Running   0          9s
  5. Deploy virtual nodes.

    1. On the Clusters page, click the name of the cluster in which you want to deploy a virtual node.

    2. In the left-side navigation pane of the details page, choose Operations > Add-ons.

    3. Click the Others tab, find ack-virtual-node, and then click Install.

Method 2: Deploy Virtual Kubelet in your clusters

You can deploy Virtual Kubelet in your clusters to use Elastic Container Instance. You must obtain the latest Virtual Kubelet version before you can deploy Virtual Kubelet. For more information, see ack-virtual-node.

Perform the following steps:

  1. Prepare the vk.yaml configuration file required to deploy Virtual Kubelet.

    The following code provides an example of the YAML configuration file content. You must replace the parameter values with your own information.

    Notice

    The version of Virtual Kubelet must be v2.0.0.121-eff0e01c0-aliyun or later.

    apiVersion: v1
    kind: Secret
    metadata:
      name: vk-accesskey
      namespace: kube-system
    type: Opaque
    data:
      # Specify the Base64-encoded AccessKey key.
      accesskey.id: {{ ECI_ACCESS_KEY Base64 Encoding }}
      # Specify the Base64-encoded AccessKey secret.
      accesskey.secret: {{ ECI_SECRET_KEY Base64 Encoding }}
    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: eci-profile
      namespace: kube-system
    data:
      # If Elastic Container Instance cannot be connected to API Server, set this parameter to true.
      # Specify whether to run pods of Kubernetes clusters on both Elastic Container Instance and traditional servers. For example, if the offline self-managed Kubernetes cluster is not connected to a VPC, set this parameter to true. If the API Server of the self-managed cluster and the elastic container instance are deployed in different VPCs, set this parameter to true.
      enableHybridMode: "false"
      # Specify whether to enable the Cluster IP feature of Kubernetes. If you want to use this feature, set this parameter to true.
      enableClusterIp: "true"
      # Specify whether to enable PrivateZone. If you want to use this feature, set this parameter to true.
      enablePrivateZone: "false"
      # Specify the ID of the resource group. If you do not want to use this feature, set this parameter to "".
      resourceGroupId: ""
      # Specify the required security group ID.
      securityGroupId: "sg-2ze**********"
      # Specify the scheduling selector based on the actual situation.
      selectors: ""
      # Specify the required vSwitch ID.
      vSwitchIds: "vsw-2zeq***********"
      # Specify the required VPC ID.
      vpcId: "vpc-2ze0z************"
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        role: webhook
      name: vk-webhook
      namespace: kube-system
    spec:
      ports:
      - port: 443
        targetPort: 443
      selector:
        app: virtual-kubelet
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: virtual-kubelet
      namespace: kube-system
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: virtual-kubelet
    subjects:
    - kind: ServiceAccount
      name: virtual-kubelet
      namespace: kube-system
    roleRef:
      kind: ClusterRole
      name: cluster-admin
      apiGroup: rbac.authorization.k8s.io
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: ack-virtual-node-controller
      namespace: kube-system
      labels:
        app: virtual-kubelet
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: virtual-kubelet
      template:
        metadata:
          labels:
            app: virtual-kubelet
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: type
                    operator: NotIn
                    values:
                    - virtual-kubelet
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchExpressions:
                    - key: app
                      operator: In
                      values:
                      - virtual-node-eci
                  topologyKey: kubernetes.io/hostname
                weight: 100
          containers:
          - name: vk
            # Replace region-id with the ID of the region where the Virtual Kubelet is located. Replace vk-tag with the latest Virtual Kubelet version. You can find the latest Virtual Kubelet version in the version record of Virtual Kubelet.
            # Note: After you deploy Virtual Kubelet, pay attention to the release of Virtual Kubelet version and update Virtual Kubelet in a timely manner to obtain continuous support from Alibaba Cloud.
            image: registry-vpc.{{ region-id }}.aliyuncs.com/acs/virtual-nodes-eci:{{ vk-tag }}
            imagePullPolicy: IfNotPresent
            args:
            - --provider
            - alibabacloud
            - --nodename
            - virtual-kubelet
            resources:
              requests:
                memory: 30M
                cpu: 100m
            env:
            - name: WEBHOOK
              value: "true"
            - name: VKUBELET_TAINT_KEY
              value: "virtual-kubelet.io/provider"
            - name: VKUBELET_TAINT_VALUE
              value: "alibabacloud"
            - name: VKUBELET_TAINT_EFFECT
              value: "NoSchedule"
              # Replace region-id with the ID of the region where the elastic container instance is located, such as cn-beijing. You must specify this parameter.
            - name: ECI_REGION
              value: {{ region-id }}
            - name: ECI_QUOTA_CPU
              value: "1000000"
            - name: ECI_QUOTA_MEMORY
              value: 6400Ti
            - name: ECI_QUOTA_POD
              value: "10000"
            - name: ECI_KUBE_PROXY
              value: "true"
            - name: ECI_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: vk-accesskey
                  key: accesskey.id
            - name: ECI_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: vk-accesskey
                  key: accesskey.secret
            - name: ALIYUN_CLUSTERID
              # Replace cluster-id with the ID of the self-managed Kubernetes cluster. The ID of each Kubernetes cluster in an Alibaba Cloud account must be unique. You must specify this parameter.
              value: {{ cluster-id }}
            - name: KUBELET_PORT
              value: "10250"
            - name: VKUBELET_POD_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
          terminationGracePeriodSeconds: 60
          serviceAccountName: virtual-kubelet
  2. Deploy Virtual Kubelet.

    kubectl apply -f vk.yaml
  3. View the deployment status.

    kubectl get deploy/virtual-node-controller -n kube-system

    The following code provides an example of the command output if Virtual Kubelet is deployed:

    NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
    virtual-node-controller   1/1         1                      1                  161m
  4. View the node information after Virtual Kubelet is deployed.

    kubectl get node -o wide

    After Virtual Kubelet is deployed, the generated virtual node is named virtual-kubelet. If the state of virtual-kubelet is Ready, Virtual Kubelet is deployed. Example:

    NAME                  STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                     KERNEL-VERSION                CONTAINER-RUNTIME
    k8s-master01      Ready    <none>   50d    v1.14.2      192.168.*.*     <none>           CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.6
    k8s-master02      Ready    <none>   50d    v1.14.2      192.168.*.*     <none>           CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.6
    k8s-master03      Ready    <none>   50d    v1.14.2      192.168.*.*     <none>           CentOS Linux 7 (Core)   3.10.0-957.21.3.el7.x86_64   docker://18.9.6
    virtual-kubelet    Ready     agent      82m   v1.11.2      172.30.*.*     <none>           <unknown>                  <unknown>                         <unknown>

References

After you connect Kubernetes to Elastic Container Instance, you may need to manually schedule pods to run on Elastic Container Instance or use the features provided by Elastic Container Instance. For more information about how to use Elastic Container Instance, see Overview.