All Products
Search
Document Center

Use Elastic Container Instance in offline Kubernetes clusters

Last Updated: Apr 20, 2021

If you have offline Kubernetes clusters in data centers or other offline environments, you must deploy Virtual Kubelet before Elastic Container Instance can be used in the clusters. This topic describes how to deploy Virtual Kubelet.

Prerequisites

Limits

The following features are not supported in hybrid cloud environments:

  • Annotations update.

  • Labels update.

  • Spec.ActiveDeadlineSeconds update.

  • ConfigMap and Secret update.

  • kubectl logs-f command. The kubectl logs command is supported, but kubectl logs-f is not.

  • kubectl attach command.

  • kubectl port-forward command.

  • Some parameters of Downward API such as status.hostIP are unavailable.

  • ClusterIP.

    You must establish connections between offline clusters and Alibaba Cloud networks by using Express Connect or Smart Access Gateway. For more information, see Express Connect or Smart Access Gateway.

Preparations

The following table describes the parameters that you must obtain before you deploy Virtual Kubelet.

Parameter

Description

Obtaining method

ECI_ACCESS_KEY

The AccessKey ID of your Alibaba Cloud account.

For more information, see Obtain an AccessKey pair.

ECI_SECRET_KEY

The AccessKey secret of your Alibaba Cloud account.

For more information, see Obtain an AccessKey pair.

ALIYUN_CLUSTERID

The ID of the cluster.

You can customize the ID of the cluster. The ID is the unique identifier of the cluster.

ECI_REGION

The ID of the region.

You can query available regions by using the Elastic Container Instance console or calling the DescribeRegions operation.

ECI_VPC

The ID of the VPC.

You can query the VPC ID on the VPCs page in the VPC console.

ECI_VSWITCH

The ID of the vSwitch.

You can query the ID of the vSwitch on the VSwitches page in the VPC console based on the selected VPC.

ECI_SECURITY_GROUP

The ID of the security group.

You can log on to the VPC console and find the selected VPC. Click the VPC ID to go to the VPC details page. On the Resources tab in the lower part of the page, click the number corresponding to the security group to go to the Security Groups page and obtain the ID of the security group.

Deploy Virtual Kubelet

  1. Create a service account for Virtual Kubelet and bind a cluster role to the account to create pods.

    1. Run the following command to create a service account named vk-admin:

      kubectl create serviceaccount vk-admin -n kube-system
    2. Run the following command to bind the cluster-admin cluster role to the vk-admin account:

      kubectl create clusterrolebinding vk-admin-binding --clusterrole=cluster-admin --serviceaccount=kube-system:vk-admin
  2. Create the vk.yaml template file required to deploy Virtual Kubelet.

    The following code provides an example of the YAML file content. You must replace the parameter values with your own information.

    Notice

    The version of Virtual Kubelet must be v2.0.0.608-0b919e1d2-aliyun or later.

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: virtual-node-controller
      name: virtual-node-controller
      namespace: kube-system
    spec:
      serviceName: vk-svc
      replicas: 1
      selector:
        matchLabels:
          app: virtual-node-controller
      template:
        metadata:
          labels:
            app: virtual-node-controller
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: type
                    operator: NotIn
                    values:
                    - virtual-kubelet
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchExpressions:
                    - key: app
                      operator: In
                      values:
                      - virtual-node-controller
                  topologyKey: kubernetes.io/hostname
                weight: 100
          containers:
          - name: virtual-node-controller
            # The image and its version required to deploy Virtual Kubelet.
            image: registry.cn-beijing.aliyuncs.com/acs/virtual-nodes-eci:v2.0.0.34-252556a33-aliyun
            imagePullPolicy: Always
            args:
            - --provider
            - alibabacloud
            - --nodename
            - $(VN_INSTANCE)
            env:
            - name: VN_INSTANCE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: KUBELET_PORT
              value: "10250"
            - name: VKUBELET_POD_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
            - name: VKUBELET_TAINT_KEY
              value: "virtual-kubelet.io/provider"
            - name: VKUBELET_TAINT_VALUE
              value: "alibabacloud"
            - name: VKUBELET_TAINT_EFFECT
              value: "NoSchedule"
            - name: ECI_QUOTA_CPU
              value: "1000000"
            - name: ECI_QUOTA_MEMORY
              value: 6400Ti
            - name: ECI_QUOTA_POD
              value: "3000"
            - name: ECI_KUBE_PROXY
              value: "true"
            # The ID of the region used to create the elastic container instance.
            - name: ECI_REGION
              value: <region-id>
            # The AccessKey ID used to create the elastic container instance.
            - name: ECI_ACCESS_KEY
              value: <access-key>
            # The AccessKey secret used to create the elastic container instance.
            - name: ECI_SECRET_KEY
              value: <secret-kty>
            # The ID of the VPC used to create the elastic container instance.
            - name: ECI_VPC
              value: <vpc-id>
            # The ID of the vSwitch used to create the elastic container instance.
            - name: ECI_VSWITCH
              value: <vsw-id>
            # The ID of the security group used to create the elastic container instance.
            - name: ECI_SECURITY_GROUP
              value: <security-group-id>
            # The ID of the cluster customized when the elastic container instance is being created.
            - name: ALIYUN_CLUSTERID
              value: <cluster-id>
            - name: ALIYUN_PRIVATE_ZONE
              value: "false"
            # The mode of the hybrid cloud environments when the elastic container instance is being created.
            - name: ECI_HYBRID_MODE
              value: "true"
          dnsPolicy: ClusterFirst
          restartPolicy: Always
          schedulerName: default-scheduler
          serviceAccount: admin
          serviceAccountName: admin
  3. Deploy Virtual Kubelet.

    kubectl apply -f vk.yaml
  4. View the deployment status.

    kubectl get deploy/virtual-node-controller -n kube-system

    The following code provides an example of the returned result after the deployment is complete:

    NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
    virtual-node-controller            1/1         1                      1                  161m
  5. View node information after Virtual Kubelet is deployed.

    kubectl get nodes

    If the virtual-kubelet node is displayed in the Ready state, Virtual Kubelet is deployed. Example:

    NAME                STATUS   ROLES    AGE   VERSION
    master-1            Ready    <none>   19d   v1.18.8-aliyun.1
    master-2           Ready    <none>   19d   v1.18.8-aliyun.1
    virtual-kubelet    Ready    agent     18d   v1.18.8-aliyun.1

Schedule a pod to a virtual node

If virtual nodes exist in the cluster, you can schedule pods to the virtual nodes. Perform the following operations:

  • Virtual nodes have specific taints. You must set node selectors and tolerations for a pod before you can schedule the pod to a virtual node. Example:

    apiVersion: v1
    kind: Pod
    metadata:
      name: nginx
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: nginx
      nodeSelector:
        type: virtual-kubelet
      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Exists
  • DaemonSets cannot be deployed because Elastic Container Instance connects to Kubernetes clusters by using Virtual Kubelet. Virtual nodes are used, instead of real compute nodes. When you create a DaemonSet, you must configure the anti-affinity scheduling policy to prevent the DaemonSet from being scheduled to a virtual node. Example:

     affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: type
                    operator: NotIn
                    values:
                    - virtual-kubelet

Upgrade Virtual Kubelet

If you want to use new features, you may need to update the image version of Virtual Kubelet.

You can run the following command to edit corresponding resources and modify the value of the image tag to the desired version:

kubectl edit deployment -n kube-system virtual-node-controller