All Products
Search
Document Center

Elastic Container Instance:Connect a self-managed Kubernetes cluster to a VNode

Last Updated:Jul 28, 2022

If you have self-managed Kubernetes clusters in a data center or on Alibaba Cloud Elastic Compute Service (ECS), you need to deploy virtual nodes (VNodes) to use elastic container instances. This topic describes how to connect a self-managed Kubernetes cluster to a VNode.

Prerequisites

  • A Kubernetes cluster whose version is between 1.13 and 1.22 is deployed.

  • If your Kubernetes cluster is deployed in a data center, make sure that the data center and the virtual private cloud (VPC) are connected with each other by using Express Connect, Smart Access Gateway, or VPN Gateway. For more information, see the following topics:

Preparations

Before you connect a self-managed Kubernetes cluster to a VNode, understand the permissions required by the VNnode and prepare the following parameter information required to create the VNode.

Parameter

Description

Operation

Region

A region is a geographic location where Alibaba Cloud data centers are deployed. Select a region based on the geographical location of your end users and resource prices. For more information, see Regions and zones.

The elastic container instance and relevant resources are created in the region that you selected.

You can query the regions where Elastic Container Instance is available by using the Elastic Container Instance console or by calling the DescribeRegions API operation.

VPC

A VPC is a private network established on Alibaba Cloud. VPCs are logically isolated from each other. For more information, see What is a VPC?

The elastic container instance and relevant resources are created in the VPC that you configured.

Note

If your Kubernetes cluster is deployed in a data center, make sure that the network of the data center and the VPC are connected with each other.

You can create and view VPCs on the VPCs page in the VPC console.

vSwitch

A vSwitch is a basic network device that connects different cloud resources. If you want to create an elastic container instance and relevant resources in a VPC, you must specify a vSwitch. You can also specify multiple vSwitches. In this case, the system automatically selects a vSwitch from the pool of vSwitches.

You can create and view vSwitches on the vSwitch page in the VPC console and select a vSwitch in the VPC that you selected.

Security Group

Security groups are virtual firewalls that control the inbound and outbound traffic of resources in the group to improve network security. For more information, see Overview.

The elastic container instance and relevant resources are added to the security group that you selected.

Note

We recommend that you select an advanced security group and add the following inbound security group rules:

  • One rule allows access to all ports from 1 to 65535 over the CIDR blocks of the VPC.

  • The other rule allows access to the service ports 10250 and 10255 of the VNode over the endpoint of the Kubernetes API server.

You can create and view security groups on the Security Groups page in the ECS console and select a security group in the VPC that you selected.

To deploy a VNode, you need to configure standard permissions for native Kubernetes nodes and additional permissions for the VNode, as shown in the following table.

Type

Permissions

Description

Standard permissions

system:node

Standard permissions for nodes.

system:node-proxier

Standard permissions for kube-proxy.

system:certificates.k8s.io:certificatesigningrequests:nodeclient

The permission for nodes to initiate the request to create a certificate.

system:certificates.k8s.io:certificatesigningrequests:selfnodeclient

The permission for nodes to initiate a request to create a certificate based on existing certificates. The permission is to request for certificate rotation.

Additional permissions

pods update and pods patch

Update annotations of pods. For example, you can add computed annotations such as k8s.aliyun.com/eci-instance-id to pods.

pvc update and pvc patch

Update annotations of PersistentVolumeClaims (PVCs).

endpoints create and endpoints update

Leader election assurance.

You do not need to configure these permissions for clusters of versions later than V1.13.

Step 1: Generate a KubeConfig certificate

To connect VNodes to Kubernetes cluster, a KubeConfig certificate is required. Therefore, before you create a VNode, you need to issue a KubeConfig certificate to the VNode. Perform the following operations:

Note

If you use an admin certificate, skip this section.

  1. Download and execute a configuration script to generate a KubeConfig certificate.

    In TLS Bootstrap or ServiceAcoount mode, run the appropriate command to generate a KubeConfig certificate based on the version of your Kubernetes cluster.

    • TLS Bootstrap

      If you have security requirements, we recommend that you use the TLS Bootstrap mode. In this mode, the vnode-approver component is automatically deployed. The vnode-approver component is used to issue certificate signing requests (CSRs) that are sent by Vnodes. vnode-approver has open source code on GitHub. For more information, see vnode-approver.

      • If the cluster version is between 1.14 and 1.22, run the following command to generate a KubeConfig certificate:

        curl -fsSL https://eci-release.oss-cn-beijing.aliyuncs.com/vnode-deploy.sh | bash -s -- bootstrap
      • If the cluster version is 1.13, run the following command to generate a KubeConfig certificate:

        For clusters of v1.13, the endpoint resource needs to select a leader. You must configure additional permissions by setting EXTRA_PRIVILEDGE=true in the script.

        curl -fsSL https://eci-release.oss-cn-beijing.aliyuncs.com/vnode-deploy.sh | env EXTRA_PRIVILEDGE=true bash -s -- bootstrap
    • ServiceAcoount

      This mode does not support certificate rotation. When you issue a KubeConfig certificate, ensure that the token has a long enough or permanent validity period to prevent the VNode from failing due to expiration of the KubeConfig.

      • If the cluster version is between 1.14 and 1.22, run the following command to generate a KubeConfig certificate:

        curl -fsSL https://eci-release.oss-cn-beijing.aliyuncs.com/vnode-deploy.sh | bash -s -- common
      • If the cluster version is 1.13, run the following command to generate a KubeConfig certificate:

        For clusters of v1.13, the endpoint resource needs to select a leader. You must configure additional permissions by setting EXTRA_PRIVILEDGE=true in the script.

        curl -fsSL https://eci-release.oss-cn-beijing.aliyuncs.com/vnode-deploy.sh | env EXTRA_PRIVILEDGE=true bash -s -- common
  2. View KubeConfig.

    The KubeConfig is named vnode-config. Run the following command to view KubeConfig:

    cat vnode-config

    When you view KubeConfig, take note of the following items:

    • Make sure that the VPC in which the VNode is to be created can be connected to the server. To ensure the VPC can be connected to the server, the VNode and the server must reside in the same VPC, or the VPC and the server can communicate with each other.

      • If the server address is an IP address, make sure that you can access the IP address from the VPC in which the VNode resides.

      • If the server address is a domain name, make sure that the domain name can be resolved in an Alibaba Cloud VPC. In addition, from the VPC in which the VNode resides, you must be able to access the IP address that is obtained after the domain name is resolved.

    • If the content of certificate-authority-data is empty, check whether the PATH field is used in ~/.kube/config.

    The following code provides a sample content of the KubeConfig.

    • TLS Bootstrap

      apiVersion: v1
      kind: Config
      current-context: kubernetes-admin@kubernetes
      contexts:
      - name: kubernetes-admin@kubernetes
        context:
          cluster: kubernetes-admin@kubernetes
          user: vnode
          namespace: kube-system
      clusters:
      - name: kubernetes-admin@kubernetes
        cluster:
          certificate-authority-data: "*****************************************=="
          server: https://10.16.XX.XX:6443
      users:
      - name: vnode
        user:
          token: ******.****************
    • ServiceAcoount

      apiVersion: v1
      kind: Config
      current-context: kubernetes-admin@kubernetes
      contexts:
      - name: kubernetes-admin@kubernetes
        context:
          cluster: kubernetes-admin@kubernetes
          user: vnode
          namespace: kube-system
      clusters:
      - name: kubernetes-admin@kubernetes
        cluster:
          certificate-authority-data: "*****************************************=="
          server: https://10.16.XX.XX:6443
      users:
      - name: vnode
        user:
          token: ***********************************************

Step 2: Create a VNode

You can use the Elastic Container Instance console or call an API operation to create a VNode.

Method 1: Use the Elastic Container Instance console

  1. Log on to the Elastic Container Instance console.

  2. In the top navigation bar, select a region.

  3. In the left-side navigation pane, click Virtual Node.

  4. Click Create Virtual Node.

  5. Configure parameters and click OK.

    The following table describes the parameters that you need to configure.

    Parameter

    Description

    VPC, vSwitch, and Security Group

    The VPC and security group to which the VNode belongs. The vSwitch that the VNode is connected with. For information about how to create a VPC, vSwitch, and security group, see the Preparations section in this topic.

    KubeConfig

    The KubeConfig of the Kubernetes cluster with which the VNode is connected. For information about how to generate a KubeConfig certificate, see Step 1.

    TlsBootstrapEnabled

    Specifies whether to enable TLS bootstrapping. If a KubeConfig certificate has been generated by using the TLS Bootstrap mode, enable the configuration. After you enable the configuration, certificate rotation is also enabled. The system automatically applies for a new KubeConfig certificate for the VNode when the current KubeConfig certificate is about to expire.

    Label

    The label that is bound to the VNode.

    Taint

    The taint that is bound to the VNode.

    Elastic IP Address

    If the VNode needs to access the Internet, you must associate an elastic IP address (EIP) with the VNode.

    • Automatically Create EIP: The system automatically creates an EIP and associates it with the VNode.

    • Use Existing EIP: The system associates an existing EIP with the VNode.

    Note

    An EIP is a public IP address that can be separately purchased and owned. For more information, see What is an EIP?

    Resource group

    A resource group is a mechanism in which resources are managed by groups under an Alibaba Cloud account. Each account is assigned with a default resource group. For more information, see What is Resource Management?

    If you do not specify a resource group, the VNode is added to the default resource group.

Method 2: Call an API operation

You can call the CreateVirtualNode API operation to create a VNode. The following table lists the major parameters. For more information, see CreateVirtualNode.

Parameter

Type

Example

Description

RegionId

String

cn-shanghai

The region ID of the VNode.

VSwitchId

String

vsw-2ze23nqzig8inprou****

The ID of the vSwitch. The vSwitch is connected to the VNode and the elastic container instances in the VNode.

You can specify 1 to 10 vSwitches for a VPC.

SecurityGroupId

String

sg-2ze81zoc3yl7a3we****

The ID of the security group. The VNode and the elastic container instances in the VNode are added to the security group.

KubeConfig

String

JTVDbmFwaVZlcnNpb24lM0ElMjB2MSU1Q25jbHVzdGVycyUzQSU1Q24tJTIwY2x1c3RlciUzQSU1Q24uLi******

The KubeConfig of the Kubernetes cluster with which the VNode is connected. The value must be Base64-encoded.

TlsBootstrapEnabled

Boolean

true

Specifies whether to enable TLS bootstrapping. If a KubeConfig certificate has been generated by using the TLS Bootstrap mode, enable the configuration. After you enable the configuration, certificate rotation is also enabled. The system automatically applies for a new KubeConfig certificate for the VNode when the current KubeConfig certificate is about to expire.

Check the result

Wait for a few minutes after you create the VNode. Then, log on to the Kubernetes cluster and run the kubectl get nodes command to view the node information.

  • If the information of the VNode is displayed, the VNode is created. The following command output is expected to return:

    vnode1
  • If no information of the VNode is displayed, the VNode fails to be created. We recommend that you troubleshoot the issue based on VNode-related events.

    On the Virtual Node page of the Elastic Container Instance console, click the ID of the VNode that fails to be created. On the Events tab, you can view the events about the VNode.

    Common event errors:

    • failed to get kubernetes server version, Get \"https://10.50.XX.XX:6443/version?timeout=32s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers

      This error message indicates that the VPC in which the VNode resides cannot be connected to the Kubernetes API server. To solve this issue, check whether the VNode and the API server reside in the same VPC. If they are not in the same VPC, make sure that the VPC in which the VNode resides can communicate with the API server.

    • failed to get kubernetes server version, Get \"https://lb.kubesphere.local:6443/version?timeout=32s\": dial tcp: lookup lb.kubesphere.local on 100.100.X.:53: no such host

      This error message indicates that the domain name of the Kubernetes API server cannot be resolved in an Alibaba Cloud VPC. You can use Alibaba Cloud DNS PrivateZone to resolve the domain name.

Note

If you have any requirements or if you encounter problems when you connect a self-managed Kubernetes cluster to a VNode, you can join the DingTalk group with the ID of 44666389 to obtain assistance.

Step 3: Configure an anti-affinity policy

DaemonSets cannot run on VNodes because VNodes are not real nodes. After you create a VNode, you need to modify the DaemonSet in kube-proxy and configure nodeAffinity to prevent DaemonSets from being scheduled to the VNode. Perform the following operations:

  1. Run the following command to modify the configurations of the DaemonSet:

    kubectl -n kube-system edit ds kube-proxy
  2. Configure nodeAffinity.

    Add the following YAML content to spec > template > spec:

    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: type
              operator: NotIn
              values:
              - virtual-kubelet

What to do next

After you create the VNode, you can use one of the following methods to schedule pods to the VNode. Then, you can run the pods on the elastic container instances that are deployed in the VNode.

  • Manual scheduling

    You can configure the nodeSelector and tolerations parameters or specify the nodeName parameter to schedule pods to the VNode. For more information, see Schedule pods to a VNode.

  • Automatic Scheduling

    The ECI Profile feature allows you to customize selectors. Then, Elastic Container Instance automatically schedules pods that meet the conditions of the selectors to the VNode. For more information about specific operations, see Configure an ECI Profile to orchestrate pods.