Use ECI with a self-managed cluster on Alibaba Cloud ECS - Elastic Container Instance

Deploy a virtual node (VNode) in your self-managed Kubernetes cluster on ECS to schedule pods through Elastic Container Instance (ECI) within the same VPC.

Background

ECI manages pod scheduling and resources at the infrastructure layer; Kubernetes handles workload management above it. Add a VNode to your self-managed cluster on ECS to use ECI. Connect a self-managed Kubernetes cluster to ECI covers the full integration process.

Prerequisites

A self-managed Kubernetes cluster (v1.13–1.30) on ECS, created with kubeadm.
Flannel, Calico, or Cilium network plug-in deployed in the cluster.

Install vnodectl

ECI provides vnodectl to connect and manage VNodes. We recommend installing it on a control plane node of your cluster.

Connect to your Kubernetes cluster.

Download the VNodectl package.

wget https://eci-docs.oss-cn-beijing.aliyuncs.com/vnode/vnodectl_0.0.5-beta_linux_amd64.tar.gz -O vnodectl.tar.gz

Extract the package.
```
tar xvf vnodectl.tar.gz
```
Move the binary to a directory in your PATH.
```
cp vnodectl /usr/local/bin/vnode
```

Configure the ~/.vnode/config file

Modify the ~/.vnode/config file.

vim ~/.vnode/config

Add your configuration. Example:

Important

The kubeconfig requires cluster-admin permissions. Limited-permission alternatives are covered in Configure the cluster.
The VNode must be able to reach the API server address in the kubeconfig.

kind: vnode
contexts:
    - name: default                                          # The name of the context.
      region-id: cn-hangzhou                                 # The region ID.
      access-key-id: <yourAccessKeyID>                       # The AccessKey ID.
      access-key-secret: <yourAccessKeySecret>               # The AccessKey secret.
      vswitch-id: vsw-7xv2yk45qp5etidgf****                  # The ID of the vSwitch to which the VNode belongs.
      security-group-id: sg-7xv5tcch4kjdr65t****             # The ID of the security group to which the VNode belongs.
      kubeconfig: /path/to/kubeconfig                        # The path to the cluster's kubeconfig file.
current-context: default

Set the active context.
```
vnode config set-context <context-name>
```

Create a VNode

Create a VNode.

vnode create

The output includes the VirtualNodeId:

{"RequestId":"AB772F9D-2FEF-5BFD-AAFB-DA3444851F29","VirtualNodeId":"vnd-7xvetkyase7gb62u****"}

Verify the node.

kubectl get node

The VNode appears with STATUS Ready:

NAME                                    STATUS     ROLES                  AGE    VERSION
cn-hangzhou.vnd-7xvetkyase7gb62u****    Ready      agent                  174m   v1.20.6
vnode-test001                           Ready      control-plane,master   23h    v1.20.6
vnode-test002                           Ready      <none>                 22h    v1.20.6

Prevent DaemonSet scheduling to a VNode

Because a VNode is a virtual node, it cannot run DaemonSets. After creating a VNode, you must modify the kube-proxy DaemonSet and configure nodeAffinity to prevent it from being scheduled to the VNode.

Modify the DaemonSet configuration.

kubectl -n kube-system edit ds kube-proxy

Configure nodeAffinity.

Add the following YAML content to spec.template.spec:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: type
          operator: NotIn
          values:
          - virtual-kubelet

Schedule pods to a VNode

After creating a VNode, you can schedule pods to it by using one of the following methods:

Manual scheduling

You can manually schedule pods to a VNode by configuring nodeSelector and tolerations, or by specifying nodeName. For more information, see Schedule pods to a VNode.
Automatic scheduling

After you deploy the eci-profile component, you can configure a custom selector to automatically schedule eligible pods to a VNode. For more information, see Use eci-profile to schedule pods to a VNode.

Establish pod network connectivity

Pods on the VNode receive an ENI and an internal IP address from the VPC vSwitch.

By default, VNode pods cannot reach pods on the cluster overlay network (Flannel, Calico, or Cilium), though the reverse works. Add a route entry to the VPC route table to direct outbound VNode pod traffic to the correct ECS nodes.

Example route configuration:

Example scenario

In this example, test1 runs on the VNode and test2 on an ECS node. By default, test2 can reach test1, but not vice versa.

NAME      READY     RESTARTS    AGE    IP                NODE                                   NOMINATED NODE   READINESS NODE
test1     1/1       0           58s    192.168.0.245     cn-hangzhou.vnd-7xvetkyase7gb62u****   <none>           <none>
test2     1/1       0           35s    10.88.1.4         vnode-test002                          <none>           <none>

Procedure
1. Log on to the VPC console.
2. In the left-side navigation pane, click Route Tables.
3. Switch to your cluster's region, find the VPC route table, and click its ID.
4. On the Route Entry List tab, click the Custom Route tab.
5. Click Add Route Entry.
6. In the Add route entry panel, configure the route entry and click OK.
  
  Example configuration:
  - Destination CIDR Block: Enter the CIDR block for the pods on the target ECS node, such as 10.88.1.0/24.
  - Next Hop Type: Select ECS Instance.
  - ECS Instance: Select the target ECS node.
Verify the result

Run kubectl exec to open a shell in test1 and ping test2. A successful response confirms that test1 can now reach test2.

Elastic Container Instance:Integrate a VNode with a self-managed Kubernetes cluster on ECS