Configure ECI Auto Scaling in ACK - Container Service for Kubernetes

When deploying services in an ACK cluster, you can use tolerations and node affinity to specify whether to use only ECS instances or elastic container instances, or to automatically request elastic container instances when ECS resources are insufficient. You can configure different scheduling policies to scale resources in various scenarios.

Terms

Taints: Rules set on nodes to repel certain pods from being scheduled to these nodes.
Tolerations: Applied to pods to allow them to be scheduled to nodes with specific taints.
Node affinity: A property of pods that attracts them to nodes with specific labels. Node affinity includes the following types:
- requiredDuringSchedulingIgnoredDuringExecution: Hard requirements that must be met for scheduling to occur. The scheduler can schedule pods only when these rules are satisfied.
- preferredDuringSchedulingIgnoredDuringExecution: Preferences that are considered during scheduling. The scheduler attempts to find nodes that satisfy the corresponding rules. If no matching nodes are found, the scheduler still schedules the pod.

For more information, see Taints and Tolerations and Node affinity.

Prerequisites

The cluster type is ACK managed cluster Pro, and the version is v1.22 or later.
The ack-virtual-node component is deployed in the cluster, and the version is v2.10.0 or later. For information about how to install and upgrade the ack-virtual-node component, see Virtual node.
The version of the kube-scheduler component in the cluster is 5.9 or later, and the virtual node scheduling policy is enabled.
On the Components page, find Kube Scheduler, click Configure, and then confirm that Enable Virtual Node Scheduling is selected.

Sample configuration

The virtual node deployed in an ACK cluster has a default taint: virtual-kubelet.io/provider=alibabacloud:NoSchedule to prevent the use of elastic container instances without your knowledge. Therefore, you need to configure a toleration for this taint to schedule pods to the virtual node and use elastic container instances.

      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Equal
        value: alibabacloud
        effect: NoSchedule

By combining tolerations and node affinity, you can implement mixed scheduling policies with priorities and requirements to meet different resource needs for different services.

Preferably use ECS instances: Prioritize ECS resources. When ECS resources in the cluster are insufficient, use elastic container instances.
Use only elastic container instances: Use only elastic container instances, not ECS resources in the cluster.
Use only ECS instances: Use only existing ECS resources in the cluster.

Preferably use ECS instances

Because virtual nodes have default taints, you need to configure tolerations for these taints to schedule pods to virtual nodes. At the same time, configure node affinity. By using preferredDuringSchedulingIgnoredDuringExecution, you can specify pod scheduling preferences: preferably schedule pods to nodes without the type: virtual-kubelet label (ECS nodes).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ecs-prefer
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Equal
        value: alibabacloud
        effect: NoSchedule
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: type
                operator: NotIn
                values:
                - virtual-kubelet
      containers:
      - name: my-container
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6

Note

When configuring node affinity, note the following information:

If you configure multiple conditions in nodeSelectorTerms (multiple matchExpressions), pods can be scheduled to the corresponding nodes as long as one of the conditions is met (conditions are combined with logical OR operations). If you configure multiple expressions in a single matchExpressions, pods can be scheduled to the corresponding nodes only when all expressions are met (expressions are combined with logical AND operations).
When you use preferredDuringSchedulingIgnoredDuringExecution to implement preferential ECS scheduling, it cannot guarantee that pods are scheduled to elastic container instances only when ECS resources are insufficient. Pods may still be scheduled to elastic container instances even when ECS resources are sufficient. If you have higher requirements, we recommend that you combine requiredDuringSchedulingIgnoredDuringExecution to configure more detailed mandatory rules.

If you want to preferably deploy pods on ECS nodes with specific labels (such as label_1=key_1) and use virtual nodes for elastic scaling when resources on these nodes are insufficient, you can use the following YAML configuration.

Expand to view YAML details

apiVersion: apps/v1
kind: Deployment
metadata:
  name: some-ecs-prefer
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Equal
        value: alibabacloud
        effect: NoSchedule
      affinity:
        nodeAffinity:
    # Schedule pods only to nodes with the label_1:key_1 or type:virtual-kubelet label.
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: label_1
                operator: In
                values:
                - key_1
            - matchExpressions:
              - key: type
                operator: In
                values:
                - virtual-kubelet
    # Schedule pods preferably to nodes with the label_1:key_1 label, and then to nodes with the type:virtual-kubelet label.
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            preference:
              matchExpressions:
              - key: label_1
                operator: In
                values:
                - key_1
          - weight: 1
            preference:
              matchExpressions:
              - key: type
                operator: In
                values:
                - virtual-kubelet
      containers:
      - name: my-container
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6

Use only elastic container instances

Because virtual nodes have default taints, you need to configure tolerations for these taints to schedule pods to virtual nodes. At the same time, configure node affinity. By using requiredDuringSchedulingIgnoredDuringExecution, you can specify that pods must be scheduled to nodes with the type: virtual-kubelet label (virtual nodes).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: eci-only
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Equal
        value: alibabacloud
        effect: NoSchedule
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: type
                operator: In
                values:
                - virtual-kubelet
      containers:
      - name: my-container
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6

Use only ECS instances

Because virtual nodes have default taints, as long as you do not configure tolerations for these taints, pods will be scheduled only to ECS nodes.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ecs-only
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-container
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6