Specify the scheduling of Arm architecture or multi-architecture workloads to Arm virtual nodes - Container Service for Kubernetes

By default, ACK clusters and ACK Serverless clusters schedule all workloads to x86 virtual nodes. If your cluster has both Arm and non-Arm virtual nodes, use Kubernetes native scheduling (nodeSelector or nodeAffinity) so Arm-only workloads land on Arm virtual nodes and multi-architecture images can prefer Arm nodes.ACK clusters and to schedule Arm-only or multi-architecture workloads to Arm virtual nodes. By default, all workloads are scheduled to x86 virtual nodes.

Prerequisites

Cluster:
An ACK cluster running Kubernetes 1.20 or later. See Create an ACK managed cluster and Manually upgrade a cluster.
Note
Arm-based ECS instances are available only in specific regions and zones. Ensure your cluster is in a supported region. See Overview of instance types available for purchase.
The ack-virtual-node add-on v2.9.0 or later is installed.ack-virtual-node component v2.9.0 or later is installed.

Usage notes

For clusters running Kubernetes earlier than 1.24, add a toleration for the kubernetes.io/arch=arm64:NoSchedule taint when using nodeSelector or nodeAffinity to schedule workloads to Arm nodes. Kubernetes 1.24+ clusters automatically recognize the kubernetes.io/arch=arm64:NoSchedule taint, so the toleration is not required.

Billing

ARM-based ECS instance types and pricing:

Step 1: Add Arm virtual nodes

Create an Arm virtual node by editing the eci-profile ConfigMap with one of the following methods. See Configure an eci-profile.

Console

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Configurations > ConfigMaps.
From the Namespace drop-down list, select kube-system. Find eci-profile and click Edit. Set enableLinuxArm64Node to true, then click OK.
Note
If no vSwitch in your cluster is in a zone that supports Arm-based instances, create one in a supported zone and add its ID to the vSwitchIds field. See Create and manage vSwitches.
After about 30 seconds, the virtual node virtual-kubelet-<zoneId>-linux-arm64 appears on the Nodes page.

Kubectl

Prerequisites

Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

Procedure

Edit the ConfigMap:

kubectl edit configmap eci-profile -n kube-system

Set the enableLinuxArm64Node parameter to true.
Set the vSwitchIds parameter. Ensure at least one vSwitch in the vSwitchIds list is in a zone that supports Arm-based instances.
Note
If no vSwitch in your cluster is in a zone that supports Arm-based instances, create one in a supported zone and add its ID to the vSwitchIds field. See Create and manage vSwitches.
After about 30 seconds, the virtual node virtual-kubelet-<zoneId>-linux-arm64 appears on the Nodes page.

Step 2: Schedule to an Arm virtual node

Schedule Arm-only workloads

If a cluster has both Arm and non-Arm nodes and the application supports only the Arm architecture, schedule it to Arm nodes to prevent startup failures. All Arm nodes have the label kubernetes.io/arch=arm64. Use nodeSelector or nodeAffinity to target them.

nodeSelector

Add the following nodeSelector to the pod spec to target Arm virtual nodes. All Arm virtual nodes in an ACK cluster or have the arm64 label.

nodeSelector:
  kubernetes.io/arch: arm64 # Specify an Arm node.

The following sample YAML deploys a stateless application to an Arm virtual node.

Expand to view the YAML file

Note

The following YAML adds a toleration for the kubernetes.io/arch=arm64:NoSchedule taint. If your cluster is an ACK Pro cluster that runs Kubernetes 1.24 or later, the ACK scheduler automatically recognizes this taint and you do not need to add the toleration.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: only-arm
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      nodeSelector:
        kubernetes.io/arch: arm64 # Specify an Arm node.
      tolerations:
      # Tolerate the taint of the virtual node.
        - key: virtual-kubelet.io/provider
          operator: Exists
          effect: NoSchedule
      # Tolerate the taint on the Arm virtual node.
        - key: kubernetes.io/arch
          operator: Equal
          value: arm64
          effect: NoSchedule
      containers:
      - name: nginx
        image: alibaba-cloud-linux-3-registry.cn-hangzhou.cr.aliyuncs.com/alinux3/nginx_optimized:20240221-1.20.1-2.3.0

nodeAffinity

Prerequisites

The virtual node scheduling feature is enabled for the cluster, and the cluster and component versions meet the requirements.

Example

Add the following nodeAffinity to the pod spec to target Arm nodes with the kubernetes.io/arch=arm64 label.

With this constraint, the scheduler automatically tolerates the kubernetes.io/arch=arm64:NoSchedule taint.

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/arch
          operator: In
          values:
          - arm64

The following sample YAML deploys a stateless application to an Arm virtual node.

Expand to view the YAML file

Note

apiVersion: apps/v1
kind: Deployment
metadata:
  name: only-arm
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                - arm64
      tolerations:
       # Tolerate the taint of the virtual node.
        - key: virtual-kubelet.io/provider
          operator: Exists
          effect: NoSchedule
       # Tolerate the taint on the Arm virtual node.
        - key: kubernetes.io/arch
          operator: Equal
          value: arm64
          effect: NoSchedule         
      containers:
      - name: nginx
        image: nginx

Schedule multi-architecture images

Prerequisites

The virtual node scheduling feature is enabled for the cluster, and the cluster and component versions meet the requirements.

Example

ACK clusters and ACK Serverless clusters schedule all workloads to x86 virtual nodes by default and keep waiting for x86 capacity when x86 resources are insufficient. and schedule all workloads to x86 virtual nodes by default and continue to wait for x86 resources when they are insufficient. For multi-architecture images, configure cross-architecture scheduling to use both x86 and Arm nodes.

Configure node affinity to preferentially schedule workloads to Arm or x86 virtual nodes. If preferred resources are insufficient, the scheduler falls back to the other architecture.

      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                - arm64

Prefer Arm architecture

Sample YAML for a workload preferentially scheduled to an Arm virtual node:

Expand to view the YAML file

Note

apiVersion: apps/v1
kind: Deployment
metadata:
  name: arm-prefer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      # Tolerate the taint of the virtual node.
      - key: virtual-kubelet.io/provider
        operator: Exists
        effect: NoSchedule
      # Tolerate the taint on the Arm virtual node.
      - key: kubernetes.io/arch
        operator: Equal
        value: arm64
        effect: NoSchedule
      # Preferentially schedule the workload to a node that uses the Arm architecture.
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                - arm64
      containers:
      - name: my-container
        image: nginx

Prefer x86 architecture

Expand to view the YAML file

Note

apiVersion: apps/v1
kind: Deployment
metadata:
  name: amd-prefer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      # Tolerate the taint of the virtual node.
      - key: virtual-kubelet.io/provider
        operator: Exists
        effect: NoSchedule
      # Tolerate the taint on the Arm virtual node.
      - key: kubernetes.io/arch
        operator: Equal
        value: arm64
        effect: NoSchedule     
      # Preferentially schedule the workload to a node that uses the x86 architecture.
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                - amd64
      containers:
      - name: my-container
        image: nginx

FAQ

Scheduling priority of Arm and x86 nodes

The cluster scheduler prioritizes ECS instances over virtual nodes by default. Without custom scoring plugin weights, pods may still be scheduled to x86 ECS instances even with nodeAffinity set to prefer Arm nodes. The nodeAffinity settings in this topic only guarantee priority between virtual node architectures (Arm vs. x86), not between virtual nodes and ECS instances.

Can I use Arm preemptible instances?

Yes. See Use preemptible instances.

Network configuration for Arm virtual nodes

After creating an ACK cluster or , set the vSwitchIds field in the eci-profile to include a vSwitch in a zone that supports Arm-based instances.

What are the limitations of using Arm architecture nodes in ACK cluster?

App Marketplace add-ons are not supported on the Arm architecture. In the Component Center, only the following categories are supported:

Core components
Logging and monitoring
Storage
Network

References

Container Registry Enterprise Edition (ACR EE) lets you build multi-architecture container images.
To create and manage regular Arm ECS nodes, see Schedule workloads to Arm nodes.
To run big data tasks without managing underlying cluster resources, run Spark jobs on Arm virtual nodes.