Specify that Arm or multi-architecture workloads are scheduled to Arm virtual nodes - Container Service for Kubernetes

Prerequisites

Cluster:

An ACK Serverless cluster that runs version 1.20 or later is created. For more information, see Create an ACK Serverless cluster and Manually upgrade a cluster.

Note
Arm-based ECS instances are available only in specific regions and zones. Make sure that your cluster is in a supported region. To view the regions and zones where Arm-based ECS instances are available, see Overview of regions and zones where ECS instance types are available for purchase.
Add-on: The ack-virtual-node add-on v2.9.0 or later is installed. For more information, see ACK Virtual Node.

Usage notes

If your cluster runs a version earlier than 1.24 and you use nodeSelector or nodeAffinity to schedule an application to an Arm node, you must also declare a toleration for the kubernetes.io/arch=arm64:NoSchedule taint in tolerations. If your cluster runs version 1.24 or later, the scheduler automatically identifies the kubernetes.io/arch=arm64:NoSchedule taint on Arm nodes, and you do not need to declare additional tolerations.

Billing

ARM-based ECS instance types and pricing:

Step 1: Add Arm virtual nodes

Before you deploy Arm workloads in a cluster, create Arm virtual nodes. Configure Elastic Container Instance (ECI) Profile to create virtual nodes that support the Arm architecture. You can edit the eci-profile ConfigMap in either of the following two ways. For more information about ECI Profile, see Configure an eci-profile.

Console

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Configurations > ConfigMaps.
Set the Namespace to kube-system. Find eci-profile and click Edit. Change the value of the enableLinuxArm64Node key to true, and then click OK.

Note
If none of the zones that contain your cluster's existing vSwitches support Arm-based ECS instances, first create a vSwitch in a supported zone. After the vSwitch is created, add its ID to vSwitchIds. For instructions, see Create and manage vSwitches.

kubectl

Prerequisites

Connect to an ACK cluster using kubectl.

Procedure

Run the following command to edit the ConfigMap.

kubectl edit configmap eci-profile -n kube-system

Add the enableLinuxArm64Node parameter or modify its value to true.
Configure vSwitchIds and make sure that the vSwitchIds of the current cluster include at least one vSwitch in a zone that supports Arm-based ECS instances.

Note
If none of the zones that contain your cluster's existing vSwitches support Arm-based ECS instances, first create a vSwitch in a supported zone. After the vSwitch is created, add its ID to vSwitchIds. For instructions, see Create and manage vSwitches.

Step 2: Schedule workloads to Arm virtual nodes

Schedule Arm workloads to Arm virtual nodes

If your cluster has both Arm nodes and non-Arm nodes, and your application supports only the Arm architecture, you can specify that the application runs on Arm nodes. This prevents the application pods from being scheduled to non-Arm nodes and failing to start. By default, all Arm nodes have the label kubernetes.io/arch=arm64. You can use either nodeSelector or nodeAffinity to deploy the application to Arm nodes.

nodeSelector

You can add the following constraint to a pod and use nodeSelector to schedule the pod to an Arm virtual node. nodeSelector schedules this workload only to nodes that have the arm64 label. All Arm virtual nodes in ACK Serverless cluster have this label.

nodeSelector:
  kubernetes.io/arch: arm64 # Specify an Arm node.

You can use the following sample code to deploy a stateless application to an Arm virtual node.

Expand to view the YAML file

Note

The following YAML file adds a toleration for the kubernetes.io/arch=arm64:NoSchedule taint. If your cluster is an ACK managed Pro cluster that runs version 1.24 or later, the ACK scheduler automatically identifies this taint, and you do not need to declare an additional toleration.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: only-arm
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      nodeSelector:
        kubernetes.io/arch: arm64 # Specify an Arm node.
      tolerations:
      # Tolerate the virtual node taint.
        - key: virtual-kubelet.io/provider
          operator: Exists
          effect: NoSchedule
      # Tolerate the taint on the Arm virtual node.
        - key: kubernetes.io/arch
          operator: Equal
          value: arm64
          effect: NoSchedule
      containers:
      - name: nginx
        image: alibaba-cloud-linux-3-registry.cn-hangzhou.cr.aliyuncs.com/alinux3/nginx_optimized:20240221-1.20.1-2.3.0

nodeAffinity

Prerequisites

You have completed the operations described in Enable pod scheduling policies on virtual nodes, and the cluster version and add-on version meet the requirements.

Example

You can add the following constraint to a pod and use node affinity to deploy the application on Arm nodes. This constraint specifies that the pod can be scheduled only to nodes with the label kubernetes.io/arm=arm64.

When a pod specification contains this constraint, the scheduler automatically tolerates the kubernetes.io/arch=arm64:NoSchedule taint on the node.

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/arch
          operator: In
          values:
          - arm64

You can use the following sample code to deploy a stateless application to an Arm virtual node.

Expand to view the YAML file

Note

The following YAML file adds a toleration for the kubernetes.io/arch=arm64:NoSchedule taint. If your cluster is an ACK managed Pro cluster that runs version 1.24 or later, the ACK scheduler automatically identifies this taint, and you do not need to declare an additional toleration.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: only-arm
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                - arm64
      tolerations:
       # Tolerate the virtual node taint.
        - key: virtual-kubelet.io/provider
          operator: Exists
          effect: NoSchedule
       # Tolerate the taint on the Arm virtual node.
        - key: kubernetes.io/arch
          operator: Equal
          value: arm64
          effect: NoSchedule         
      containers:
      - name: nginx
        image: nginx

Schedule multi-architecture images to Arm virtual nodes

Prerequisites

You have completed the operations described in Enable pod scheduling policies on virtual nodes, and the cluster version and add-on version meet the requirements.

Example

ACK Serverless cluster schedules all workloads to x86 virtual nodes by default. When x86 node resources are insufficient, the cluster keeps waiting for those resources. If your application image is a multi-architecture image, such as one that supports both x86 and Arm, configure node scheduling across the two architectures.

For example, you can configure node affinity to preferentially schedule a workload to Arm or x86 virtual nodes, and to try to schedule it to a virtual node of the other architecture when resources of the target type are insufficient.

      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                - arm64

Prefer Arm architecture

The following example declares a preference for scheduling workloads to Arm virtual nodes.

Expand to view the YAML file

Note

The following YAML file adds a toleration for the kubernetes.io/arch=arm64:NoSchedule taint. If your cluster is an ACK managed Pro cluster that runs version 1.24 or later, the ACK scheduler automatically identifies this taint, and you do not need to declare an additional toleration.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: arm-prefer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      # Tolerate the virtual node taint.
      - key: virtual-kubelet.io/provider
        operator: Exists
        effect: NoSchedule
      # Tolerate the taint on the Arm virtual node.
      - key: kubernetes.io/arch
        operator: Equal
        value: arm64
        effect: NoSchedule
      # Prefer scheduling to a node that uses the Arm architecture.
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                - arm64
      containers:
      - name: my-container
        image: nginx

Prefer x86 architecture

Expand to view the YAML file

Note

The following YAML file adds a toleration for the kubernetes.io/arch=arm64:NoSchedule taint. If your cluster is an ACK managed Pro cluster that runs version 1.24 or later, the ACK scheduler automatically identifies this taint, and you do not need to declare an additional toleration.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: amd-prefer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      # Tolerate the virtual node taint.
      - key: virtual-kubelet.io/provider
        operator: Exists
        effect: NoSchedule
      # Tolerate the taint on the Arm virtual node.
      - key: kubernetes.io/arch
        operator: Equal
        value: arm64
        effect: NoSchedule     
      # Prefer scheduling to a node that uses the x86 architecture.
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                - amd64
      containers:
      - name: my-container
        image: nginx

FAQ

Why are pods scheduled to x86 ECS nodes when I configure nodeAffinity to preferentially schedule them to Arm nodes?

By default, the cluster scheduler prioritizes ECS nodes and schedules workloads to virtual nodes when ECS node resources are insufficient. If you do not change the weights of the scheduler scoring plugins, pods may be scheduled to x86 ECS nodes that have sufficient resources, even when nodeAffinity is configured to preferentially schedule them to Arm nodes. Therefore, the nodeAffinity configuration in this topic guarantees priority only among virtual nodes of different architectures (Arm and x86), and does not guarantee priority between virtual nodes and ECS nodes.

Can I use Arm-based spot instances?

Arm-based spot instances are available. For instructions, see Use preemptible instances.

After I create a cluster in a supported region, how do I configure the network to create virtual nodes in zones that support Arm-based ECS instances?

After you create ACK Serverless cluster in the corresponding zone, configure the vSwitchIds field in eci-profile to select vSwitches in zones that support Arm-based ECS instances. This ensures that the virtual nodes created support the Arm architecture.

What are the limits of using Arm nodes in ACK Serverless cluster?

App Marketplace add-ons are not supported on the Arm architecture. Component Center supports add-ons only in the following categories:

Core components
Logging and monitoring
Storage
Networking

References

You can use Alibaba Cloud Container Registry Enterprise Edition (ACR EE) to build multi-architecture container images. For instructions, see Build multi-architecture container images.
If you need to create and manage standard Arm ECS nodes, see Schedule workloads to ARM nodes.
If you have big data jobs to run and do not want to handle O&M for the underlying cluster resources, you can Run Spark jobs on ARM-based virtual nodes.