All Products
Search
Document Center

Container Service for Kubernetes:Schedule ARM or multi-arch workloads to ARM-based virtual nodes

Last Updated:Jan 25, 2025

By default, the ACK cluster and schedule all workloads to x86-based virtual nodes. If your cluster includes ARM-based virtual nodes alongside other types, such as x86-based nodes, you can configure Kubernetes native scheduling to ensure that workloads compatible only with the ARM architecture are scheduled to ARM-based virtual nodes, or that multi-arch images are preferably scheduled to ARM-based virtual nodes.

Prerequisites

Considerations

If your cluster version is earlier than 1.24, when using nodeSelector or nodeAffinity to schedule applications to ARM nodes, you must also declare the taint toleration kubernetes.io/arch=arm64:NoSchedule in tolerations. If your cluster version is 1.24 or later, the scheduler can automatically detect the taint kubernetes.io/arch=arm64:NoSchedule on ARM nodes, and you do not need to declare tolerations additionally.

Billing

Pods deployed on elastic container instances that adopt the ARM architecture are billed based on the ECS instance type used to create the elastic container instances. The pods are not billed based on the usage of vCPUs and memory.

Important

After an Elastic Container Instance-based pod is created, you can run the kubectl describe pod command to view the YAML content of the pod. The k8s.aliyun.com/eci-instance-spec parameter indicates the ECS instance type used by the pod. The pod is billed based on the ECS instance type.

For more information about the ECS instance types that use the ARM architecture, refer to the following topics:

Step 1: Add ARM-based virtual nodes

Before deploying ARM workloads in the cluster, you need to create ARM-based virtual nodes. You can create virtual nodes that support the ARM architecture by configuring the ECI Profile. You can edit the eci-profile configuration file in the following two ways. For more information about the ECI Profile, see Configure eci-profile.

Console

  1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

  2. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose Configurations > ConfigMaps.

  3. Select the Namespace as kube-system. Locate eci-profile and click Edit. Modify the key value enableLinuxArm64Node to true, then click Confirm.

    image.png

    Note

    If all vSwitches of the cluster are in zones that do not support ARM instances, you need to create a vSwitch in a specified zone. After the vSwitch is created, add the ID of the vSwitch to vSwitchIds. For more information about creating a vSwitch in a specified zone, see Create and manage a vSwitch.

    Once the modification is complete, please wait approximately 30 seconds before viewing the newly created virtual node named virtual-kubelet-<zoneId>-linux-arm64 on the Node page.

kubectl

Prerequisites

Obtain the cluster KubeConfig and connect to the cluster using the kubectl tool.

Procedure

Execute the following command to edit the ConfigMap.

kubectl edit configmap eci-profile -n kube-system
  1. Add or modify the parameter enableLinuxArm64Node to true.

  2. Set vSwitchIds. Ensure that at least one vSwitch in the vSwitchIds used by the current cluster is in a zone that supports ARM instances.

    Note

    If all vSwitches of the cluster are in zones```html that do not support ARM instances, you need to create a vSwitch in a specified zone. After the vSwitch is created, add the ID of the vSwitch to vSwitchIds. For more information about creating a vSwitch in a specified zone, see Create and manage a vSwitch.

    Once the modification is finished, please wait approximately 30 seconds before viewing the newly created virtual node named virtual-kubelet-<zoneId>-linux-arm64 on the Node page.

Step 2: Schedule workloads to ARM virtual nodes

Schedule ARM-based workloads to ARM virtual nodes

If your cluster contains both ARM nodes and other nodes and your application supports only the ARM architecture, you can specify the application to run on ARM nodes to prevent the application pod from being scheduled to non-ARM nodes, which may cause the pod to fail to start. All ARM nodes have the label kubernetes.io/arch=arm64 by default. You can use either nodeSelector or nodeAffinity to specify the application to be deployed on ARM nodes.

nodeSelector

You can add the following constraint to the pod to schedule the pod to ARM-based virtual nodes using nodeSelector. nodeSelector specifies that this workload is scheduled only to nodes with the arm64 label. ARM-based virtual nodes in the ACK cluster have this label.

nodeSelector:
  kubernetes.io/arch: arm64 # Specify ARM nodes.

You can use the following example code to deploy a stateless application to ARM virtual nodes.

Expand to view YAML file

Note

The following YAML adds a toleration for kubernetes.io/arch=arm64:NoSchedule. If your cluster is version 1.24 or later of the ACK Pro Edition, the ACK scheduler will automatically detect this taint, and you do not need to declare the toleration additionally.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: only-arm
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      nodeSelector:
        kubernetes.io/arch: arm64 # Specify ARM nodes.
      tolerations:
      # Tolerate the taint of the virtual node.
        - key: virtual-kubelet.io/provider
          operator: Exists
          effect: NoSchedule
      # Tolerate the taint of the ARM-based virtual node.
        - key: kubernetes.io/arch
          operator: Equal
          value: arm64
          effect: NoSchedule
      containers:
      - name: nginx
        image: alibaba-cloud-linux-3-registry.cn-hangzhou.cr.aliyuncs.com/alinux3/nginx_optimized:20240221-1.20.1-2.3.0

nodeAffinity

Prerequisites

Enable the cluster virtual node scheduling policy, and ensure that the cluster version and component version meet the requirements.

Example operation

You can add the following constraint to the pod to specify the application to be deployed on ARM nodes using node affinity. This constraint specifies that the pod can only be scheduled to nodes with the label kubernetes.io/arm=arm64.

When this constraint is added to the pod spec, the scheduler automatically tolerates the taint kubernetes.io/arch=arm64:NoSchedule on the node.

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/arch
          operator: In
          values:
          - arm64

You can use the following example code to deploy a stateless application to ARM virtual nodes.

Expand to view YAML file

Note

The following YAML adds a toleration for kubernetes.io/arch=arm64:NoSchedule. If your cluster is version 1.24 or later of the ACK Pro Edition, the ACK scheduler will automatically detect this taint, and you do not need to declare the toleration additionally.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: only-arm
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                - arm64
       tolerations:
       # Tolerate the taint of the virtual node.
        - key: virtual-kubelet.io/provider
          operator: Exists
          effect: NoSchedule
       # Tolerate the taint of the ARM-based virtual node.
        - key: kubernetes.io/arch
          operator: Equal
          value: arm64
          effect: NoSchedule         
      containers:
      - name: nginx
        image: nginx

Schedule multi-arch images to ARM virtual nodes

Prerequisites

Enable the cluster virtual node scheduling policy, and ensure that the cluster version and component version meet the requirements.

Example operation

ACK cluster by default schedules all workloads to x86-based virtual nodes and waits for x86 node resources when x86 node resources are insufficient. If your application image is a multi-arch image, such as an image that supports both x86 and ARM architectures, you need to configure cross-architecture node scheduling.

For example, you can configure node affinity to preferably schedule workloads to ARM-based virtual nodes or x86-based virtual nodes. When the requested virtual nodes are insufficient, the scheduler will attempt to schedule workloads to other types of virtual nodes.

      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                - arm64

Preferentially schedule to ARM architecture

The following example preferably schedules workloads to ARM-based virtual nodes.

Expand to view YAML file

Note

The following YAML adds a toleration for kubernetes.io/arch=arm64:NoSchedule. If your cluster is version 1.24 or later of the ACK Pro Edition, the ACK scheduler will automatically detect this taint, and you do not need to declare the toleration additionally.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: arm-prefer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      # Tolerate the taint of the virtual node.
      - key: virtual-kubelet.io/provider
        operator: Exists
        effect: NoSchedule
      # Tolerate the taint of the ARM-based virtual node.
      - key: kubernetes.io/arch
        operator: Equal
        value: arm64
        effect: NoSchedule
      # Preferentially schedule the workload to an ARM-based node.
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                - arm64
      containers:
      - name: my-container
        image: nginx

Preferentially schedule to x86 architecture

The following example preferably schedules workloads to x86-based virtual nodes.

Expand to view YAML file

Note

The following YAML adds a toleration for kubernetes.io/arch=arm64:NoSchedule. If your cluster is version 1.24 or later of the ACK Pro Edition, the ACK scheduler will automatically detect this taint, and you do not need to declare the toleration additionally.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: amd-prefer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      # Tolerate the taint of the virtual node.
      - key: virtual-kubelet.io/provider
        operator: Exists
        effect: NoSchedule
      # Tolerate the taint of the ARM-based virtual node.
      - key: kubernetes.io/arch
        operator: Equal
        value: arm64
        effect: NoSchedule     
      # Preferentially schedule the workload to a x86-based node.
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                - amd64
      containers:
      - name: my-container
        image: nginx

FAQ

Why is the pod scheduled to x86-based ECS nodes when nodeAffinity is configured to preferentially schedule the pod to ARM-based nodes?

The cluster scheduler by default preferentially schedules to ECS nodes. When ECS node resources are insufficient, it schedules to virtual nodes. Without modifying the weight of the scheduler scoring plug-in, if there are sufficient resources on x86 ECS nodes in the cluster, even if nodeAffinity is configured to preferentially schedule to ARM-based nodes, the pod may still be scheduled to x86-based ECS nodes. Therefore, the nodeAffinity configuration in this topic can only ensure the priority of virtual nodes of different architectures (ARM/x86) but cannot ensure the priority between virtual nodes and ECS nodes.

Can ARM-based preemptible (Spot) instances be used?

ARM-based preemptible instances are available. For more information, see Use preemptible instances.

How do I configure a network that supports ARM-based virtual nodes after I create a cluster in the corresponding region?

After you create an ACK cluster in the corresponding zone, configure the vSwitchIds field in the eci-profile to select a virtual switch in a zone that supports ARM instances, ensuring that ARM-based virtual nodes are created.

What are the limitations of using ARM-based nodes in an ACK cluster?

Currently, the ARM architecture does not support components from the application marketplace. The component center supports only the following modules:

  • Core components

  • Logging and monitoring

  • Storage

  • Network

References