Container Service for Kubernetes:FAQs about node Auto Scaling

Index

Known limitations

Unable to accurately estimate 100% of node available resources

Because the underlying ECS system consumes some of its resources, the available memory for an instance is less than what the instance type definition suggests. For more information, see Why is the memory size different from the instance type definition after purchasing an instance?. Owing to this constraint, the resources projected by the cluster-autoscaler component may exceed the actual schedulable resources of the node, leading to less than perfect accuracy in estimations. When configuring pod requests, you need to consider the following precautions.

• Ensure pod requests for total resources, including CPU, memory, and disk, are less than the instance type definition. It is recommended to keep total requests below 70% of the node resources.

• In assessing a node's resource adequacy, the cluster-autoscaler component takes into account only the resources of Kubernetes pods, which encompasses both pending and DaemonSet pods. Should there be static pods on the node not generated by DaemonSets, it's necessary to pre-allocate resources for these pods.

• If the resources requested by the pods are large, such as exceeding 70% of the node resources, test in advance to confirm the pods can be scheduled to nodes of the same instance type to ensure feasibility.

Limited support for scheduling policies

The cluster-autoscaler component supports only a limited set of scheduling policies for determining if unschedulable pods can be scheduled on a node pool with auto scaling. For more information, see the scheduling policies used by the cluster-autoscaler component.

Node pools with multiple instance types do not support scaling out specific instance types

If your node pool is configured with multiple instance types, specifying an instance type for scaling out is not possible. The cluster-autoscaler component assesses the resource capacity for each instance type based on the minimum resources the scaling group can provide. For more information, see How the auto scaling component calculates the resources of a scaling group with multiple instance types configured.

Pods that depend on specific zones cannot trigger scale-out of multi-zone node pools

If your node pool spans multiple zones and a pod requires a specific zone, for instance, a pod that utilizes a persistent volume (PV) in a particular zone or a pod with a nodeSelector targeting that zone, the cluster-autoscaler component might not manage to scale out nodes in the needed zone. To explore additional scenarios where the cluster-autoscaler component is unable to scale out nodes, see Why can't the node auto scaling component scale out nodes?.

What scheduling policies does the cluster-autoscaler component use to determine whether unschedulable pods can be scheduled to a node pool with Auto Scaling enabled??

The cluster-autoscaler component uses the following scheduling policies:

• PodFitsResources

• GeneralPredicates

• PodToleratesNodeTaints

• MaxGCEPDVolumeCount

• NoDiskConflict

• CheckNodeCondition

• CheckNodeDiskPressure

• CheckNodeMemoryPressure

• CheckNodePIDPressure

• CheckVolumeBinding

• MaxAzureDiskVolumeCount

• MaxEBSVolumeCount

• Ready

• NoVolumeZoneConflict

What resources can the cluster-autoscaler component simulate and determine?What resources can be impersonated for judgment?

The cluster-autoscaler component currently enables simulation and determination of resources such as:

cpu
memory
sigma/eni
ephemeral-storage
aliyun.com/gpu-mem (shared GPUs only)
nvidia.com/gpu

If you need other resource types, see How do I configure custom resources for a node pool with auto scaling enabled?.

Why can't the node Auto Scaling component scale out nodes?

Check for the following scenarios:

• The node auto scaling feature is effective only for node pools with auto scaling enabled. Make sure that the node auto scaling feature is active and the node pool's scaling mode is set to automatic. For detailed instructions, see Enable node auto scaling.

• The instance types in the scaling group may not fulfill the resource requests of pods. The resources provided by ECS instance types comply with ECS specifications. During runtime, ACK reserves a certain amount of node resources to run Kubernetes components and system processes. This ensures that the OS kernel, system services, and Kubernetes daemons can run as expected. This causes the total resource capacity of the node to differ from the allocatable resources. For more information, see Node resource reservation policy.

  • During instance creation, resources are used for virtualization or consumed by the operating system. For more information, see Why is the memory size different from the instance type definition after purchasing an instance?.

  • Resources are occupied to run components such as the kubelet, kube-proxy, Terway, and container runtime. For more information, see Node resource reservation policy.

  • By default, system components are installed for each node. Therefore, the requested pod resources must be less than the resource capacity of the instance type.

• Pods with zone limits cannot trigger scale-out of node pools configured with multiple zones.

• Check if you have completed authorization as instructed. Authorization is required for each cluster involved in the scale-out activity. For more information about authorization, see Prerequisites.

• The following issues occur when you activate node autoscaling:

  • The instance fails to be added to the cluster and a timeout error occurs.

  • The node is NotReady and a timeout error occurs.

  To ensure accurate scaling, the auto scaling component does not perform any scaling activities before it fixes the abnormal nodes.

• Since there are no nodes in the cluster, the cluster-autoscaler component cannot function. It is recommended to configure a minimum of two nodes in your node pool during creation to guarantee the smooth functioning of cluster components.

  If your scenario requires scaling out from zero nodes or scaling in to zero nodes, you can use instant elasticity for nodes. For more information, see Enable instant elasticity for nodes.

How does the Auto Scaling component calculate the resources of a scaling group that has multiple instance types configured?

For a scaling group with multiple instance types, the auto scaling component evaluates the resource capacity based on the least amount of resources the scaling group can provide.

For example, consider a scaling group that utilizes two instance types: one offering 4 vCores and 32 GB of memory, and the other providing 8 vCores and 16 GB of memory. The auto scaling component assumes the scaling group can add instances with at least 4 vCores and 16 GB of memory. Thus, if the requests for resources of a pending pod exceed 4 vCores or 16 GB, the scaling group will not initiate a scale-out.

If you configure multiple instance types but need to consider resource reservation, see Why can't the node auto scaling component scale out nodes?.

How do I choose between multiple node pools with Auto Scaling enabled during a scaling activity?

When pods cannot be scheduled to nodes, the auto scaling component simulates the scheduling process based on the scaling group configurations, which include labels, taints, and instance specifications. Should a scaling group fulfill these requirements, it is chosen for the scale-out operation. In cases where multiple node pools with auto scaling enabled satisfy the simulation criteria, the node auto scaling component defaults to the least-waste principle. This means the node pool with the smallest amount of resources left after the simulated scale-out is selected.

How do I configure custom resources for a node pool with Auto Scaling enabled?

To enable the auto scaling component to recognize custom resources provided by a node pool with auto scaling enabled, or to identify the amounts of specific resource types provided by the node pool, add an ECS tag with the following prefix to the node pool:

k8s.io/cluster-autoscaler/node-template/resource/{resource name}:{resource size}

Example:

k8s.io/cluster-autoscaler/node-template/resource/enormouspages-1Gi:2Gi

Why does the configuration of Auto Scaling for a node pool fail?

The issue may occur due to the following causes:

• The node pool is a default node pool and does not support the node auto scaling feature.

• The node pool already contains manually added nodes. You need to remove the manually added nodes first. We recommend creating a new node pool with auto scaling enabled.

• The node pool includes instances with a subscription billing method. However, the node auto scaling feature is not compatible with nodes billed through subscription.

Why can't thecluster-autoscaler component scale in nodes?

Check for the following scenarios:

• The requested resource threshold of each pod on the node is higher than the specified scale-in threshold.

• The pod runs in the kube-system namespace on the node.

• A scheduling policy forces the pods to run on the current node, preventing them from being scheduled to other nodes.

• The pods on the node are governed by a PodDisruptionBudget and have met the minimum value specified by the PodDisruptionBudget.

You can find more FAQs about the node auto scaling component in the open-source community here.

How do I enable or disable eviction for a specific DaemonSet?

The cluster-autoscaler component decides whether to evict DaemonSet pods based on the setting of Whether To Enable Daemonset Pod Eviction. These settings are applicable to the entire cluster and affect all DaemonSet pods within it. For more information, see Step 1: Enable node auto scaling. To determine if a specific DaemonSet pod should be evicted, you can add the annotation "cluster-autoscaler.kubernetes.io/enable-ds-eviction": "true" to that DaemonSet pod.

Similarly, if the annotation "cluster-autoscaler.kubernetes.io/enable-ds-eviction": "false" is added to the DaemonSet pod, it will prevent the Cluster Autoscaler from evicting the DaemonSet pod.

What types of pods can prevent thecluster-autoscaler component from removing nodes?

If a pod is not created by a native Kubernetes controller, such as a deployment, ReplicaSet, job, or StatefulSet, or if the pods on the node cannot be safely terminated or migrated, the cluster-autoscaler component may prevent the node from being removed. For more information, see What types of pods can prevent CA from removing a node?.

Does the cluster-autoscaler component support CRD?

The cluster-autoscaler component currently only supports standard Kubernetes objects and does not accommodate Kubernetes CRDs.

How do I delay the scale-out reaction time of thecluster-autoscaler component for unschedulable pods?

You can set a scale-out delay time for each pod by using the annotation cluster-autoscaler.kubernetes.io/pod-scale-up-delay. If Kubernetes does not schedule the pods after the delay ends, CA may consider scaling them out. Example of an annotation: "cluster-autoscaler.kubernetes.io/pod-scale-up-delay": "600s".

How do I use pod annotations to affect the scale-in of nodes by thecluster-autoscaler component?

You can determine if a pod prevents a node from being scaled in by the cluster-autoscaler component.

• To prevent a node from being scaled in, add the annotation "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" to the pod.

• To allow a node to be scaled in, add the annotation "cluster-autoscaler.kubernetes.io/safe-to-evict": "true" to the pod.

How do I specify that a node cannot bescaled in by the cluster-autoscaler component?

To prevent a node from being scaled in by the cluster-autoscaler, configure the annotation "cluster-autoscaler.kubernetes.io/scale-down-disabled": "true" for the target node. The following command shows an example of how to add the annotation.

kubectl annotate node <nodename> cluster-autoscaler.kubernetes.io/scale-down-disabled=true

How do I upgrade thecluster-autoscaler component to the latest version?

If your cluster has auto scaling enabled, you can upgrade the cluster-autoscaler component using the method below.

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose Nodes > Node Pools.

3. Click Node Scaling on the right side of the page, then click Edit , and finally click OK at the bottom of the panel to upgrade the component to the latest version.

What operations trigger the automatic update of thecluster-autoscaler component?

To maintain the real-time configuration, version, and compatibility of the cluster-autoscaler component with the cluster, the following operations will automatically update the cluster-autoscaler component:

• Update the auto scaling configuration.

• Create, delete, or update node pools for which auto scaling is enabled.

• The cluster is updated.

The ACK managed cluster has completed role authorization, butnode scaling activities still cannot run properly?

The issue may be caused by the absence of addon.aliyuncsmanagedautoscalerrole.token in the secret dictionary of the kube-system namespace of the cluster. If this token is missing, use one of the following methods to add the token:

• or Submit a ticket or to request technical support.

• Manually add the AliyunCSManagedAutoScalerRolePolicy permission: By default, ACK assumes the worker RAM role to use the relevant capabilities. Follow these steps to manually assign the AliyunCSManagedAutoScalerRolePolicy permission to the worker role:

  1. On the Cluster List page, click the name of the desired cluster. Then, in the left-side navigation pane, select Cluster Information.

  2. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose Nodes > Node Pools.

  3. On the Node Pools page, click Enable next to Node Scaling.

     

  4. Complete the authorization of the KubernetesWorkerRole role and the AliyunCSManagedAutoScalerRolePolicy system policy as prompted. The following figure shows the console page where you can complete the authorization.

     

  5. Manually restart the Deployment cluster-autoscaler (auto scaling of nodes) or ack-goatscaler (instant elasticity of nodes) within the kube-system namespace to ensure immediate application of the permission.