Enable auto scaling for a node pool - Container Service for Kubernetes

Prerequisites

A node pool is created.
Read Node scaling to understand its working principles and features.

Step 1: Configure RAM permissions

Create a RAM user and grant it the following custom permissions. For more information, see Grant permissions to access clusters and cloud resources by using RAM.

Custom policy

{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "ess:DescribeScalingGroups",
        "ess:DescribeScalingInstances",
        "ess:DescribeScalingActivities",
        "ess:DescribeScalingConfigurations",
        "ess:DescribeScalingRules",
        "ess:DescribeScheduledTasks",
        "ess:DescribeLifecycleHooks",
        "ess:DescribeNotificationConfigurations",
        "ess:DescribeNotificationTypes",
        "ess:DescribeRegions",
        "ess:CreateScalingRule",
        "ess:ModifyScalingGroup",
        "ess:RemoveInstances",
        "ess:ExecuteScalingRule",
        "ess:ModifyScalingRule",
        "ess:DeleteScalingRule",
        "ecs:DescribeInstanceTypes",
        "ess:DetachInstances",
        "ess:CompleteLifecycleAction",
        "ess:ScaleWithAdjustment",
        "ess:DescribePatternTypes",
        "vpc:DescribeVSwitches",
        "cs:DeleteClusterNodes",
        "cs:DescribeClusterNodes",
        "cs:DescribeClusterNodePools",
        "cs:DescribeClusterNodePoolDetail",
        "cs:DescribeTaskInfo",
        "cs:ScaleClusterNodePool",
        "cs:RemoveNodePoolNodes",
        "ecs:DescribeAvailableResource",
        "ecs:DescribeInstanceTypeFamilies",
        "ecs:DescribeInstances",
        "cs:GetClusterAddonInstance",
        "cs:DescribeClusterDetail",
        "ecs:DescribeCapacityReservations",
        "ecs:DescribeElasticityAssurances",
        "ecs:DescribeImages"
      ],
      "Resource": [
        "*"
      ],
      "Effect": "Allow"
    }
  ]
}

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Configurations > Secrets.

On the Secrets page, click Create from YAML. Enter the following YAML to create a secret named alibaba-addon-secret.

Terway uses the AccessKey pair in this secret to access cloud services. Skip if the secret already exists.

apiVersion: v1
kind: Secret
metadata:
  name: alibaba-addon-secret
  namespace: kube-system
type: Opaque
stringData:
  access-key-id: <AccessKey ID of the RAM user>
  access-key-secret: <AccessKey secret of the RAM user>

Step 2: Configure node scaling

Node autoscaling

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.
On the Node Pools page, next to Node Scaling, click Enable.
If you are using node autoscaling for the first time, follow the on-screen instructions to activate the Auto Scaling service.

On the Node Scaling Configuration page, set Node Scaling Plan to Auto Scaling, configure the parameters, and then click OK.

You can switch the scaling solution after it is selected. To switch, change the selection to node instant scaling, read the on-screen prompts carefully, and follow the instructions to complete the operation.

Parameter	Description
Node Pool Scale-out Policy	Random Policy: If multiple node pools are eligible for scale-out, one is chosen at random. Default Policy: If multiple node pools are eligible for scale-out, it chooses the one that results in the least resource waste. Priority-based Policy: If multiple node pools are eligible for scale-out, it chooses the one with the highest priority. Node pool priority is defined by the Node Pool Scale-out Priority parameter.
Node Pool Scale-out Priority	Sets the scale-out priority for node pools. This parameter takes effect only when Node Pool Scale-out Policy is set to Priority-based Policy. Value range: an integer from 1 to 100. A larger value indicates a higher priority. You must click Add next to the parameter, select an autoscaled node pool, and set a priority for it. If no autoscaled node pools are available, you can ignore this parameter for now and set the priority after you complete Step 2: Configure autoscaled node pools.
Scaling Sensitivity	The interval at which the system checks whether to perform scaling. The default value is 60s.
During auto scaling, the scaling component automatically triggers scale-out events based on scheduling status. Important ECS nodes: The scaling component can scale in a node only when the conditions for Scale-in Threshold, Scale-in Trigger Delay, and Cooldown Period are all met. GPU nodes: The scaling component can scale in a GPU node only when the conditions for GPU Scale-in Threshold, Scale-in Trigger Delay, and Cooldown Period are all met.
Allow Scale-in	Specifies whether to allow node scale-in. If this is turned off, scale-in-related configurations do not take effect. Use this setting with caution.
Scale-in Threshold	The ratio of resource requests to the total resource capacity of a node in a node pool with node autoscaling enabled. A node is eligible for scale-in only if its CPU and memory resource utilization are both below the Scale-in Threshold.
GPU Scale-in Threshold	The scale-in threshold for GPU instances. A GPU instance is eligible for scale-in only when its CPU, memory, and GPU utilization all fall below the configured GPU Scale-in Threshold.
Scale-in Trigger Delay	The delay between when a node becomes eligible for scale-in and when the scale-in operation is performed. Unit: minutes. Default value: 10 minutes. Important The scaling component can perform a node scale-in only after the Scale-in Threshold condition is met and the Scale-in Trigger Delay duration has passed.
Cooldown Period	The period after the most recent scale-out event during which the scaling component will not perform a scale-in. During the cooldown period, scale-in actions are paused, but the system continues to evaluate nodes for eligibility. Once the cooldown ends, any node that has met the scale-in conditions (threshold and deferral time) is scaled in. For example, with a 10-minute cooldown and a 5-minute deferral, no nodes are removed for 10 minutes after a scale-out. If a node becomes eligible at minute 2, it will be removed at minute 10, as it has been eligible for 8 minutes, which exceeds the 5-minute deferral.

Advanced parameters

Parameter	Description
Pod Termination Timeout	Maximum wait time for pod termination during scale-in. Unit: seconds. If a pod is not evicted before the timeout, the node is not released.
Minimum Pod Replicas	Scale-in protection threshold. Nodes with `ReplicationController` or `ReplicaSet` pods are not scaled in if the replica count falls below this value. Applies only to `ReplicationController` and `ReplicaSet` pods, not `StatefulSet` or `DaemonSet`.
Enable DaemonSet Pod Eviction	When enabled, DaemonSet pods are evicted when their node is scaled in.
Skip nodes with pods in the kube-system namespace	When enabled, nodes with kube-system namespace pods are excluded from scale-in. Note This does not apply to DaemonSet pods or mirror pods.

Node instant scaling

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.
On the Node Pools page, next to Node Scaling, click Enable.
If this is the first time you are using auto scaling, follow the on-screen instructions to activate the Auto Scaling service.

On the Node Scaling Configuration page, set Node Scaling Plan to Instant Scaling, configure the scaling parameters, and click OK.

The scaling component automatically triggers node scale-outs based on scheduling conditions.

Switch the Node Scaling Plan to Auto Scaling at any time. Follow the on-screen prompts. This feature is in beta. To participate, submit a ticket.

Parameter	Description
Scale-in Threshold	The ratio of resource requests to the total resource capacity of a node in a node pool with node autoscaling enabled. A node is eligible for scale-in only if its CPU and memory resource utilization are both below the Scale-in Threshold.
GPU Scale-in Threshold	The scale-in threshold for GPU instances. A GPU instance is eligible for scale-in only when its CPU, memory, and GPU utilization all fall below the configured GPU Scale-in Threshold.
Scale-in Trigger Delay	The delay between when a node becomes eligible for scale-in and when the scale-in operation is performed. Unit: minutes. Default value: 10 minutes. Important The scaling component can perform a node scale-in only after the Scale-in Threshold condition is met and the Scale-in Trigger Delay duration has passed.

Advanced configurations

Parameter	Description
Pod Termination Timeout	Maximum wait time for pod termination during scale-in. Unit: seconds. If a pod is not evicted before the timeout, the node is not released.
Minimum Pod Replicas	Scale-in protection threshold. Nodes with `ReplicationController` or `ReplicaSet` pods are not scaled in if the replica count falls below this value. Applies only to `ReplicationController` and `ReplicaSet` pods, not `StatefulSet` or `DaemonSet`.
Enable DaemonSet Pod Eviction	When enabled, DaemonSet pods are evicted when their node is scaled in.
Skip nodes with pods in the kube-system namespace	When enabled, nodes with kube-system namespace pods are excluded from scale-in. Note This does not apply to DaemonSet pods or mirror pods.

Step 3: Configure a node pool

You can enable auto scaling for an existing node pool by setting its Scaling Mode to Auto, or create a new node pool with this feature enabled. For more information, see Create and manage a node pool. The following table describes the key parameters.

Parameter	Description
Scaling Mode	Manual: ACK adjusts the number of nodes in the node pool according to the configured Expected Number of Nodes to always maintain the number of nodes at the Expected Number of Nodes. For more information, see Manually scale a node pool. Auto: When the cluster's capacity planning cannot meet application pod scheduling needs, ACK automatically scales node resources based on the configured minimum and maximum number of instances. For clusters of version 1.24 and later, node instant scaling is enabled by default. For clusters earlier than version 1.24, node autoscaling is enabled by default. For more information, see Node scaling. Important If you enable scale-in, the system will automatically remove nodes based on resource usage, which may cause business interruptions. We recommend configuring the scaling policy appropriately and setting up graceful shutdown for your application containers. During automatic scale-in, local data on the node will be destroyed along with the node. Do not use methods such as HostPath to write business data to the host directory.
Instances	The scalable Min. Instances and Max. Instances in the node pool do not include your existing instances. Note If Min. Instances is greater than 0, the corresponding number of ECS instances are automatically created after the scaling group takes effect. We recommend setting Max. Instances to a value that is not less than the current number of nodes in the node pool. Otherwise, the nodes in the node pool will be scaled in immediately after the auto scaling feature is enabled.
Instance-related parameters	When a node pool scales out, it allocates instances from the configured ECS instance family. To improve the success rate of scale-outs, select multiple instance types across multiple zones to avoid unavailability or stock shortages. The specific instance type for scale-out is determined by the configured Scaling Policy. To ensure business stability and accurate resource scheduling, do not mix GPU and non-GPU instance types in the same node pool. You can configure the instance types used for scale-out in the following two ways: Specific types: Specify instance types based on dimensions such as vCPU, memory, instance family, and CPU architecture (the instance's CPU architecture must be consistent with the OS image architecture). When using Terway, you can view the maximum number of pods per node for the target instance type in the instance type list. Generalization configuration: Select a list of instance types to use or exclude based on attributes (such as vCPU and memory) to further improve the scale-out success rate. You can refer to the elasticity strength recommendations in the console or view the node pool elasticity strength after the node pool is created. For information about unsupported instance types and node configuration recommendations, see ECS instance type configuration recommendations. Cloud resources and billing: ECS Instance, GPU-accelerated Instance
Operating System	For autoscaled nodes, Alibaba Cloud Linux, Windows, and Windows Core images are supported. When you select a Windows or Windows Core image, the system automatically adds the taint `{ effect: 'NoSchedule', key: 'os', value: 'windows' }`.
Node Labels	Node labels added to the cluster configuration are automatically applied to nodes created during a scale-out. Important Node autoscaling recognizes node labels and taints only after they are mapped to node pool tags, and there is a limit to the number of node pool tags. Therefore, for an autoscaled node pool, ensure that the total number of configured ECS tags, taints, and node labels does not exceed 12.
scaling policy	Configure how the node pool selects instances during node scaling. Priority-based Policy: Scales based on the priority of the vSwitches configured for the cluster (vSwitch priority decreases from top to bottom). If an instance cannot be created in the zone of a higher-priority vSwitch, the next-priority vSwitch is automatically used. Cost Optimization: Scales based on the vCPU unit price from low to high. If the node pool uses Preemptible Instance, spot instances are prioritized. You can also configure the Percentage of On-Demand Instances (%). If spot instance types cannot be created due to inventory or other reasons, pay-as-you-go instances are automatically used as a supplement. Distribution Balancing: Distributes ECS instances evenly across multiple zones, but only in multi-zone scenarios. If the distribution becomes unbalanced due to insufficient inventory or other reasons, you can perform a rebalancing operation.
Use Pay-as-you-go Instances When Spot Instances Are Insufficient	This requires the billing method to be set to Spot Instance. When enabled, if not enough spot instances can be created due to price, inventory, or other reasons, ACK will automatically try to create on-demand instances as a supplement. Cloud resources and billing: ECS Instance
Enable Supplemental Spot Instance	This requires the billing method to be set to Spot Instance. When enabled, upon receiving a system message that a spot instance is about to be reclaimed (5 minutes before reclamation), ACK will attempt to scale out a new instance as compensation. Compensation successful: ACK drains the old node and removes it from the cluster. Compensation failed: ACK does not drain the old node. The instance will still be reclaimed and released after 5 minutes. When inventory is restored or price conditions are met, ACK will automatically purchase an instance to maintain the desired number of nodes. For more information, see Best practices for spot instance node pools. The proactive release of spot instances can cause business disruptions. To improve the compensation success rate, we recommend also enabling Use Pay-as-you-go Instances When Spot Instances Are Insufficient. Cloud resources and billing: ECS Instance
Scaling Mode	This requires node pool Auto Scaling to be enabled and the Scaling Mode to be set to Auto. Standard: Scales by creating and releasing ECS instances. Swift: Scales by creating, stopping, and then restarting ECS instances. This allows for faster scaling by restarting stopped instances when needed. You are not charged for computing resources when an ECS instance is stopped, only for storage fees (except for instance families with local storage capabilities, such as big data and local SSD types). For billing details and related notes on the ECS instance stop mode, see Economical mode.
Taints	Prevents pods from being scheduled to the node.

Step 4: (Optional) Verify the results

After you complete these steps, the node autoscaling feature is enabled. The node pool status indicates that auto scaling is active, and the cluster-autoscaler is automatically installed in your cluster.

Auto scaling for the node pool

On the Node Pools page, the list shows the node pools that have auto scaling enabled.

Installed cluster-autoscaler

In the left navigation pane of the cluster management page, choose Workload > Deployments.
In the kube-system namespace, find the cluster-autoscaler.