Container Service for Kubernetes:Scale a node pool

Overview of node pool scaling

The expected number of nodes refers to the number of nodes to be retained in a node pool. It indicates the number of nodes in the node pool when the node pool reaches the final state. After you specify the expected number of nodes in a node pool, the nodes in the node pool are automatically scaled to the specified number.

• Scale out the node pool: Set the expected number of nodes to a value that is greater than the current value. Then, the node pool is automatically scaled out. If you want to scale out a node pool, we recommend that you use this method. This way, the system can automatically retry when it fails to add nodes to the node pool.
  Note The scale-out configuration varies based on the node pool configuration. The instance type and zone of the nodes depend on the scaling policy that is used. For more information about node pool scaling policies, see Scaling policies.
  The system performs the following steps to scale out a node pool:

  1. Add ECS instances: Auto Scaling, the underlying service used by ACK to scale node pools, automatically creates ECS instances. After you modify the expected number of nodes, ACK automatically changes the expected number of instances in the scaling group of Auto Scaling to scale out the node pool. The status of the node pool changes to Expanding. After Auto Scaling creates ECS instances, the status of the node pool changes to Activated. For information about the expected number of instances, see Expected number of instances.
     Important Instances of GPU-accelerated ECS Bare Metal Instance families ebmgn7 and ebmgn7e cannot automatically delete the Multi-Instance GPU (MIG) configuration. When ACK adds instances of the preceding instance families, ACK automatically resets the MIG configuration retained on the instances. The reset may be time-consuming. In this case, you may fail to add the instances to a cluster.

     • For more information about how to troubleshoot the issue, see What do I do if I fail to add ECS Bare Metal instances that are equipped with NVIDIA A100 GPUs?.
     • For more information about the ebmgn7e instance family, see ebmgn7e, GPU-accelerated compute-optimized ECS Bare Metal Instance family
     • For more information about how to enable the MIG feature, see Use a node pool to partition an NVIDIA A100 GPU into multiple GPU instances
  2. Add the ECS instances to the cluster: After Auto Scaling creates ECS instances, the ECS instances automatically run the cloud-init script maintained by ACK to initialize the nodes and add the nodes to the node pool. The operational log is saved to the /var/log/messages file on each node. You can log on to a node and run the grep cloud-init /var/log/messages command to view the log.
     Note

     • After a node is added to the node pool, the operational log in the /var/log/messages file is automatically deleted. Therefore, the log records only information about failures to add nodes to the node pool.
     • If the system fails to add a node to the node pool, the relevant log data in the /var/log/messages file is synchronized to the task result. You can view the task details on the Cluster Tasks tab of the cluster details page.
• Scale in the node pool: Set the expected number of nodes to a value that is smaller than the current value. Then, the node pool is automatically scaled in.
  Note

  • When the system scales in a node pool:
    • If the scaling policy is set to Priority, the newly created ECS instances are preferably removed from the scaling group.
    • If the scaling policy is set to Distribution Balancing, the zones where the ECS instances are deployed are filtered based on the policy. Then, the newly created ECS instances are preferably removed from the scaling group to ensure that the numbers of ECS instances in different zones of the scaling group are close or the same.
    • If the scaling policy is set to Cost Optimization, ECS instances are removed from the scaling group in the descending order of vCPU prices.
    If you want to remove specified nodes, see Remove a node.
  • When a scale-in activity is triggered by changing the expected number of nodes, ACK can remove nodes without the need to drain the nodes first. If you want to drain the nodes before they are removed, refer to Remove a node to drain and remove the nodes.

Procedure

1. Log on to the ACK console and click Clusters in the left-side navigation pane.
2. On the Clusters page, click the name of a cluster and choose Nodes > Node Pools in the left-side navigation pane.
3. Find the node pool that you want to scale and click Scale in the Actions column.
4. Create the AliyunOOSLifecycleHook4CSRole role to grant permissions to access Operation Orchestration Service (OOS).
   1. Click AliyunOOSLifecycleHook4CSRole.
      Note

      • If the current account is an Alibaba Cloud account, click AliyunOOSLifecycleHook4CSRole.
      • If the current account is a RAM user, make sure the Alibaba Cloud account is assigned the AliyunOOSLifecycleHook4CSRole role. Then, attach the AliyunRAMReadOnlyAccess system policy to the RAM user. For more information, see Grant permissions to the RAM user.
   2. On the Cloud Resource Access Authorization page, click Agree to Authorization.
5. Set the Expected Nodes parameter and click Confirm.
   • If the status of the node pool in the node pool list displays Expanding, the system is scaling out the node pool. If the status of the node pool changes to Activated, the node pool is scaled out.
   • If the status of the node pool in the node pool list displays Removing, the system is scaling in the node pool. If the status of the node pool changes to Activated, the node pool is scaled in.

Unrecommended operations and solutions

Error codes for scaling failures and solutions

Node pool scaling may fail due to reasons such as insufficient inventory. You can click the name of your ACK cluster on the Clusters page of the ACK console, click the Cluster Tasks tab, and then click View Cause to view the cause of a node pool scaling failure.

References

• Add existing ECS instances to an ACK cluster
• Remove a node
• Manage node pools
• Best practice for node pools