Documentation
Home Page Container Service for Kubernetes ACK Managed and Dedicated Clusters User Guide Nodes and node pools Node pools Manually scale a node pool

Manually scale a node pool

Updated at:
Copy as MD
Product

Set the expected node count for a node pool to scale out for traffic growth or scale in to cut costs.

ACK supports node auto scaling or node instant scaling to adjust node resources based on workload.

How it works

The Expected Nodes field sets the target number of nodes for a node pool. Once you submit a change, ACK drives the actual node count to that target — creating or removing ECS instances as needed.

Scale out — increase Expected Nodes above the current count. ACK provisions ECS instances via Auto Scaling, runs the cloud-init script on each, then adds them to the pool. If creation fails, ACK retries automatically. Instance types and zones depend on the node pool's scaling policy.

Scale in — decrease Expected Nodes below the current count. Only pay-as-you-go ECS instances are released during scale-in. The lifecycle of a node's system disk and data disks is bound to the node. When a node is released during scale-in, its disks are released with it and all data on those disks is permanently lost and cannot be recovered. For data that requires persistence, manage it with a PersistentVolume (PV) so that storage is decoupled from the node lifecycle.

Important

When scale-in is triggered by changing Expected Nodes, ACK removes nodes even if node drain fails. If you have a strict drain requirement, remove the target node manually instead. See Remove a node.

Removal order during scale-in depends on the scaling policy:

Scaling policy Removal order
Priority Newly created instances are removed first.
Distribution balancing Zones are filtered by policy; newly created instances are removed first to balance counts across zones.
Cost optimization Instances are removed in descending order of vCPU price.

Billing during scale-out

Billing is based on the actual specifications of the ECS instances created. For example, with two instance types, pay-as-you-go billing, and the Priority scaling policy:

  • Two Node A instances are created in the zone of the first-priority vSwitch.

  • If Node A inventory is insufficient, three Node B instances are created in the zone of the second-priority vSwitch.

Cost for one hour = (Node A unit price × 2 × 1) + (Node B unit price × 3 × 1).

GPU node pools

When adding instances from the ebmgn7 or ebmgn7e ECS Bare Metal Instance families, ACK resets any Multi-Instance GPU (MIG) configuration retained on those instances. This reset is time-consuming and may prevent instances from joining the cluster.

Prerequisites

Make sure you have:

  • An ACK cluster with at least one node pool

  • (Optional, RAM users) The AliyunOOSLifecycleHook4CSRole role is assigned to your Alibaba Cloud account, and AliyunRAMReadOnlyAccess is attached to the RAM user.

Scale a node pool

Important

Do not run kubectl delete node. This removes nodes from Kubernetes without releasing their ECS instances, causing the node pool to lose track of capacity. Use the ACK console to scale node pools.

  1. Log in to the ACK console. In the left navigation pane, click Clusters.

  2. On the Clusters page, click the name of the target cluster. In the left navigation pane, choose Nodes > Node Pools.

  3. Find the node pool to scale and click Scale in the Actions column.

  4. (Optional) If CloudOps Orchestration Service (OOS) has not been authorized, click AliyunOOSLifecycleHook4CSRole to create the role and complete the authorization.

    If using a RAM user, ensure the Alibaba Cloud account has the AliyunOOSLifecycleHook4CSRole role before attaching AliyunRAMReadOnlyAccess to the RAM user.

  5. Set the Expected Nodes value and confirm the change. The Status column cycles through Updating, Scaling Out (or Removing), and then Active.

    • When the status shows Scaling Out, the node pool is being scaled out. When it returns to Active, scale-out is complete.

    • When the status shows Removing, the node pool is being scaled in. When it returns to Active, scale-in is complete.

    Important

    If the cluster's security group blocks access to 100.64.0.0/10, new nodes cannot join the cluster.

To view scale-out task details, click the node pool name, then open the Scaling Activities tab.

View the operational log on a node:

grep cloud-init /var/log/messages
After a node joins the pool, /var/log/messages is cleared and retains only failed join entries. If a node fails to join, the log is synced to the task result on the Cluster Tasks tab.

Operations to avoid

Important

The following operations bypass the expected-count mechanism and can cause reconciliation loops or data loss.

Operation What happens What to do instead
Delete nodes with kubectl delete node ACK tracks expected nodes against ECS instances in the scaling group, not the Kubernetes node list. The ECS instance keeps running, but the node shows as Unknown. Click the node pool name, open the Nodes tab, and remove nodes. Select Release ECS Instance as needed. Manually added and subscription nodes must be released from the ECS console.
Release ECS instances directly from the ECS console or API The node pool detects the discrepancy and creates new ECS instances to restore the expected count. Use the ACK console to remove nodes. See Remove a node. Manually added nodes and subscription nodes require manual release from the ECS console.
Remove ECS instances from the scaling group in the Auto Scaling console (without changing expected count) The node pool detects the change and creates replacement instances. Do not modify scaling groups used by node pools directly in the Auto Scaling console.
Let subscription ECS instances expire The node pool detects the release and creates replacement instances. Remove or renew subscription instances before they expire. See Remove a node and Renew a subscription instance.
Enable health checks on the scaling group in the Auto Scaling console When a health check identifies an unhealthy (such as suspended) instance, Auto Scaling creates a replacement ECS instance. Health checks are disabled by default for scaling groups used by ACK. Do not modify these settings.

Troubleshoot scaling failures

If scaling fails, click the cluster name on the Clusters page, open the Cluster Tasks tab, and click View Cause to see the error details.

Error code Cause Solution
RecommendEmpty.InstanceTypeNoStock Insufficient ECS inventory in the current zone. Modify the node pool to add vSwitches in different zones and configure multiple instance types. See Check the scalability of a node pool.
NodepoolScaleFailed.FailedJoinCluster Nodes failed to join the ACK cluster. Log in to the node and run grep cloud-init /var/log/messages to check the error.
InvalidAccountStatus.NotEnoughBalance / InsufficientBalance.CreditPay / Account.Arrearage Insufficient account balance. Top up your account.
InvalidParameter.NotMatch (boot mode mismatch) The instance type does not support the OS image's boot mode (BIOS). Select a different instance type. Check the OS and image ID on the node pool's Overview tab. Call DescribeImageSupportInstanceTypes to query supported instance types.
QuotaExceed.ElasticQuota ECS quota exceeded for the specified instance type in the current region. Select a different instance type, reduce existing ECS instances, or request a quota increase at the Quota Center.
InvalidResourceType.NotSupported The instance type is not supported or out of stock in the current zone. Call DescribeAvailableResource to find supported instance types in the zone, then update the node pool.
InvalidImage.NotSupported The OS image does not support security-enhanced instances. Select a different instance type. See Create a trusted instance in the ECS console for supported images.
InvalidParameter.NotMatch (vTPM image mismatch) The OS image requires a vTPM-enabled instance type. Select a different instance type. Check the OS and image ID on the node pool's Overview tab.
QuotaExceeded.PrivateIpAddress The current vSwitch has no available private IP addresses. Add more vSwitches to the node pool and retry.
InvalidParameter.KmsNotEnabled The Key Management Service (KMS) key is disabled. Log in to the KMS console and enable the key.
InvalidInstanceType.NotSupported The instance type does not support the OS image architecture. Select a different instance type. Call DescribeImageSupportInstanceTypes to query compatible instance types. See OS images for ACK-supported images.
ApiServer.InternalError / Err.QueryEndpoints The ACK cluster's API server is unreachable. Check API server accessibility. See Troubleshoot cluster access issues in the ACK console.
RecommendEmpty.InstanceTypeNotAuthorized No permission to use the specified instance type. Submit a ticket to request ECS permissions.
RecommendEmpty.DiskTypeNoStock Insufficient cloud disk inventory in the zone. Add more vSwitches or select a different disk type.
InvalidParameter.KMSKeyId.KMSUnauthorized No permission to access KMS. In the ECS console, assign the AliyunECSDiskEncryptDefaultRole role to ECS. See Encryption-related permissions.
InvalidParameter.Conflict The cloud disk type is not compatible with the instance type. Select a different instance type or disk type.
NotSupportSnapshotEncrypted.DiskCategory System disk encryption requires an ESSD. Select an ESSD. For disk types and encryption, see Disk types and encryption.
ScalingActivityInProgress The node pool is already being scaled. Wait for the current scaling activity to complete before retrying. Do not scale node pools directly in the Auto Scaling console.
Instance.StartInstanceFailed ECS instances failed to start. Retry. If the issue persists, submit a ticket to the ECS team.
OperationDenied.NoStock The instance type is out of stock in the specified zone. Select a different instance type. See Check the scalability of a node pool.
NodepoolScaleFailed.WaitForDesiredSizeTimeout The scale-out task timed out. In the ACK console, go to Clusters > cluster name > Nodes > Node Pools, click the node pool name, and open the Scaling Activities tab to view task details.
ApiServer.TooManyRequests The Kubernetes API server throttled the request. Reduce request frequency or retry later.
NodepoolScaleFailed.PartialSuccess Some nodes failed to be created due to insufficient inventory. Change the instance types configured for the node pool. See Check the scalability of a node pool. See Edit a node pool.

Next steps

Previous：Create and manage node poolsNext：Add existing nodes