Configure auto scaling for a cluster - Elastic High Performance Computing

Benefits

Adds compute nodes based on real-time workloads to improve cluster availability.
Removes compute nodes to reduce costs while ensuring cluster availability.
Improves fault tolerance by stopping unhealthy nodes and creating new nodes to replace them.

Limitations

Auto scaling is supported only for clusters where all nodes run a Linux operating system.
Auto scaling is supported only for clusters that use PBS, Slurm, Deadline, or SGE (Open Grid Scheduler) as the scheduler.
Auto scaling based on memory metrics is not supported.

Important
We recommend specifying the required number of vCPUs when you submit a job to enable auto scaling. The memory size for the job must not exceed the memory capacity of the ECS instance.

Usage notes

The auto scaling service requires the scheduler and domain account services to be operating normally. After you enable auto scaling, the management node must remain in the Running state.

Important
If you shut down or restart the management node, ensure that no jobs are running on compute nodes and that auto scaling has released all idle nodes. We recommend first disabling auto scaling, and then re-enabling it after the management node restarts.

The recommended management node specifications and job limits vary by cluster size. For more information, see Cluster overview.

Cluster size	Recommended configuration for management nodes	Job recommendations
Number of compute nodes ≤ 100	vCPUs ≥ 16 Memory ≥ 64 GiB	Number of queued jobs < 5,000 Number of unfinished jobs < 10,000
Number of compute nodes ≤ 500	vCPUs ≥ 32 Memory ≥ 128 GiB	Number of queued jobs < 10,000 Number of unfinished jobs < 20,000
Number of compute nodes > 500	vCPUs ≥ 64 Memory ≥ 256 GiB	Number of queued jobs < 10,000 Number of unfinished jobs < 20,000

Procedure

Go to the Auto Scale page.
1. Log on to the E-HPC console.
2. In the top navigation bar, select a region.
3. In the left-side navigation pane, choose Elasticity > Auto Scale.
On the Auto Scale page, select the cluster for which you want to configure auto scaling from the cluster list.

In the Global Configurations section, configure the parameters.

Parameter	Description
Enable Autoscale	Enables automatic scale-out and scale-in for all queues in the cluster. Note If a queue-specific setting conflicts with a global setting, the queue-specific setting takes precedence.
Compute Nodes	The allowed range of compute nodes in the cluster after a scaling activity. The upper limit is the sum of the maximum node counts specified for all queues in the cluster. The lower limit is the sum of the minimum node counts specified for all queues.
Scale-in Time (minutes)	The system releases a compute node if its continuous idle time exceeds this value. Continuous idle time = Scale-in interval × Number of consecutive idle checks. The default scale-in interval is 2 minutes. The number of consecutive idle checks is the number of times a node is found to be idle during resource shrink checks.
ImageType	The image type for new nodes. Only images that match those of existing compute nodes in the cluster are supported.
Exception Node List	The nodes that are excluded from auto scaling. If you want to permanently retain a node, you can add it to the exception list. The system does not release an exception node, even when it is idle.
Hyper-threading	By default, Hyper-Threading (HT) is enabled for all ECS instances. For some instance types, you can disable HT to achieve better performance. For more information, see Change CPU options and Disable CPU Hyper-Threading to improve cluster performance.

In the Queue Configuration section, select a queue, and then click Edit to configure the parameters.

Parameter	Description
Auto Grow and Auto Shrink	Specifies whether to enable automatic scale-out and scale-in. This feature is disabled by default. Note If a queue-specific setting conflicts with a global setting, the queue-specific setting takes precedence.
Queue Compute Nodes	The allowed range of compute nodes in the queue. Maximum: This value affects scale-out. Valid values: 0 to 5,000. Minimum: This value affects scale-in. Valid values: 0 to 1,000. Important If you set the minimum number of nodes to a value other than 0, the system retains the specified number of nodes in the queue during a scale-in event, even if they are idle. Set this parameter with caution to prevent resource waste and unnecessary costs.
Hostname Prefix	The prefix of the node hostname, used to distinguish nodes in different queues.
Maximum Nodes in Each Round of Scale-out	The maximum number of nodes added in each auto scaling cycle. The default value is 0, which indicates no limit. If you have cost constraints, set this parameter to limit the number of added nodes. Assume you set this parameter to A and the number of nodes required by queued jobs is B. The following rules apply: If B ≤ A, B nodes are added. If B > A, only A nodes are added. Note The number of nodes to be added is also limited by the maximum number of nodes allowed for the queue and the cluster.
Minimum Nodes in Each Round of Scale-out	The minimum number of nodes added in each auto scaling cycle. The default value is 1. In some scenarios, you may need to add a minimum number of nodes to ensure your service can run. In this case, you can set this parameter. If resources are insufficient to add the specified minimum number of nodes, the system adds no nodes. This prevents resource waste. Assume you set this parameter to A and the number of nodes required by queued jobs is B. The following rules apply: If B ≤ A, B nodes are added. If resources are insufficient to add B nodes, no scale-out occurs. If B > A, B nodes are added. If resources are insufficient to add B nodes, the system tries to add at least A nodes. If A nodes cannot be added, no scale-out occurs.
Automatic Configuration of the Minimum Node Number for Each Scale-out	If you enable this switch, the minimum number of nodes for each scale-out equals the number of nodes required by the job, up to 99.
Hostname Suffix	The suffix of the node hostname, used to distinguish nodes in different queues.
ImageType	The image type for nodes added to a single queue. You can configure different image types for different queues.
Image ID	The ID of the image for the new nodes. You can configure different image IDs for different queues. Note This parameter is specific to the current queue. If you do not specify an image type or image ID for the queue, the image of the added nodes depends on the image type specified in the global configurations. If you do not specify an image type in the global configurations, the added nodes use the cluster's default image type.
Whether instance types are unordered	If you enable this switch, auto scaling prioritizes instance types with the highest availability.
Configuration List	Configurations for the new instances. The configuration list includes the following information: Zone: The availability zones within the cluster's region. vSwitch ID: The ID of a vSwitch that is in the availability zone and associated with the cluster's VPC. Instance Type: The instance types for new compute nodes in a queue. Note If multiple instance types are configured for a queue, auto scaling selects an instance type for scale-out based on the configured order, stock availability, and the job's requirements for tasks and GPUs. For example, a queued job requires a single node to have 16 cores. The queue is configured with 8-core, 16-core, and 32-core instance types. Auto scaling selects the 16-core instance type for the scale-out. However, if the 16-core instance type is out of stock, auto scaling selects the 32-core instance type. Bid Strategy: The bidding strategy for new instances. You can create preemptible instances. Maximum Hourly Price: This parameter is available only when you set Bid Strategy to Set Maximum Price for a preemptible instance.
System Disk	System disk configuration for new instances.
Data Disk	The data disks to attach to the new instances. You can configure the data disk type, size, and performance level based on your business requirements, and specify whether to encrypt the data disks and release them with the instances.

In the upper-right corner of the page, read and select Alibaba Cloud International Website Product Terms of Service, and then click OK.
(Optional) View the cluster auto scaling diagram.

The cluster auto scaling diagram shows how the number of nodes changes over time and the duration of instance scale-out and scale-in events at key points in the auto scaling process, based on your configured scaling policies.

Note
You can also set the number of concurrent nodes in the diagram area to simulate compute node changes during auto scaling.