Auto scaling lets E-HPC dynamically add or remove compute nodes based on real-time job demand—no manual intervention needed. When jobs queue up, the cluster scales out to meet demand. When nodes sit idle, it scales in to cut costs. If a node fails, the system stops it and launches a replacement.
Benefits
Adds compute nodes based on real-time workloads to improve cluster availability.
Reduces the number of compute nodes to save costs without compromising cluster availability.
Stops faulty nodes and creates replacement nodes to improve fault tolerance.
Limitations
Auto scaling requires all nodes in the cluster to run Linux.
Custom clusters do not support auto scaling. All other cluster types do.
Memory-based auto scaling is only supported for clusters using the Slurm scheduler.
Configure auto scaling
Before you begin
Make sure that:
The scheduler service and domain account service are working correctly.
The management node is in the running state after you enable auto scaling.
Before shutting down or restarting the management node, wait until all idle nodes are released and no jobs are running on compute nodes. Disable auto scaling first, then shut down or restart the management node, and re-enable auto scaling after the restart.
Specify the required vCPU count when submitting jobs to trigger auto scaling correctly. The memory size specified in a job cannot exceed the memory specifications of the target ECS instance type.
Step 1: Open cluster auto scaling settings
Log on to the E-HPC console.
In the left part of the top navigation bar, select a region.
In the left-side navigation pane, click Cluster.
On the Cluster List page, find the cluster you want to configure and click Auto Scale.
Step 2: Configure global scaling settings
In the Cluster Auto Scaling dialog box, configure the Cluster Global Configuration section.
These settings apply to all queues in the cluster. Queue-level settings override global settings when they differ.
| Parameter | Description |
|---|---|
| Auto Grow / Auto Shrink | Enable automatic scale-out and scale-in for all queues in the cluster. |
| Scale-out Waiting Time | How long the system waits after a job is submitted before starting to scale out. Default: 2 minutes. |
| Scale-in Waiting Time | How long a node must stay idle before the system releases it. Default: 4 minutes. |
| Maximum number of cluster nodes | The upper limit on the total number of nodes the cluster can create. |
| Maximum number of cores in the cluster | The upper limit on the total number of cores the cluster can create. |
Step 3: Configure queue-level scaling
Queue-level settings let you fine-tune scaling behavior per queue and override global settings.
To configure a queue:
Click the target cluster.
In the left navigation pane, click Nodes and Queues > Queue.
Find the queue you want to configure, then click Edit in the Actions column.
On the Edit Queue page, configure the parameters in each section.
Basic settings
| Parameter | Description |
|---|---|
| Automatic queue scaling | Off by default. Turn this on, then select Auto Grow, Auto Shrink, or both based on your requirements. Queue settings override global settings. |
| Queue Compute Nodes | Set the minimum and maximum node count for this queue:
|
Queue node configuration
If you enable Automatic queue scaling or set the initial node count above 0, configure the following parameters so the system knows how to create compute nodes.
| Parameter | Description |
|---|---|
| Inter-node interconnection | How compute nodes communicate with each other:
|
| Use Preset Node Pool | Select a reserved node pool to draw from when scaling out. The system picks IP addresses and hostnames from unassigned reserved nodes in the pool, giving you faster scale-out with pre-allocated resources. See Use reserved node pools in clusters. |
| Virtual Switch | The vSwitch for compute nodes. The system assigns IP addresses from the vSwitch CIDR block. |
| Instance type Group | Click Add Instance to select instance types. With Automatic queue scaling disabled, only one instance type is allowed. With it enabled, you can add multiple instance types as fallback options. |
Adding multiple vSwitches and instance types acts as a fallback for inventory shortages. When creating a node, the system tries each instance type in the order you specified, starting with the zone of the first vSwitch. The actual instance specification may vary depending on available inventory.
Auto scale settings
| Parameter | Description |
|---|---|
| Scaling Policy | The strategy for selecting where to create nodes. Currently, only Supply Priority Strategy is supported: nodes are created in the zones specified by your vSwitches, in the order they are configured. |
| Maximum number of single expansion nodes | The maximum number of nodes added or removed in a single scale-out or scale-in cycle. Default: 0 (no limit). Set a value to control burst costs. |
| Prefix of Hostnames | A hostname prefix for compute nodes in this queue. Use this to distinguish nodes across different queues. |
| Hostname Suffix | A hostname suffix for compute nodes in this queue. |
| Instance RAM role | A Resource Access Management (RAM) role that grants compute nodes access to Alibaba Cloud services. The default role AliyunECSInstanceForEHPCRole covers standard E-HPC requirements. |
Step 4: Save the configuration
Review your settings and click Save.
Scaling policy
When multiple instance types are configured in a queue, the cluster scales out based on available instance types and job requirements. For example, if a queue requires at least 16 cores per node and has 8-core, 16-core, and 32-core instance types configured:
The system first tries to create 16-core Elastic Compute Service (ECS) instances.
If 16-core instances are unavailable due to inventory, it falls back to 32-core instances.
Monitor your cluster
After configuring auto scaling, monitor your cluster's health and resource usage to verify that the scaling thresholds are working as expected. See View the monitoring information.
FAQ
The instance was released, but I can't delete the node in the console
When you use spot instances for auto scaling, an instance may be reclaimed while it still has unfinished computing tasks. In this situation, the scheduler cannot cleanly delete the instance, and the node appears stuck in the BusyNodes status in the console.
Wait for the auto-scaling cleanup cycle to run. Nodes in this state are automatically removed after a set period. Once the scheduler updates its status, the node exits BusyNodes and can be deleted normally.