HPC clusters experience variable workloads — jobs queue up during peak periods and nodes sit idle during off-peak hours. E-HPC auto scaling dynamically adds compute nodes when jobs are pending and removes idle nodes when workloads drop, improving cluster availability and reducing costs without manual intervention. E-HPC also stops faulty nodes and creates replacement nodes to improve fault tolerance.
Supported schedulers and operating systems
Auto scaling is supported only for clusters that meet all of the following conditions:
All nodes run Linux operating systems.
The cluster uses PBS, Slurm, Deadline, or SGE as its scheduler.
Auto scaling based on memory usage is not supported. When you submit a job, specify the number of required vCPUs. The memory size you specify for a job cannot exceed the memory capacity of the Elastic Compute Service (ECS) instances in the cluster.
Before you configure auto scaling
Verify cluster services
Before you enable auto scaling, confirm that:
The scheduler service is running normally.
The domain account service is running normally.
After you enable auto scaling, the management node must remain in the Running state. If you need to shut down or restart the management node:
Wait until all idle nodes are released and no jobs are running on compute nodes.
Disable auto scaling.
Shut down or restart the management node.
Re-enable auto scaling after the management node is back online.
Size the management node
The management node must meet the minimum specifications for your expected cluster scale. Under-provisioned management nodes can cause scheduling delays and failed scale-out operations.
| Compute nodes | Management node specs | Job queue capacity |
|---|---|---|
| Up to 100 | 16+ vCPUs, 64+ GiB memory | <5,000 queued; <10,000 uncompleted |
| Up to 500 | 32+ vCPUs, 128+ GiB memory | <10,000 queued; <20,000 uncompleted |
| More than 500 | 64+ vCPUs, 256+ GiB memory | <10,000 queued; <20,000 uncompleted |
For details, see Overview.
Configure auto scaling
Log on to the E-HPC console. In the top navigation bar, select a region.
In the left-side navigation pane, choose Elasticity > Auto Scale.
From the Cluster drop-down list, select the cluster to configure.
In the Global Configurations section, set the parameters. See Global configuration parameters for details.
In the Queue Configuration section, select a queue and click Edit. See Queue configuration parameters for details.
In the upper-right corner, read and select Alibaba Cloud International Website Product Terms of Service, then click OK.
(Optional) Review the auto scaling diagram to verify expected behavior. See View the auto scaling diagram.
Global configuration parameters
Global configurations apply to all queues in the cluster. If a queue's own settings differ from the global settings, the queue settings take precedence.
| Parameter | Description |
|---|---|
| Enable Autoscale | Enables Auto Grow and Auto Shrink for all queues in the cluster. Queue-level settings override this when they differ. |
| Compute Nodes | The allowed range of compute nodes across the cluster after scaling. The upper limit equals the sum of all queues' maximum node counts; the lower limit equals the sum of all queues' minimum node counts. |
| Scale-in Time (Minute) | How long a compute node must remain continuously idle before it is released. Idle time is measured as the scale-in interval (default: 2 minutes) multiplied by the number of consecutive idle checks. For example, if a node is idle for 5 consecutive checks, its idle time is 10 minutes. |
| Image Type | The image type for newly added compute nodes. Only images compatible with the cluster's existing compute node image are supported. |
| Exceptional Nodes | Compute nodes excluded from auto scaling. Exceptional nodes are never released, even when idle. |
| Hyper-threading | Hyper-Threading (HT) is enabled by default for all ECS instances. For specific instance types, you can disable HT for better performance. For supported instance types, see Instance type limits. For instructions, see Disable HT for compute nodes. |
Queue configuration parameters
Queue-level settings override the corresponding global configurations when they differ.
Scaling switches
| Parameter | Description |
|---|---|
| Auto Grow | Enables automatic scale-out for this queue. Off by default. |
| Auto Shrink | Enables automatic scale-in for this queue. Off by default. |
Node count limits
| Parameter | Description |
|---|---|
| Queue Compute Nodes | The allowed range of compute nodes in this queue. Maximum: 0–5,000 nodes. Minimum: 0–1,000 nodes. |
Setting Minimum Nodes to a non-zero value keeps that many nodes running during scale-in — idle nodes in the queue are not released. Set this only when your workload requires always-on nodes, as idle reserved nodes consume resources and incur costs.
Scale-out behavior
These parameters work together to control how many nodes are added in each scale-out round.
| Parameter | Description |
|---|---|
| Maximum Nodes in Each Round of Scale-out | The maximum number of nodes added per scale-out round. Default 0 (no limit). If set to A and B nodes are needed: B ≤ A → B nodes added; B > A → A nodes added. |
| Minimum Scale-out Nodes in Each Round | The minimum number of nodes added per scale-out round. Default 1. If fewer ECS instances are available than the configured minimum, the cluster does not scale out. If set to A and B nodes are needed: B ≤ A → B nodes added if ECS stock ≥ B, else no scale-out; B > A → B added if ECS stock ≥ B; A added if ECS stock ≥ A; no scale-out if ECS stock < A. |
| Automatic Configuration of the Minimum Node Number for Each Scale-out | When enabled, the minimum scale-out count equals the number of nodes required by the job. Cannot exceed 99. |
The total nodes added is also constrained by the queue's maximum node count and the cluster's overall maximum node count.
Node identity
| Parameter | Description |
|---|---|
| Prefix of Hostnames | The hostname prefix for compute nodes in this queue. Use different prefixes across queues to distinguish their nodes. |
| Hostname Suffix | The hostname suffix for compute nodes in this queue. |
Image settings
| Parameter | Description |
|---|---|
| Image Type | The image type for nodes added to this queue. Overrides the global image type for this queue. |
| Image ID | The image ID for nodes added to this queue. If not set, inherits the global configuration. If the global configuration is also unset, uses the cluster's default image. |
Instance selection
| Parameter | Description |
|---|---|
| Whether instance types are unordered | When enabled, the system selects instance types in descending order of available stock during scale-out, maximizing the chance of successful resource delivery. |
| Configuration List | The compute node configurations for this queue. Each entry includes: Zone (a zone in the region where the cluster resides), vSwitch ID (the vSwitch bound to the cluster's VPC in that zone), Instance Type, Bid Strategy, and Maximum Price per Hour (required only when Bid Strategy is set to Preemptible instance with maximum bid price). |
When multiple instance types are configured, the system selects among them during scale-out based on available stock, job count, and GPU count — in that order. For example, if each job requires at least 16 cores, the system adds 16-core instances first. If none are available, it falls back to 32-core instances.
Storage
| Parameter | Description |
|---|---|
| System Disk | The system disk configuration for newly added compute nodes. |
| Data disk | The data disk for newly added compute nodes. Configure the disk type, size, performance level, whether to release the disk with the node, and whether to encrypt the disk. |
View the auto scaling diagram
The auto scaling diagram shows how the number of compute nodes changes over time based on your configured policy, including the time consumed by scale-in and scale-out at key points.
To simulate scaling behavior before it occurs, specify the number of concurrent nodes in the diagram. The simulation helps you verify that your policy settings produce the expected node count changes.
Related topics
Overview — Understand cluster management fundamentals and supported management node configurations.
Disable HT for compute nodes — Optimize compute performance for workloads sensitive to Hyper-Threading.