Elastic High Performance Computing:Best practices for managed PowerSaving Slurm clusters

Use cases

HPC workloads often exhibit clear peaks and valleys. You need large amounts of computing power during batch job submissions but require almost no resources during idle periods. Static clusters continue running nodes even when idle, incurring ongoing costs. PowerSaving mode ties compute resources to the job lifecycle. The scheduler pre-registers a set of virtual nodes. It automatically creates ECS instances when you submit jobs and automatically releases them after an idle timeout. You pay for compute resources only while jobs run.

How PowerSaving works

The Slurm scheduler pre-registers a group of virtual nodes (in the idle~ state) for each queue. These nodes appear in the scheduler but do not consume ECS resources. After you submit a job, the scheduler creates an ECS instance (alloc# state). When the instance is ready, it runs the job (alloc state). After the job finishes, the node starts an idle timer. If the idle time exceeds the threshold—defaulting to 1 minute—the system releases the instance and returns the node to the idle~ state.

Node state identifiers and transitions:

For more information, see Slurm Power Saving official documentation.

Create a managed Slurm cluster

Cluster creation takes about 10 to 15 minutes. Fill in other parameters as needed.

1. Create a Standard Edition cluster with the following settings.

   Configuration item	Example value
   Series	Managed
   Deployment mode	Public cloud cluster
   Cluster type	SLURM
   Maximum number of nodes in the cluster	1000 (set based on your peak workload)
   Number of nodes in the queue	Skip this step when creating the cluster. Configure the PowerSaving queue separately after the cluster is ready.
   Shared file storage	By default, mount /home and /opt as shared storage directories.
   Logon node	Instance type: ecs.c7.large (2 vCPU / 4 GiB). Image: CentOS 7.6 64-bit. Enable EIP as needed.

2. Go to the Cluster Details page.

   1. Log on to the E-HPC console.

   2. In the left part of the top navigation bar, select a region.

   3. In the left-side navigation pane, click Cluster.

   4. On the Cluster List page, find the cluster that you want to manage and click the cluster ID.

3. In the left-side navigation pane, click User Management.

4. You must submit jobs as a non-root user. In the User management page, click Add user.

5. In the dialog box that appears, configure user information. Use usertest as an example.

   • Username: usertest.

   • User permission: Select Ordinary permission group. This group is suitable for users who submit and debug jobs.

   • Password and Confirm password: Set the logon password.

Configure a PowerSaving queue

After the cluster is ready, configure a PowerSaving queue to define elastic scaling behavior.

1. In the Clusters list page, click the name of your target cluster.

2. In the navigation pane on the left, choose Nodes and queues > Queues.

3. Click Create queue and fill in the parameters using the following settings.

Basic settings

Node configuration

Auto scaling

Submit a job and verify auto scaling

After you configure the queue, submit a CPU stress test job to verify auto scaling.

cd /tmp
wget https://fossies.org/linux/privat/stress-1.0.7.tar.gz
tar -xzf stress-1.0.7.tar.gz
cd stress-1.0.7/
./configure --prefix=/opt/stress
make && make install

/opt/stress/bin/stress --version

su usertest
cd ~

cat > cpu_stress.slurm << EOF
#!/bin/bash
#SBATCH --job-name=cpu_stress_test
#SBATCH --partition=testq
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=2
#SBATCH --time=00:10:00
#SBATCH --output=cpu_stress_%j.out
#SBATCH --error=cpu_stress_%j.err

echo "=== CPU Stress Test ==="
echo "Job ID: \$SLURM_JOB_ID"
echo "Node: \$(hostname)"
echo "Requested CPUs: \$SLURM_CPUS_PER_TASK"
echo "Start time: \$(date)"

NUM_CPUS=\${SLURM_CPUS_PER_TASK:-\$(nproc)}

if command -v /opt/stress/bin/stress >/dev/null 2>&1; then
    /opt/stress/bin/stress --cpu \$NUM_CPUS --timeout \${SLURM_TIME_LIMIT:-600}
else
    echo "ERROR: stress is not installed"
    exit 1
fi

echo "End time: \$(date)"
echo "=== CPU Stress Test Completed ==="
EOF

sbatch -p testq cpu_stress.slurm

Limits

• Region limits: Supported only in Hangzhou, Beijing, Heyuan, and Shanghai.

• Maximum nodes per cluster: 1000.

• Queues support only single-zone, single-instance-type, and homogeneous instance types.

• Queues do not support exception nodes, hostname prefix changes, hyper-threading, or eRDMA.

• Maximum nodes per queue: 500.

• Manual scale-out and scale-in are not supported.

• srun and MPI jobs are not supported. Only sbatch batch jobs are supported.