Elastic High Performance Computing:Scheduling container jobs using a standard slurm cluster

Step 1: Create a cluster

1. Create a standard SLURM cluster.

   The following table describes the example cluster configuration used in this topic. Configure other parameters as needed.

   	Parameter	Configuration
   Cluster Configuration	Series	Standard Edition
   	Deployment Mode	Public cloud cluster
   	Cluster Type	SLURM
   	Management Node	Instance Type: Use the ecs.r7.xlarge instance type, which has 4 vCPUs and 32 GiB of memory. Image: centos_7_6_x64_20G_alibase_20211130.vhd
   Compute Node and Queue	Queue Compute Nodes	Initial nodes: 1.
   	Inter-node interconnection	VPC Network
   	Instance type Group	Instance Type: Use the ecs.gn7i-c56g1.14xlarge instance type, which has 56 vCPUs and 346 GiB of memory. Important You must use an instance type that supports GPUs. For more information, see Instance family. Image: centos_7_6_x64_20G_alibase_20211130.vhd
   Shared File Storage	/home Cluster Mount Directory	By default, the /home and /opt directories of the control plane node are mounted to the file system as shared storage directories.
   	/opt Cluster Mount Directory
   Software and Service Component	Software To Be Installed	Select docker.
   	Installable service components	Logon Node: Instance Type: Use the ecs.r7.xlarge instance type, which has 4 vCPUs and 32 GiB of memory. Image: centos_7_6_x64_20G_alibase_20211130.vhd.

2. Create a cluster user.

   In this topic, the usertest user is used as an example.

Step 2: Set up the basic software environment

1. Attach an Elastic IP Address to the compute node.

2. Connect to the compute node.

3. Download and install CUDA.

   1. Download the CUDA installation package.

      cd /opt
      wget https://developer.download.nvidia.com/compute/cuda/12.4.1/local_installers/cuda_12.4.1_550.54.15_linux.run

   2. Install CUDA.

      yum install -y git
      sh /opt/cuda_12.4.1_550.54.15_linux.run

      When the following figure appears, CUDA is installed.

      

   3. Configure the environment variables.

      echo 'export PATH=/usr/local/cuda-12.4/bin:$PATH' >> ~/.bashrc
      echo 'export LD_LIBRARY_PATH=/usr/local/cuda-12.4/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
      
      source ~/.bashrc

   4. View the installation status and version information of the NVIDIA CUDA toolkit and GPU driver.

      # Version information of the NVIDIA CUDA compiler driver
      nvcc --version
      
      # Detailed status information of the GPU
      nvidia-smi

      When the following figure appears, CUDA and the GPU driver work fine.

      

4. Install and configure NVIDIA Container Toolkit.

   distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
   curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
   sudo yum install -y nvidia-container-toolkit
   sudo systemctl restart docker

5. Download and install Singularity.

   Singularity is a containerization tool that allows you to run containers without changing the user environment. It is commonly used in HPC environments.

   cd /opt
   wget https://public-ehs.oss-cn-hangzhou.aliyuncs.com/softwares/packages/CentOS_7.2_64/singularity-3.8.3-1.el7.x86_64.rpm
   yum install -y /opt/singularity-3.8.3-1.el7.x86_64.rpm

6. Create job dependency data.

   1. Pull the PyTorch container image. There is no need to change anything for the command below.

      docker pull ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/pytorch:2.4.0-cuda12.1.1-py310-alinux3.2104

   2. Use the usertest user to create a main.py file.

      vim /home/usertest/main.py

   3. The script content of the main.py file is as follows:

      # -*- coding: utf-8 -*-
      
      import torch
      import torchvision
      import torchvision.transforms as transforms
      from torch import nn
      from torch.utils.data import DataLoader
      from torch.optim import SGD
      
      class SimpleNet(nn.Module):
          def __init__(self):
              super(SimpleNet, self).__init__()
              self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1)
              self.conv2 = nn.Conv2d(32, 64, kernel_size=3, padding=1)
              self.fc1 = nn.Linear(64 * 8 * 8, 128)
              self.fc2 = nn.Linear(128, 10)
              self.pool = nn.MaxPool2d(2, 2)
              self.relu = nn.ReLU()
              self.dropout = nn.Dropout(0.5)
      
          def forward(self, x):
              x = self.pool(self.relu(self.conv1(x)))
              x = self.pool(self.relu(self.conv2(x)))
              x = x.view(-1, 64 * 8 * 8)
              x = self.relu(self.fc1(x))
              x = self.dropout(x)
              x = self.fc2(x)
              return x
      
      device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
      print(f"Using device: {device}")
      
      transform = transforms.Compose([
          transforms.ToTensor(),
          transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
      ])
      
      train_dataset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
      test_dataset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)
      
      train_loader = DataLoader(dataset=train_dataset, batch_size=64, shuffle=True)
      test_loader = DataLoader(dataset=test_dataset, batch_size=64, shuffle=False)
      
      model = SimpleNet().to(device)
      criterion = nn.CrossEntropyLoss()
      optimizer = SGD(model.parameters(), lr=0.001, momentum=0.9)
      
      num_epochs = 10
      for epoch in range(num_epochs):
          model.train()
          running_loss = 0.0
          for i, data in enumerate(train_loader, 0):
              inputs, labels = data
              inputs, labels = inputs.to(device), labels.to(device)
      
              optimizer.zero_grad()
              outputs = model(inputs)
              loss = criterion(outputs, labels)
              loss.backward()
              optimizer.step()
      
              running_loss += loss.item()
              if i % 100 == 99: 
                  print(f"[{epoch + 1}, {i + 1}] loss: {running_loss / 100:.3f}")
                  running_loss = 0.0
      
          model.eval()
          correct = 0
          total = 0
          with torch.no_grad():
              for data in test_loader:
                  images, labels = data
                  images, labels = images.to(device), labels.to(device)
                  outputs = model(images)
                  _, predicted = torch.max(outputs.data, 1)
                  total += labels.size(0)
                  correct += (predicted == labels).sum().item()
      
          print(f"Accuracy on test set: {100 * correct / total:.2f}%")
      
      print("Training finished.")

Step 3: Schedule jobs

#!/bin/bash
#SBATCH --job-name=your_job_name
#SBATCH --output=your_output
#SBATCH --nodes=1
#SBATCH --gres=gpu:1
#SBATCH --ntasks=2

# args
iamges="your_image"
run_cmd="your_command"
work_dir="your_workdir"
share_dir="/ehpcdata/:/mnt"
devices="device=$CUDA_VISIBLE_DEVICES"

# cleanup docker handle
function cleanup {
    echo "Caught signal, stopping Docker container: " $SLURM_JOB_NAME
    docker ps -q --filter label=$SLURM_JOB_NAME | xargs -r docker stop
    docker ps -qa --filter label=$SLURM_JOB_NAME | xargs -r docker rm
}
trap cleanup SIGINT SIGTERM

# start docker
docker pull $iamges
srun docker run \
  --label $SLURM_JOB_NAME \
  --rm --net=host \
  --gpus '"'$devices'"' \
  -v $share_dir \
  $iamges \
  /bin/bash -c "$run_cmd" &

# wait to complete
wait
cleanup

#!/bin/bash
#SBATCH --job-name=my_job_name
#SBATCH --output=output.log
#SBATCH --nodes=1
#SBATCH --gres=gpu:1
#SBATCH --ntasks=2

# args
image=your_image.sif
run_cmd="your cmd"
share_dir="/ehpcdata/:/mnt"

singularity exec --nv --bind $share_dir $image $run_cmd