Container Service for Kubernetes:Colocated scheduling for Slurm HPC and Kubernetes

The chart creates the following resources.

The following table describes the key parameters.

SlurmCopilot and Slurmrestd use JSON Web Token (JWT) authentication. Generate a JSON Web Key (JWK) and import it into the cluster before installing the chart.

The JWT authentication plug-in uses JWKs (RFC7517) to sign and verify tokens. The private key signs tokens; the public key is configured in the plug-in to verify them.

Generate a JWK using an online tool such as mkjwk.org or generate one yourself. The following is a valid JWK example:

{
  "kty": "RSA",
  "e": "AQAB",
  "kid": "O9fpdhrViq2zaaaBEWZITz",
  "use": "sig",
  "alg": "RS256",
  "n": "qSVxcknOm0uCq5vGsOmaorPDzHUubBmZZ4UXj-9do7w9X1uKFXAnqfto4TepSNuYU2bA_-tzSLAGBsR-BqvT6w9SjxakeiyQpVmexxnDw5WZwpWenUAcYrfSPEoNU-0hAQwFYgqZwJQMN8ptxkd0170PFauwACOx4Hfr-9FPGy8NCoIO4MfLXzJ3mJ7xqgIZp3NIOGXz-GIAbCf13ii7kSStpYqN3L_zzpvXUAos1FJ9IPXRV84tIZpFVh2lmRh0h8ImK-vI42dwlD_hOIzayL1Xno2R0T-d5AwTSdnep7g-Fwu8-sj4cCRWq3bd61Zs2QOJ8iustH0vSRMYdP5oYQ"
}

This example is in JSON format. Convert to YAML if configuring the plug-in via YAML. Only configure the public key in the JWT authentication plug-in. Store the private key securely.

The plug-in supports the following signing algorithms.

Import the JWK into the cluster.

kubectl create configmap jwt --from-literal=jwt_hs256.key='<Your-JWK>'

Edit slurmConfigs.filesInConfigMap in values.yaml to declare the Generic Resource Scheduling (GRES) configuration, database address, and authentication settings.

slurmConfigs:
  ...
  filesInConfigMap:
    gres.conf: |
      # Used by SlurmCopilot to sync Kubernetes allocated resources to Slurm.
      Name=k8scpu Flags=CountOnly
      Name=k8smemory Flags=CountOnly
    slurmdbd.conf: |
      # Log path. Must match the path used for verification.
      LogFile=/var/log/slurmdbd.log
      # JWT authentication is required when using Slurmrestd.
      AuthAltTypes=auth/jwt
      # Slurmdbd uses this key to authenticate tokens. Mount the key into the Pod as shown below.
      AuthAltParameters=jwt_key=/var/jwt/jwt_hs256.key
      AuthType=auth/munge
      SlurmUser=slurm
      # Set MySQL database account information.
      StoragePass=
      StorageHost=
      StorageType=accounting_storage/mysql
      StorageUser=root
      StoragePort=3306
    slurm.conf: |
      # Sets k8scpu and k8smemory extended resource properties when a node joins the cluster,
      # preventing the node from being set to DOWN state.
      NodeFeaturesPlugins=node_features/k8s_resources
      # Automatically adds k8scpu and k8smemory extended resources when submitting Slurm jobs.
      JobSubmitPlugins=k8s_resource_completion
      AccountingStorageHost=slurm-test-slurmdbd
      # JWT authentication is required when using Slurmrestd.
      AuthAltTypes=auth/jwt
      # Slurmctld uses this key to generate tokens. Mount the key into the Pod as shown below.
      AuthAltParameters=jwt_key=/var/jwt/jwt_hs256.key
      # Used by SlurmCopilot to sync Kubernetes allocated resources to Slurm.
      GresTypes=k8scpu,k8smemory
      # Enter ${slurmClusterName}-slurmdbd.
      AccountingStorageHost=
      AccountingStoragePort=6819
      AccountingStorageType=accounting_storage/slurmdbd
      # MySQL database settings for the JobComp plug-in.
      JobCompHost=
      JobCompLoc=/var/log/slurm/slurm_jobcomp.log
      JobCompPass=
      JobCompPort=3306
      JobCompType=jobcomp/mysql
      JobCompUser=root
      # High availability configuration.
      SlurmctldHost=

Configure the Slurmctld, Slurmdbd, and Slurmrestd Pods to mount the JWT key.

...
headNodeConfig:
  image: "registry-cn-hangzhou.ack.aliyuncs.com/acs/slurm:23.06-1.6-aliyun-49259f59"
  # Mount the JWT key into Slurm. The mount path must match the path in the configuration above.
  volumes:
  - configMap:
      defaultMode: 444
      name: jwt
    name: config-jwt
  volumeMounts:
  - mountPath: /var/jwt
    name: config-jwt
slurmdbdConfigs:
  nodeSelector: {}
  tolerations: []
  affinity: {}
  resources: {}
  image: "registry-cn-hangzhou.ack.aliyuncs.com/acs/slurm:23.06-1.6-aliyun-49259f59"
  imagePullSecrets: []
  # if .slurmConfigs.createConfigsByConfigMap is true, slurmConfPath, volume, and volumeMounts
  # are automatically set. This also applies to mungeConfigs.createConfigsBySecret.
  volumes:
  - configMap:
      defaultMode: 444
      name: jwt
    name: config-jwt
  volumeMounts:
  - mountPath: /var/jwt
    name: config-jwt

slurmrestdConfigs:
  nodeSelector: {}
  tolerations: []
  affinity: {}
  resources: {}
  image: "registry-cn-hangzhou.ack.aliyuncs.com/acs/slurm:23.06-1.6-aliyun-49259f59"
  imagePullSecrets: []
  volumes:
  - configMap:
      defaultMode: 444
      name: jwt
    name: config-jwt
  volumeMounts:
  - mountPath: /var/jwt
    name: config-jwt

When submitting Slurm jobs without the auto-fill plug-in, calculate the required GRES manually.

`srun` and `sbatch` parameters

Using the auto-fill plug-in

The job_submit/k8s_resource_completion plug-in automatically populates --gres from CPU and memory requests. When using this plug-in:

• Specify task count using -n or --ntasks (required)

• Request GPU resources using --gpus-per-task, not --gpus or --gpus-per-socket

A code sample is available for compiling the plug-in.

Example job script

#!/bin/bash
#SBATCH --job-name=test_job                   # Job name
#SBATCH --nodes=2                             # Number of nodes required
#SBATCH --ntasks-per-node=4                   # Number of tasks per node
#SBATCH --cpus-per-task=2                     # Number of vCPUs per task
#SBATCH --time=01:00:00                       # Maximum run time
#SBATCH --output=job_output_%j.txt            # Standard output file
#SBATCH --error=job_error_%j.txt              # Standard error file

srun my_program

Alternatively, pass parameters on the command line:

sbatch --nodes=2 --ntasks-per-node=4 --cpus-per-task=2 --time=01:00:00 --job-name=test_job my_job_script.sh