Container Service for Kubernetes:Implement colocated scheduling for Slurm and Kubernetes in ACK clusters

Why do we need to implement colocated scheduling in ACK clusters?

• Default scheduling solution: ACK statically allocates resources. ACK and Slurm schedule workloads separately. In a Slurm cluster, each Slurm pod pre-occupies cluster resources. Kubernetes cannot use idle resources that are pre-occupied by Slurm pods. This results in cluster resource fragments. If you want to modify the resource configuration of a Slurm pod, you must delete and then recreate the pod. Therefore, in scenarios where there is a great gap between resources occupied by Slurm clusters and resources occupied by Kubernetes clusters, it is hard to migrate workloads to other nodes.

• Colocated scheduling solution: ACK provides the ack-slurm-operator component to implement colocated scheduling for Slurm and Kubernetes in ACK clusters. This solution runs a copilot in the Kubernetes cluster and an extended resource plug-in in the Slurm cluster. This allows Kubernetes and Slurm to share cluster resources and avoids repetitively allocating resources.

The following figure shows the preceding resource sharing solutions.

Static resource allocation and separate workload scheduling	Colocated scheduling for Slurm and Kubernetes

The following figure shows how the colocated scheduling solution works.

1.1 Install ack-slurm-operator

An ACK cluster that runs Kubernetes 1.26 or later is created. For more information, see Add GPU-accelerated nodes to a cluster and Update clusters.

1.2 Install and configure ack-slurm-cluster

To quickly deploy and manage a Slurm cluster and flexibly modify the configurations of the cluster, you can use Helm to install the SlurmClusterart package provided by Alibaba Cloud. Download the Helm chart for a Slurm cluster from charts-incubator and set the relevant parameters. Then, Helm will create role-based access control (RBAC) resources, ConfigMaps, Secrets, and a Slurm cluster for you.

nodeSelector:
  example: example

tolerations:
- key:
  value:
  operator:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.kubernetes.io/zone
          operator: In
          values:
          - zone-a
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 1
      preference:
        matchExpressions:
        - key: another-node-label-key
          operator: In
          values:
          - another-node-label-value

resources:
  requests:
    cpu: 1
  limits:
    cpu: 1

imagePullSecrets:
- name: example

podSecurityContext:
  runAsUser: 1000
  runAsGroup: 3000
  fsGroup: 2000
  supplementalGroups: [4000]

securityContext:
  allowPrivilegeEscalation: false

Perform the following operations:

1. Run the following command to add a chart repository provided by Alibaba Cloud to your Helm client: After the repository is added, you can access the Helm charts provided by Alibaba Cloud, such as the chart of the ack-slurm-cluster component.

   helm repo add aliyun https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/charts-incubator/

2. Run the following command to pull and decompress the ack-slurm-cluster chart. This operation creates a subdirectory named ack-slurm-cluster in the current directory. The ack-slurm-cluster directory contains all the files and templates included in the chart.

   helm pull aliyun/ack-slurm-cluster --untar=true

3. Run the following commands to modify the chart parameters in the values.yaml file.

   The values.yaml file contains the default configurations of the chart. You can modify the parameter settings in the file based on your business requirements. The settings include Slurm configurations, resource requests and limits, and storage configurations.

   cd ack-slurm-cluster
   vi values.yaml

   Show the steps to generate a JSON Web Token (JWT) and submit the JWT to the cluster

   Generate a key for a JWT authentication plug-in and run a command to import the key to the cluster. For more information, see JWT authentication plug-in.

   1. Obtain a JSON Web Key (JWK) to sign and authenticate a JWT.

      A JWT authentication plug-in uses a JWK (RFC7517) to sign and authenticate a JWT. To configure a JWT authentication plug-in, you must first generate a valid JWK. You can manually generate a JWK or use an online JWK generator, such as mkjwk.org, to generate a JWK. The following sample code shows the content of a JWK. The private key is used to sign the JWT. The public key in the JWT authentication plug-in is used to authenticate the JWT. Sample JWK content:

      {
        "kty": "RSA",
        "e": "AQAB",
        "kid": "O9fpdhrViq2zaaaBEWZITz",
        "use": "sig",
        "alg": "RS256",
        "n": "qSVxcknOm0uCq5vGsOmaorPDzHUubBmZZ4UXj-9do7w9X1uKFXAnqfto4TepSNuYU2bA_-tzSLAGBsR-BqvT6w9SjxakeiyQpVmexxnDw5WZwpWenUAcYrfSPEoNU-0hAQwFYgqZwJQMN8ptxkd0170PFauwACOx4Hfr-9FPGy8NCoIO4MfLXzJ3mJ7xqgIZp3NIOGXz-GIAbCf13ii7kSStpYqN3L_zzpvXUAos1FJ9IPXRV84tIZpFVh2lmRh0h8ImK-vI42dwlD_hOIzayL1Xno2R0T-d5AwTSdnep7g-Fwu8-sj4cCRWq3bd61Zs2QOJ8iustH0vSRMYdP5oYQ"
      }        

      The preceding JWK is in the JSON format. If you want to configure a JWT authentication plug-in in the YAML format, you must use a JWK in the YAML format.*

      • For a JWT authentication plug-in, you need only to configure a public key. Keep your private key credential. The following table describes the signature signing algorithms supported by the JWT authentication plug-in.

      Signature algorithm	Supported alg setting
      RSASSA-PKCS1-V1_5 with SHA-2	RS256, RS384, RS512
      Elliptic Curve (ECDSA) with SHA-2	ES256, ES384, ES512
      HMAC using SHA-2	HS256, HS384, HS512

      Important

      When you configure a key of the HS256, HS384, or HS512 type, the key value is Base64URL-encoded. If the signature is invalid, check whether your key is in the same format as the key used to generate the token.

   2. Import the JWK to the cluster.

      kubectl create configmap jwt --from-literal=jwt_hs256.key={{ .jwtkey }}

   Show the steps to specify a database address and configure Generic Resources (GREs)

   Enable slurmrestd and slurmdbd. Modify the .Values.slurmConfigs.filesInConfigMap parameter in the values.yaml file to specify a database address and configure GREs.

   slurmConfigs:
     ...
     filesInConfigMap:
       gres.conf: |
         # Configure Copilot to update the information about the resources allocated from Kubernetes to Slurm.
         Name=k8scpu Flags=CountOnly
         Name=k8smemory Flags=CountOnly
       slurmdbd.conf: |
         # The path of the logs. The path must be the same as the path specified in the following authentication configurations.
         LogFile=/var/log/slurmdbd.log
         # When you use slurmrestd, you must specify JWT authentication.
         AuthAltTypes=auth/jwt
         # slurmdbd needs to use the key in the following path to authenticate the token. Use the following configurations to mount the key to the pod.
         AuthAltParameters=jwt_key=/var/jwt/jwt_hs256.key
         AuthType=auth/munge
         SlurmUser=slurm
         # Specify a MySQL database account.
         StoragePass=
         StorageHost=
         StorageType=accounting_storage/mysql
         StorageUser=root
         StoragePort=3306
       slurm.conf: |
         # Specify the following extended resource attributes for nodes in the Slurm cluster: k8scpu and k8smemory. This prevents the nodes from entering the DOWN state.
         NodeFeaturesPlugins=node_features/k8s_resources
         # Enable Slurm to automatically add k8scpu and k8smemory when you use Slurm to submit jobs.
         JobSubmitPlugins=k8s_resource_completion
         AccountingStorageHost=slurm-test-slurmdbd
         # When you use slurmrestd, you must specify JWT authentication.
         AuthAltTypes=auth/jwt
         # slurmctld needs to use the key in the following path to authenticate the token. Use the following configurations to mount the key to the pod.
         AuthAltParameters=jwt_key=/var/jwt/jwt_hs256.key
         # Configure Copilot to update the information about the resources allocated from Kubernetes to Slurm.
         GresTypes=k8scpu,k8smemory
         # Specify ${slurmClusterName}-slurmdbd. slurmOperator automatically deploys slurmdbd.
         AccuntingStorageHost=
         AccountingStoragePort=6819
         AccountingStorageType=accounting_storage/slurmdbd
         # Information used by the JobComp plug-in to access the MySQL database.
         JobCompHost=
         JobCompLoc=/var/log/slurm/slurm_jobcomp.log
         JobCompPass=
         JobCompPort=3306
         JobCompType=jobcomp/mysql
         JobCompUser=root
         # Configurations that ensure high availability.
         SlurmctldHost=

   Show the steps to configure the slurmrestd pod and slurmdbd pod

   Configure the slurmrestd pod and slurmdbd pod.

   ...
   headNodeConfig:
     image: "registry-cn-hangzhou.ack.aliyuncs.com/acs/slurm:23.06-1.6-aliyun-49259f59"
     # Mount the generated JWT key to Slurm. The mount path must be the same as the path specified in the preceding section.
     volumes: 
     - configMap:
         defaultMode: 444
         name: jwt
       name: config-jwt
     volumeMounts: 
     - mountPath: /var/jwt
       name: config-jwt
   slurmdbdConfigs:
     nodeSelector: {}
     tolerations: []
     affinity: {}
     resources: {}
     image: "registry-cn-hangzhou.ack.aliyuncs.com/acs/slurm:23.06-1.6-aliyun-49259f59"
     imagePullSecrets: []
     # if .slurmConfigs.createConfigsByConfigMap is true, slurmConfPath and volume and volumeMounts will be auto set as:
     #  volumeMounts:
     #    - name: config-{{ .Values.slurmConfigs.configMapName }}
     #      mountPath: {{ .Values.slurmConfigs.slurmConfigPathInPod }}
     # volumes:
     #   - name: config-{{ .Values.slurmConfigs.configMapName }}
     #     configMap:
     #       name: {{ .Values.slurmConfigs.configMapName }}
     # also for mungeConfigs.createConfigsBySecret
     # Mount the generated JWT key to Slurm. The mount path must be the same as the path specified in the preceding section.
   
     volumes: 
     - configMap:
         defaultMode: 444
         name: jwt
       name: config-jwt
     volumeMounts: 
     - mountPath: /var/jwt
       name: config-jwt
   
   slurmrestdConfigs:
     nodeSelector: {}
     tolerations: []
     affinity: {}
     resources: {}
     image: "registry-cn-hangzhou.ack.aliyuncs.com/acs/slurm:23.06-1.6-aliyun-49259f59"
     imagePullSecrets: []
     # if .slurmConfigs.createConfigsByConfigMap is true, slurmConfPath and volume and volumeMounts will be auto set as:
     #  volumeMounts:
     #    - name: config-{{ .Values.slurmConfigs.configMapName }}
     #      mountPath: {{ .Values.slurmConfigs.slurmConfigPathInPod }}
     # volumes:
     #   - name: config-{{ .Values.slurmConfigs.configMapName }}
     #     configMap:
     #       name: {{ .Values.slurmConfigs.configMapName }}
     # also for mungeConfigs.createConfigsBySecret
     # Mount the generated JWT key to Slurm. The mount path must be the same as the path specified in the preceding section.
     volumes: 
     - configMap:
         defaultMode: 444
         name: jwt
       name: config-jwt
     volumeMounts: 
     - mountPath: /var/jwt
       name: config-jwt

4. Run the following command to install the ack-slurm-cluster chart. If the ack-slurm-cluster chart is already installed, you can run the helm upgrade command to update the installed chart. After you update the installed chart, you must manually delete the existing pod and the StatefulSet created for slurmctld to make the update take effect.

   cd ..
   helm install my-slurm-cluster ack-slurm-cluster # Replace my-slurm-cluster with the actual value.

5. After installing the chart, run the helm list command to check whether the ack-slurm-cluster chart is successfully installed.

   helm list

   Expected output:

   NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
   ack-slurm-cluster       default         1               2024-07-19 14:47:58.126357 +0800 CST    deployed        ack-slurm-cluster-2.0.0 2.0.0      

6. Check whether slurmrestd and slurmdbd run as expected.

   1. Use kubectl to connect to the cluster and check whether the slurmdbd pod runs as expected.

      kubectl get pod

      The following sample output shows that one worker pod and three control plane pods run in the cluster.

      NAME                          READY   STATUS    RESTARTS   AGE
      slurm-test-slurmctld-dlncz    1/1     Running   0          3h49m
      slurm-test-slurmdbd-8f75r     1/1     Running   0          3h49m
      slurm-test-slurmrestd-mjdzt   1/1     Running   0          3h49m
      slurm-test-worker-cpu-0       1/1     Running   0          166m

   2. Run the following command to query logs. You can view the logs to check whether slurmdbd runs as expected.

      kubectl exec slurm-test-slurmdbd-8f75r cat /var/log/slurmdbd.log | head

      Expected output:

      kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
      [2024-07-22T19:52:55.727] accounting_storage/as_mysql: _check_mysql_concat_is_sane: MySQL server version is: 8.0.34
      [2024-07-22T19:52:55.737] error: Database settings not recommended values: innodb_lock_wait_timeout
      [2024-07-22T19:52:56.089] slurmdbd version 23.02.7 started

2.1 Test colocated scheduling

1. Check the status of GenericNodes to view Slurm workloads and Kubernetes workloads.

   kubectl get genericnode

   Expected output:

   NAME                    CLUSTERNAME   ALIAS                     TYPE    ALLOCATEDRESOURCES
   cn-hongkong.10.1.0.19                 slurm-test-worker-cpu-0   Slurm   [{"allocated":{"cpu":"0","memory":"0"},"type":"Slurm"},{"allocated":{"cpu":"1735m","memory":"2393Mi"},"type":"Kubernetes"}]

2. Run the following commands to submit a job to the Slurm cluster and query GenericNodes. The returned GenericNode records the resource usage of a job in the Slurm cluster and the resource usage of a job in the Kubernetes cluster.

   root@iZj6c1wf3c25dbynbna3qgZ ~]# kubectl exec slurm-test-slurmctld-dlncz -- nohup srun --cpus-per-task=3 --mem=4000 --gres=k8scpu:3,k8smemory:4000 sleep inf &
   [1] 4132674
   
   [root@iZj6c1wf3c25dbynbna3qgZ ~]# kubectl scale deployment nginx-deployment-basic --replicas 2
   deployment.apps/nginx-deployment-basic scaled
   
   [root@iZj6c1wf3c25dbynbna3qgZ ~]# kubectl get genericnode
   NAME                    CLUSTERNAME   ALIAS                     TYPE    ALLOCATEDRESOURCES
   cn-hongkong.10.1.0.19                 slurm-test-worker-cpu-0   Slurm   [{"allocated":{"cpu":"3","memory":"4000Mi"},"type":"Slurm"},{"allocated":{"cpu":"2735m","memory":"3417Mi"},"type":"Kubernetes"}]

3. In this case, if you submit another job to the Slurm cluster, the second job enters the Pending state.

   [root@iZj6c1wf3c25dbynbna3qgZ ~]# kubectl exec slurm-test-slurmctld-dlncz -- nohup srun --cpus-per-task=3 --mem=4000 sleep inf &
   [2] 4133454
   
   [root@iZj6c1wf3c25dbynbna3qgZ ~]# srun: job 2 queued and waiting for resources
   
   [root@iZj6c1wf3c25dbynbna3qgZ ~]# kubectl exec slurm-test-slurmctld-dlncz -- squeue
    JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
        2     debug    sleep     root PD       0:00      1 (Resources)
        1     debug    sleep     root  R       2:34      1 slurm-test-worker-cpu-0

In this example, no GREs are manually specified. However, GREs are still displayed in the preceding srun command. This is because the Slurm cluster is pre-installed with the job_resource_completion plug-in, which automatically adds GREs based on the CPU request and memory request. If job_resource_completion is not installed, you need to manually specify GREs. In this example, the following GRE configuration is specified: --gres=k8scpu:3,k8smemory:4000. Click Show the Slurm job script description to view how to specify parameters in the Slurm job script.

#!/bin/bash
#SBATCH --job-name=test_job                   # The job name.
#SBATCH --nodes=2                             # The number of nodes required by the job.
#SBATCH --ntasks-per-node=4                   # The number of tasks to run on each node.
#SBATCH --cpus-per-task=2                     # The number of vCPUs required by each task.
#SBATCH --time=01:00:00                       # The maximum duration of the job.
#SBATCH --output=job_output_%j.txt            # The name of the stdout file.
#SBATCH --error=job_error_%j.txt              # The name of the stderr file.

# User-defined job commands.
srun my_program

sbatch --nodes=2 --ntasks-per-node=4 --cpus-per-task=2 --time=01:00:00 --job-name=test_job my_job_script.sh

Optional: 2.2 Implement colocated scheduling in non-containerized Slurm clusters