Container Service for Kubernetes:Deploy containerized Slurm on ACK

Slurm

Slurm is a powerful, open-source platform for cluster resource management and job scheduling, specifically designed to optimize the performance and efficiency of supercomputers and large compute clusters. Its core components work together to ensure efficient system operation and flexible management. The following figure shows how Slurm works.

• slurmctld (Slurm Control Daemon): As the central controller for Slurm, slurmctld monitors system resources, schedules jobs, and manages the overall state of the cluster. For high availability, you can configure a standby slurmctld to prevent service interruptions if the primary controller fails.

• slurmd (Slurm Node Daemon): Deployed on each compute node, the slurmd daemon receives instructions from slurmctld and executes jobs. It starts and runs jobs, reports job status, and prepares to accept new work. It acts as a direct interface to compute resources and is fundamental to job execution.

• slurmdbd (Slurm Database Daemon): Although an optional component, slurmdbd is critical for the long-term management and auditing of large-scale clusters. It maintains a centralized database to store job history and accounting information. It also supports data aggregation across multiple Slurm-managed clusters, which improves data management efficiency.

• SlurmCLI: Provides a suite of command-line tools to facilitate job management and system monitoring:

  • scontrol: Provides detailed control over cluster management and configuration.

  • squeue: Queries the status of the job queue.

  • srun: Submits and manages jobs.

  • sbatch: Submits a batch job.

  • sinfo: Displays the overall state of the cluster, including node availability.

Slurm on ACK

The Slurm Operator uses the SlurmCluster CustomResource (CR) to simplify the deployment and operation of Slurm clusters. This approach simplifies managing the configuration files and control plane, reducing overall complexity. The following figure shows the architecture of Slurm on ACK. A cluster administrator can deploy and manage a Slurm cluster simply by creating a SlurmCluster object. The Slurm Operator then automatically creates the corresponding Slurm control plane components. You can mount Slurm configuration files to these components by using shared storage or a ConfigMap.

For Kubernetes cluster administrators

Kubernetes cluster administrators have full operational permissions on the cluster. Because a Slurm-managed cluster runs as pods within the Kubernetes cluster, an administrator can use the kubectl command-line tool to log on to any of its pods. This access automatically grants them root permissions within the Slurm-managed cluster.

Run the following command to log on to any pod of the Slurm-managed cluster.

# Replace slurm-job-demo-xxxxx with the name of a specific pod in your cluster.
kubectl exec -it slurm-job-demo-xxxxx -- bash

For regular Slurm cluster users

Administrators or regular users of a Slurm-managed cluster might not have the permissions to run the kubectl exec command. In this case, you must log on to the Slurm-managed cluster by using SSH.

• Using a Service's external IP address to log on to a head pod provides a persistent, scalable solution for long-term, stable access. This method uses a load balancer, allowing you to access the Slurm-managed cluster from any location within your internal network.

• Using port forwarding is a temporary solution for short-term operations or debugging because it requires the kubectl port-forward command to run continuously.

User synchronization across nodes

Slurm does not provide a built-in service for centralized user authentication. When you submit a job to a SlurmCluster by using the sbatch command, the job may fail if the user account does not exist on the target node. To resolve this, you can configure Lightweight Directory Access Protocol (LDAP) as a centralized authentication backend for your SlurmCluster. This allows Slurm to verify user identities through the LDAP service. Perform the following steps:

1. Create a file named ldap.yaml with the following content. This configuration deploys a basic LDAP service instance for storing and managing user information.

   The ldap.yaml file defines a Pod to run the LDAP service and a Service to expose it on the network.

   LDAP backend Pod and Service

   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     namespace: default
     name: ldap
     labels:
       app: ldap
   spec:
     selector:
       matchLabels:
         app: ldap
     revisionHistoryLimit: 10
     template:
       metadata:
         labels:
           app: ldap
       spec:
         securityContext:
           seLinuxOptions: {}
         imagePullSecrets: []
         restartPolicy: Always
         initContainers: []
         containers:
           - image: 'osixia/openldap:1.4.0'
             imagePullPolicy: IfNotPresent
             name: ldap
             volumeMounts:
               - name: openldap-data
                 mountPath: /var/lib/ldap
                 subPath: data
               - name: openldap-data
                 mountPath: /etc/ldap/slapd.d
                 subPath: config
               - name: openldap-data
                 mountPath: /container/service/slapd/assets/certs
                 subPath: certs
               - name: secret-volume
                 mountPath: /container/environment/01-custom
               - name: container-run
                 mountPath: /container/run
             args:
               - '--copy-service'
             resources:
               limits:
               requests:
             env: []
             readinessProbe:
               tcpSocket:
                 port: openldap
               initialDelaySeconds: 20
               timeoutSeconds: 1
               periodSeconds: 10
               successThreshold: 1
               failureThreshold: 10
             livenessProbe:
               tcpSocket:
                 port: openldap
               initialDelaySeconds: 20
               timeoutSeconds: 1
               periodSeconds: 10
               successThreshold: 1
               failureThreshold: 10
             lifecycle: {}
             ports:
               - name: openldap
                 containerPort: 389
                 protocol: TCP
               - name: ssl-ldap-port
                 containerPort: 636
                 protocol: TCP
         volumes:
           - name: openldap-data
             emptyDir: {}
           - name: secret-volume
             secret:
               secretName: ldap-secret
               defaultMode: 420
               items: []
           - name: container-run
             emptyDir: {}
         dnsPolicy: ClusterFirst
         dnsConfig: {}
         terminationGracePeriodSeconds: 30
     progressDeadlineSeconds: 600
     strategy:
       type: RollingUpdate
       rollingUpdate:
         maxUnavailable: 25%
         maxSurge: 25%
     replicas: 1
   ---
   apiVersion: v1
   kind: Service
   metadata:
     annotations: {}
     labels:
       app: ldap
     name: ldap-service
     namespace: default
   spec:
     ports:
       - name: openldap
         port: 389
         protocol: TCP
         targetPort: openldap
       - name: ssl-ldap-port
         port: 636
         protocol: TCP
         targetPort: ssl-ldap-port
     selector:
       app: ldap
     sessionAffinity: None
     type: ClusterIP
   ---
   metadata:
     name: ldap-secret
     namespace: default
     annotations: {}
   data:
     env.startup.yaml: >-
       IyBUaGlzIGlzIHRoZSBkZWZhdWx0IGltYWdlIHN0YXJ0dXAgY29uZmlndXJhdGlvbiBmaWxlCiMgdGhpcyBmaWxlIGRlZmluZSBlbnZpcm9ubWVudCB2YXJpYWJsZXMgdXNlZCBkdXJpbmcgdGhlIGNvbnRhaW5lciAqKmZpcnN0IHN0YXJ0KiogaW4gKipzdGFydHVwIGZpbGVzKiouCgojIFRoaXMgZmlsZSBpcyBkZWxldGVkIHJpZ2h0IGFmdGVyIHN0YXJ0dXAgZmlsZXMgYXJlIHByb2Nlc3NlZCBmb3IgdGhlIGZpcnN0IHRpbWUsCiMgYWZ0ZXIgdGhhdCBhbGwgdGhlc2UgdmFsdWVzIHdpbGwgbm90IGJlIGF2YWlsYWJsZSBpbiB0aGUgY29udGFpbmVyIGVudmlyb25tZW50LgojIFRoaXMgaGVscHMgdG8ga2VlcCB5b3VyIGNvbnRhaW5lciBjb25maWd1cmF0aW9uIHNlY3JldC4KIyBtb3JlIGluZm9ybWF0aW9uIDogaHR0cHM6Ly9naXRodWIuY29tL29zaXhpYS9kb2NrZXItbGlnaHQtYmFzZWltYWdlCgojIFJlcXVpcmVkIGFuZCB1c2VkIGZvciBuZXcgbGRhcCBzZXJ2ZXIgb25seQpMREFQX09SR0FOSVNBVElPTjogRXhhbXBsZSBJbmMuCkxEQVBfRE9NQUlOOiBleGFtcGxlLm9yZwpMREFQX0JBU0VfRE46ICNpZiBlbXB0eSBhdXRvbWF0aWNhbGx5IHNldCBmcm9tIExEQVBfRE9NQUlOCgpMREFQX0FETUlOX1BBU1NXT1JEOiBhZG1pbgpMREFQX0NPTkZJR19QQVNTV09SRDogY29uZmlnCgpMREFQX1JFQURPTkxZX1VTRVI6IGZhbHNlCkxEQVBfUkVBRE9OTFlfVVNFUl9VU0VSTkFNRTogcmVhZG9ubHkKTERBUF9SRUFET05MWV9VU0VSX1BBU1NXT1JEOiByZWFkb25seQoKIyBCYWNrZW5kCkxEQVBfQkFDS0VORDogaGRiCgojIFRscwpMREFQX1RMUzogdHJ1ZQpMREFQX1RMU19DUlRfRklMRU5BTUU6IGxkYXAuY3J0CkxEQVBfVExTX0tFWV9GSUxFTkFNRTogbGRhcC5rZXkKTERBUF9UTFNfQ0FfQ1JUX0ZJTEVOQU1FOiBjYS5jcnQKCkxEQVBfVExTX0VORk9SQ0U6IGZhbHNlCkxEQVBfVExTX0NJUEhFUl9TVUlURTogU0VDVVJFMjU2Oi1WRVJTLVNTTDMuMApMREFQX1RMU19QUk9UT0NPTF9NSU46IDMuMQpMREFQX1RMU19WRVJJRllfQ0xJRU5UOiBkZW1hbmQKCiMgUmVwbGljYXRpb24KTERBUF9SRVBMSUNBVElPTjogZmFsc2UKIyB2YXJpYWJsZXMgJExEQVBfQkFTRV9ETiwgJExEQVBfQURNSU5fUEFTU1dPUkQsICRMREFQX0NPTkZJR19QQVNTV09SRAojIGFyZSBhdXRvbWF0aWNhbHkgcmVwbGFjZWQgYXQgcnVuIHRpbWUKCiMgaWYgeW91IHdhbnQgdG8gYWRkIHJlcGxpY2F0aW9uIHRvIGFuIGV4aXN0aW5nIGxkYXAKIyBhZGFwdCBMREFQX1JFUExJQ0FUSU9OX0NPTkZJR19TWU5DUFJPViBhbmQgTERBUF9SRVBMSUNBVElPTl9EQl9TWU5DUFJPViB0byB5b3VyIGNvbmZpZ3VyYXRpb24KIyBhdm9pZCB1c2luZyAkTERBUF9CQVNFX0ROLCAkTERBUF9BRE1JTl9QQVNTV09SRCBhbmQgJExEQVBfQ09ORklHX1BBU1NXT1JEIHZhcmlhYmxlcwpMREFQX1JFUExJQ0FUSU9OX0NPTkZJR19TWU5DUFJPVjogYmluZGRuPSJjbj1hZG1pbixjbj1jb25maWciIGJpbmRtZXRob2Q9c2ltcGxlIGNyZWRlbnRpYWxzPSRMREFQX0NPTkZJR19QQVNTV09SRCBzZWFyY2hiYXNlPSJjbj1jb25maWciIHR5cGU9cmVmcmVzaEFuZFBlcnNpc3QgcmV0cnk9IjYwICsiIHRpbWVvdXQ9MSBzdGFydHRscz1jcml0aWNhbApMREFQX1JFUExJQ0FUSU9OX0RCX1NZTkNQUk9WOiBiaW5kZG49ImNuPWFkbWluLCRMREFQX0JBU0VfRE4iIGJpbmRtZXRob2Q9c2ltcGxlIGNyZWRlbnRpYWxzPSRMREFQX0FETUlOX1BBU1NXT1JEIHNlYXJjaGJhc2U9IiRMREFQX0JBU0VfRE4iIHR5cGU9cmVmcmVzaEFuZFBlcnNpc3QgaW50ZXJ2YWw9MDA6MDA6MDA6MTAgcmV0cnk9IjYwICsiIHRpbWVvdXQ9MSBzdGFydHRscz1jcml0aWNhbApMREFQX1JFUExJQ0FUSU9OX0hPU1RTOgogIC0gbGRhcDovL2xkYXAuZXhhbXBsZS5vcmcgIyBUaGUgb3JkZXIgbXVzdCBiZSB0aGUgc2FtZSBvbiBhbGwgbGRhcCBzZXJ2ZXJzCiAgLSBsZGFwOi8vbGRhcDIuZXhhbXBsZS5vcmcKCgojIFJlbW92ZSBjb25maWcgYWZ0ZXIgc2V0dXAKTERBUF9SRU1PVkVfQ09ORklHX0FGVEVSX1NFVFVQOiB0cnVlCgojIGNmc3NsIGVudmlyb25tZW50IHZhcmlhYmxlcyBwcmVmaXgKTERBUF9DRlNTTF9QUkVGSVg6IGxkYXAgIyBjZnNzbC1oZWxwZXIgZmlyc3Qgc2VhcmNoIGNvbmZpZyBmcm9tIExEQVBfQ0ZTU0xfKiB2YXJpYWJsZXMsIGJlZm9yZSBDRlNTTF8qIHZhcmlhYmxlcy4K
     env.yaml: >-
       IyBUaGlzIGlzIHRoZSBkZWZhdWx0IGltYWdlIGNvbmZpZ3VyYXRpb24gZmlsZQojIFRoZXNlIHZhbHVlcyB3aWxsIHBlcnNpc3RzIGluIGNvbnRhaW5lciBlbnZpcm9ubWVudC4KCiPCoEFsbCBlbnZpcm9ubWVudCB2YXJpYWJsZXMgdXNlZCBhZnRlciB0aGUgY29udGFpbmVyIGZpcnN0IHN0YXJ0CiMgbXVzdCBiZSBkZWZpbmVkIGhlcmUuCiMgbW9yZSBpbmZvcm1hdGlvbiA6IGh0dHBzOi8vZ2l0aHViLmNvbS9vc2l4aWEvZG9ja2VyLWxpZ2h0LWJhc2VpbWFnZQoKIyBHZW5lcmFsIGNvbnRhaW5lciBjb25maWd1cmF0aW9uCiMgc2VlIHRhYmxlIDUuMSBpbiBodHRwOi8vd3d3Lm9wZW5sZGFwLm9yZy9kb2MvYWRtaW4yNC9zbGFwZGNvbmYyLmh0bWwgZm9yIHRoZSBhdmFpbGFibGUgbG9nIGxldmVscy4KTERBUF9MT0dfTEVWRUw6IDI1Ngo=
   type: Opaque
   kind: Secret
   apiVersion: v1
   

2. Run the following command to deploy the LDAP backend service:

   kubectl apply -f ldap.yaml

   Expected output:

   deployment.apps/ldap created
   service/ldap-service created
   secret/ldap-secret created

3. (Optional) To improve management efficiency, you can deploy a frontend interface. Create a file named phpldapadmin.yaml with the following content to deploy a frontend Pod and Service.

   LDAP frontend Pod and Service

   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     namespace: default
     name: phpldapadmin
     labels:
       io.kompose.service: phpldapadmin
   spec:
     selector:
       matchLabels:
         io.kompose.service: phpldapadmin
     revisionHistoryLimit: 10
     template:
       metadata:
         labels:
           io.kompose.service: phpldapadmin
       spec:
         securityContext:
           seLinuxOptions: {}
         imagePullSecrets: []
         restartPolicy: Always
         initContainers: []
         containers:
           - image: 'osixia/phpldapadmin:0.9.0'
             imagePullPolicy: Always
             name: phpldapadmin
             volumeMounts: []
             resources:
               limits:
               requests:
             env:
               - name: PHPLDAPADMIN_HTTPS
                 value: 'false'
               - name: PHPLDAPADMIN_LDAP_HOSTS
                 value: ldap-service
             lifecycle: {}
             ports:
               - containerPort: 80
                 protocol: TCP
         volumes: []
         dnsPolicy: ClusterFirst
         dnsConfig: {}
         terminationGracePeriodSeconds: 30
     progressDeadlineSeconds: 600
     strategy:
       type: RollingUpdate
       rollingUpdate:
         maxUnavailable: 25%
         maxSurge: 25%
     replicas: 1
   ---
   apiVersion: v1
   kind: Service
   metadata:
     namespace: default
     name: phpldapadmin
     annotations:
       k8s.kuboard.cn/workload: phpldapadmin
     labels:
       io.kompose.service: phpldapadmin
   spec:
     selector:
       io.kompose.service: phpldapadmin
     type: ClusterIP
     ports:
       - port: 8080
         targetPort: 80
         protocol: TCP
         name: '8080'
         nodePort: 0
     sessionAffinity: None

   Run the following command to deploy the LDAP frontend service:

   kubectl apply -f phpldapadmin.yaml

4. Log on to a Pod in the SlurmCluster as described in Step 3. Then, run the following commands to install the LDAP client package:

   apt update
   apt install libnss-ldapd

5. After the libnss-ldapd package is installed, configure the network authentication service for the SlurmCluster from within the Pod.

   1. Run the following commands to install the Vim package for editing scripts and files:

      apt update
      apt install vim

   2. Modify the following parameters in the /etc/ldap/ldap.conf file to configure the LDAP client:

      ...
      BASE	dc=example,dc=org # Replace this with the base DN of your LDAP directory.
      URI	ldap://ldap-service # Replace this with the address of your LDAP server.
      ...

   3. Modify the following parameters in the /etc/nslcd.conf file to define the connection to the LDAP server:

      ...
      uri ldap://ldap-service # Replace this with the address of your LDAP server.
      base dc=example,dc=org # Set this based on your LDAP directory structure.
      ...
      tls_cacertfile /etc/ssl/certs/ca-certificates.crt # Specifies the path to the CA certificate file used to verify the LDAP server certificate.
      ...

Log sharing and access

By default, job logs generated by sbatch are stored directly on the node where the job runs, which can make viewing logs inconvenient. To centralize log access, you can create a NAS file system to store all job logs. This collects logs from all nodes in a single location, simplifying management. Perform the following steps:

1. Create a NAS file system to store and share logs from all nodes. For more information, see Create a file system.

2. Log on to the ACK console and create a Persistent Volume (PV) and a Persistent Volume Claim (PVC) for the NAS file system. For more information, see Use a statically provisioned NAS volume.

3. Modify the SlurmCluster CR.

   Add the volumeMounts and volumes parameters to headGroupSpec and each workerGroupSpec to reference the created PVC and mount it to the /home directory. The following is an example:

   headGroupSpec:
   ...
   # Add a volume mount for /home.
     volumeMounts:
     - mountPath: /home
       name: test  # The name of the volume that references the PVC.
     volumes:
   # Add the PVC definition.
     - name: test  # This must match the name in volumeMounts.
       persistentVolumeClaim:
         claimName: test  # Replace this with the name of your PVC.
   ...
   workerGroupSpecs:
     # ... Repeat the preceding volume and volumeMounts configuration for each workerGroupSpec.

4. Run the following command to apply the changes to the SlurmCluster CR:

   Important

   If the SlurmCluster CR resource fails to deploy, run the kubectl delete slurmcluster slurm-job-demo command to delete the CR resource, and then deploy it again.

   kubectl  apply -f slurmcluster.yaml

   After the deployment, all worker nodes share the same file system.

Auto scaling

The root directory of the default Slurm image includes executable files and scripts such as slurm-resume.sh, slurm-suspend.sh, and slurmctld-copilot. These components interact with slurmctld to manage cluster scaling.

Slurm auto scaling with cloud nodes

• local node: A physical compute node that is directly connected to the cluster manager.

• cloud node: A logical node that represents a VM instance that can be created and terminated on demand by a cloud provider.

Auto scaling in Slurm on ACK