Container Service for Kubernetes:Deploy and run a Slurm-managed cluster as a pod in an ACK cluster

Log on as a Kubernetes cluster administrator

An administrator of a Kubernetes cluster has the permissions to manage the Kubernetes cluster. A Slurm-managed cluster runs as a pod in the Kubernetes cluster. Therefore, the Kubernetes cluster administrator can use kubectl to log on to any pod of any Slurm-managed cluster in the Kubernetes cluster, and has the permissions of the root user of the Slurm-managed cluster by default.

Run the following command to log on to a pod of the Slurm-managed cluster:

# Replace slurm-job-demo-head-x9sgs with the name of the pod in your cluster. 
kubectl exec -it slurm-job-demo-xxxxx -- bash

Log on as a regular user of the Slurm-managed cluster

The administrators or regular users of a Slurm-managed cluster may not have the permissions to run the kubectl exec command. If you use a Slurm-managed cluster as an administrator or a regular user, you need to log on to the Slurm-managed cluster by using SSH.

• You can use an external IP address of a Service to log on to the head pod for long-term connections and scalability. This solution is suitable for scenarios in which long-term and stable connections are required. You can access the Slurm-managed cluster from anywhere within the internal network by using a Server Load Balancer (SLB) instance and its associated external IP address.

• You can use the port-forward command to forward requests for a temporary period. This solution is suitable for short-term O&M or debugging because it relies on the continuous execution of the kubectl port-forward command.

Synchronize users across nodes

Slurm does not provide a centralized user authentication service. When you run the sbatch command to submit jobs to the Slurm-managed cluster, the jobs may fail to be executed if the account of the user who submits the jobs does not exist on the node that is selected to execute the jobs. To resolve this issue, you can configure Lightweight Directory Access Protocol (LDAP) for the Slurm-managed cluster to manage users. LDAP serves as a centralized backend Service for authentication. This way, Slurm can authenticate user identities based on the Service. Perform the following operations:

1. Create a file named ldap.yaml and copy the following content to the file to create a basic LDAP instance that stores and manages user information.

   The ldap.yaml file defines an LDAP backend pod and its associated Service. The pod contains an LDAP container, and the Service exposes the LDAP service to make the LDAP service accessible within the network.

   View the LDAP backend pod and its associated Service

   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     namespace: default
     name: ldap
     labels:
       app: ldap
   spec:
     selector:
       matchLabels:
         app: ldap
     revisionHistoryLimit: 10
     template:
       metadata:
         labels:
           app: ldap
       spec:
         securityContext:
           seLinuxOptions: {}
         imagePullSecrets: []
         restartPolicy: Always
         initContainers: []
         containers:
           - image: 'osixia/openldap:1.4.0'
             imagePullPolicy: IfNotPresent
             name: ldap
             volumeMounts:
               - name: openldap-data
                 mountPath: /var/lib/ldap
                 subPath: data
               - name: openldap-data
                 mountPath: /etc/ldap/slapd.d
                 subPath: config
               - name: openldap-data
                 mountPath: /container/service/slapd/assets/certs
                 subPath: certs
               - name: secret-volume
                 mountPath: /container/environment/01-custom
               - name: container-run
                 mountPath: /container/run
             args:
               - '--copy-service'
             resources:
               limits:
               requests:
             env: []
             readinessProbe:
               tcpSocket:
                 port: openldap
               initialDelaySeconds: 20
               timeoutSeconds: 1
               periodSeconds: 10
               successThreshold: 1
               failureThreshold: 10
             livenessProbe:
               tcpSocket:
                 port: openldap
               initialDelaySeconds: 20
               timeoutSeconds: 1
               periodSeconds: 10
               successThreshold: 1
               failureThreshold: 10
             lifecycle: {}
             ports:
               - name: openldap
                 containerPort: 389
                 protocol: TCP
               - name: ssl-ldap-port
                 containerPort: 636
                 protocol: TCP
         volumes:
           - name: openldap-data
             emptyDir: {}
           - name: secret-volume
             secret:
               secretName: ldap-secret
               defaultMode: 420
               items: []
           - name: container-run
             emptyDir: {}
         dnsPolicy: ClusterFirst
         dnsConfig: {}
         terminationGracePeriodSeconds: 30
     progressDeadlineSeconds: 600
     strategy:
       type: RollingUpdate
       rollingUpdate:
         maxUnavailable: 25%
         maxSurge: 25%
     replicas: 1
   ---
   apiVersion: v1
   kind: Service
   metadata:
     annotations: {}
     labels:
       app: ldap
     name: ldap-service
     namespace: default
   spec:
     ports:
       - name: openldap
         port: 389
         protocol: TCP
         targetPort: openldap
       - name: ssl-ldap-port
         port: 636
         protocol: TCP
         targetPort: ssl-ldap-port
     selector:
       app: ldap
     sessionAffinity: None
     type: ClusterIP
   ---
   metadata:
     name: ldap-secret
     namespace: default
     annotations: {}
   data:
     env.startup.yaml: >-
       IyBUaGlzIGlzIHRoZSBkZWZhdWx0IGltYWdlIHN0YXJ0dXAgY29uZmlndXJhdGlvbiBmaWxlCiMgdGhpcyBmaWxlIGRlZmluZSBlbnZpcm9ubWVudCB2YXJpYWJsZXMgdXNlZCBkdXJpbmcgdGhlIGNvbnRhaW5lciAqKmZpcnN0IHN0YXJ0KiogaW4gKipzdGFydHVwIGZpbGVzKiouCgojIFRoaXMgZmlsZSBpcyBkZWxldGVkIHJpZ2h0IGFmdGVyIHN0YXJ0dXAgZmlsZXMgYXJlIHByb2Nlc3NlZCBmb3IgdGhlIGZpcnN0IHRpbWUsCiMgYWZ0ZXIgdGhhdCBhbGwgdGhlc2UgdmFsdWVzIHdpbGwgbm90IGJlIGF2YWlsYWJsZSBpbiB0aGUgY29udGFpbmVyIGVudmlyb25tZW50LgojIFRoaXMgaGVscHMgdG8ga2VlcCB5b3VyIGNvbnRhaW5lciBjb25maWd1cmF0aW9uIHNlY3JldC4KIyBtb3JlIGluZm9ybWF0aW9uIDogaHR0cHM6Ly9naXRodWIuY29tL29zaXhpYS9kb2NrZXItbGlnaHQtYmFzZWltYWdlCgojIFJlcXVpcmVkIGFuZCB1c2VkIGZvciBuZXcgbGRhcCBzZXJ2ZXIgb25seQpMREFQX09SR0FOSVNBVElPTjogRXhhbXBsZSBJbmMuCkxEQVBfRE9NQUlOOiBleGFtcGxlLm9yZwpMREFQX0JBU0VfRE46ICNpZiBlbXB0eSBhdXRvbWF0aWNhbGx5IHNldCBmcm9tIExEQVBfRE9NQUlOCgpMREFQX0FETUlOX1BBU1NXT1JEOiBhZG1pbgpMREFQX0NPTkZJR19QQVNTV09SRDogY29uZmlnCgpMREFQX1JFQURPTkxZX1VTRVI6IGZhbHNlCkxEQVBfUkVBRE9OTFlfVVNFUl9VU0VSTkFNRTogcmVhZG9ubHkKTERBUF9SRUFET05MWV9VU0VSX1BBU1NXT1JEOiByZWFkb25seQoKIyBCYWNrZW5kCkxEQVBfQkFDS0VORDogaGRiCgojIFRscwpMREFQX1RMUzogdHJ1ZQpMREFQX1RMU19DUlRfRklMRU5BTUU6IGxkYXAuY3J0CkxEQVBfVExTX0tFWV9GSUxFTkFNRTogbGRhcC5rZXkKTERBUF9UTFNfQ0FfQ1JUX0ZJTEVOQU1FOiBjYS5jcnQKCkxEQVBfVExTX0VORk9SQ0U6IGZhbHNlCkxEQVBfVExTX0NJUEhFUl9TVUlURTogU0VDVVJFMjU2Oi1WRVJTLVNTTDMuMApMREFQX1RMU19QUk9UT0NPTF9NSU46IDMuMQpMREFQX1RMU19WRVJJRllfQ0xJRU5UOiBkZW1hbmQKCiMgUmVwbGljYXRpb24KTERBUF9SRVBMSUNBVElPTjogZmFsc2UKIyB2YXJpYWJsZXMgJExEQVBfQkFTRV9ETiwgJExEQVBfQURNSU5fUEFTU1dPUkQsICRMREFQX0NPTkZJR19QQVNTV09SRAojIGFyZSBhdXRvbWF0aWNhbHkgcmVwbGFjZWQgYXQgcnVuIHRpbWUKCiMgaWYgeW91IHdhbnQgdG8gYWRkIHJlcGxpY2F0aW9uIHRvIGFuIGV4aXN0aW5nIGxkYXAKIyBhZGFwdCBMREFQX1JFUExJQ0FUSU9OX0NPTkZJR19TWU5DUFJPViBhbmQgTERBUF9SRVBMSUNBVElPTl9EQl9TWU5DUFJPViB0byB5b3VyIGNvbmZpZ3VyYXRpb24KIyBhdm9pZCB1c2luZyAkTERBUF9CQVNFX0ROLCAkTERBUF9BRE1JTl9QQVNTV09SRCBhbmQgJExEQVBfQ09ORklHX1BBU1NXT1JEIHZhcmlhYmxlcwpMREFQX1JFUExJQ0FUSU9OX0NPTkZJR19TWU5DUFJPVjogYmluZGRuPSJjbj1hZG1pbixjbj1jb25maWciIGJpbmRtZXRob2Q9c2ltcGxlIGNyZWRlbnRpYWxzPSRMREFQX0NPTkZJR19QQVNTV09SRCBzZWFyY2hiYXNlPSJjbj1jb25maWciIHR5cGU9cmVmcmVzaEFuZFBlcnNpc3QgcmV0cnk9IjYwICsiIHRpbWVvdXQ9MSBzdGFydHRscz1jcml0aWNhbApMREFQX1JFUExJQ0FUSU9OX0RCX1NZTkNQUk9WOiBiaW5kZG49ImNuPWFkbWluLCRMREFQX0JBU0VfRE4iIGJpbmRtZXRob2Q9c2ltcGxlIGNyZWRlbnRpYWxzPSRMREFQX0FETUlOX1BBU1NXT1JEIHNlYXJjaGJhc2U9IiRMREFQX0JBU0VfRE4iIHR5cGU9cmVmcmVzaEFuZFBlcnNpc3QgaW50ZXJ2YWw9MDA6MDA6MDA6MTAgcmV0cnk9IjYwICsiIHRpbWVvdXQ9MSBzdGFydHRscz1jcml0aWNhbApMREFQX1JFUExJQ0FUSU9OX0hPU1RTOgogIC0gbGRhcDovL2xkYXAuZXhhbXBsZS5vcmcgIyBUaGUgb3JkZXIgbXVzdCBiZSB0aGUgc2FtZSBvbiBhbGwgbGRhcCBzZXJ2ZXJzCiAgLSBsZGFwOi8vbGRhcDIuZXhhbXBsZS5vcmcKCgojIFJlbW92ZSBjb25maWcgYWZ0ZXIgc2V0dXAKTERBUF9SRU1PVkVfQ09ORklHX0FGVEVSX1NFVFVQOiB0cnVlCgojIGNmc3NsIGVudmlyb25tZW50IHZhcmlhYmxlcyBwcmVmaXgKTERBUF9DRlNTTF9QUkVGSVg6IGxkYXAgIyBjZnNzbC1oZWxwZXIgZmlyc3Qgc2VhcmNoIGNvbmZpZyBmcm9tIExEQVBfQ0ZTU0xfKiB2YXJpYWJsZXMsIGJlZm9yZSBDRlNTTF8qIHZhcmlhYmxlcy4K
     env.yaml: >-
       IyBUaGlzIGlzIHRoZSBkZWZhdWx0IGltYWdlIGNvbmZpZ3VyYXRpb24gZmlsZQojIFRoZXNlIHZhbHVlcyB3aWxsIHBlcnNpc3RzIGluIGNvbnRhaW5lciBlbnZpcm9ubWVudC4KCiPCoEFsbCBlbnZpcm9ubWVudCB2YXJpYWJsZXMgdXNlZCBhZnRlciB0aGUgY29udGFpbmVyIGZpcnN0IHN0YXJ0CiMgbXVzdCBiZSBkZWZpbmVkIGhlcmUuCiMgbW9yZSBpbmZvcm1hdGlvbiA6IGh0dHBzOi8vZ2l0aHViLmNvbS9vc2l4aWEvZG9ja2VyLWxpZ2h0LWJhc2VpbWFnZQoKIyBHZW5lcmFsIGNvbnRhaW5lciBjb25maWd1cmF0aW9uCiMgc2VlIHRhYmxlIDUuMSBpbiBodHRwOi8vd3d3Lm9wZW5sZGFwLm9yZy9kb2MvYWRtaW4yNC9zbGFwZGNvbmYyLmh0bWwgZm9yIHRoZSBhdmFpbGFibGUgbG9nIGxldmVscy4KTERBUF9MT0dfTEVWRUw6IDI1Ngo=
   type: Opaque
   kind: Secret
   apiVersion: v1
   

2. Run the following command to deploy the LDAP backend Service:

   kubectl apply -f ldap.yaml

   Expected output:

   deployment.apps/ldap created
   service/ldap-service created
   secret/ldap-secret created

3. Optional. Create a file named phpldapadmin.yaml and copy the following content to the file to deploy an LDAP frontend pod and its associated Service. The LDAP frontend pod and its associated Service are used to configure frontend interfaces to improve management efficiency.

   View the LDAP frontend pod and its associated Service

   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     namespace: default
     name: phpldapadmin
     labels:
       io.kompose.service: phpldapadmin
   spec:
     selector:
       matchLabels:
         io.kompose.service: phpldapadmin
     revisionHistoryLimit: 10
     template:
       metadata:
         labels:
           io.kompose.service: phpldapadmin
       spec:
         securityContext:
           seLinuxOptions: {}
         imagePullSecrets: []
         restartPolicy: Always
         initContainers: []
         containers:
           - image: 'osixia/phpldapadmin:0.9.0'
             imagePullPolicy: Always
             name: phpldapadmin
             volumeMounts: []
             resources:
               limits:
               requests:
             env:
               - name: PHPLDAPADMIN_HTTPS
                 value: 'false'
               - name: PHPLDAPADMIN_LDAP_HOSTS
                 value: ldap-service
             lifecycle: {}
             ports:
               - containerPort: 80
                 protocol: TCP
         volumes: []
         dnsPolicy: ClusterFirst
         dnsConfig: {}
         terminationGracePeriodSeconds: 30
     progressDeadlineSeconds: 600
     strategy:
       type: RollingUpdate
       rollingUpdate:
         maxUnavailable: 25%
         maxSurge: 25%
     replicas: 1
   ---
   apiVersion: v1
   kind: Service
   metadata:
     namespace: default
     name: phpldapadmin
     annotations:
       k8s.kuboard.cn/workload: phpldapadmin
     labels:
       io.kompose.service: phpldapadmin
   spec:
     selector:
       io.kompose.service: phpldapadmin
     type: ClusterIP
     ports:
       - port: 8080
         targetPort: 80
         protocol: TCP
         name: '8080'
         nodePort: 0
     sessionAffinity: None

   Run the following command to deploy the LDAP frontend Service:

   kubectl apply -f phpldapadmin.yaml

4. Log on to a pod in the Slurm-managed cluster based on Step 3 and run the following commands to install the LDAP client package:

   apt update
   apt install libnss-ldapd

5. After the libnss-ldapd package is installed, configure the network authentication service for the Slurm-managed cluster in the pod.

   1. Run the following commands to install the Vim package for editing scripts and files:

      apt update
      apt install vim

   2. Modify the following parameters in the /etc/ldap/ldap.conf file to configure the LDAP client:

      ...
      BASE	dc=example,dc=org # Replace the value with the distinguished name of the root node in the LDAP directory structure. 
      URI	ldap://ldap-service # Replace the value with the uniform resource identifier (URI) of your LDAP server. 
      ...

   3. Modify the following parameters in the /etc/nslcd.conf file to define the connection to the LDAP server:

      ...
      uri ldap://ldap-service # Replace the value with the URI of your LDAP server. 
      base dc=example,dc=org # Specify this parameter based on your LDAP directory structure. 
      ...
      tls_cacertfile /etc/ssl/certs/ca-certificates.crt # Specify the path to the certificate authority (CA) certificate file that is used to verify the certificate of the LDAP server. 
      ...

Share and access logs

By default, the job logs that are generated when you use the sbatch command are directly stored on the node that executes the jobs. This can be inconvenient for viewing logs. To view logs with ease, you can create a File Storage NAS (NAS) file system to store all job logs in accessible directories. This way, even if computing jobs are executed on different nodes, the logs that are generated for the jobs can be centrally collected and stored. This facilitates log management. Perform the following operations:

1. Create a NAS file system to store and share the logs of each node. For more information, see Create a file system.

2. Log on to the ACK console, and create a persistent volume (PV) and a persistent volume claim (PVC) for the NAS file system. For more information, see Mount a statically provisioned NAS volume.

3. Modify the SlurmCluster CR.

   Configure the volumeMounts and volumes parameters in the headGroupSpec and workerGroupSpec parameters to reference the created PVC and mount the PVC to the /home directory. Example:

   headGroupSpec:
   ...
   # Specify /home as the mount target. 
     volumeMounts:
     - mountPath: /home
       name: test  # The name of the volume that references the PVC. 
     volumes:
   # Add the definition of the PVC. 
     - name: test  # The value of this parameter must be the same as that of the name parameter in the volumeMounts parameter. 
       persistentVolumeClaim:
         claimName: test  # Replace the value with the name of the PVC. 
   ...
   workerGroupSpecs:
     # ... Repeat the preceding process of configuring the volume and volumeMounts parameters for each workerGroupSpec parameter.

4. Run the following command to deploy the SlurmCluster CR.

   Important

   If the SlurmCluster CR fails to be deployed, run the kubectl delete slurmcluster slurm-job-demo command to delete the CR and redeploy it.

   kubectl  apply -f slurmcluster.yaml

   After the SlurmCluster CR is deployed, worker nodes can share the NAS file system.

Perform auto scaling for the Slurm-managed cluster

The root path of the default Slurm image contains executable files and scripts such as slurm-resume.sh, slurm-suspend.sh, and slurmctld-copilot. They are used to interact with the slurmctld to scale the Slurm-managed cluster.

Auto scaling for Slurm clusters based on on-cloud nodes

• Local nodes: the physical compute nodes that are directly connected to the slurmctld.

• On-cloud nodes: the logical nodes. Logical nodes are VM instances that can be created and destroyed on demand by cloud service providers.

Auto scaling for Slurm on ACK

Procedure

1. Configure auto scaling permissions. When Helm is installed, the auto scaling permissions are automatically configured for the slurmctld pod. You can skip this step.

   The head pod requires permissions to access and update the SlurmCluster CR for auto scaling. Therefore, we recommend that you use RBAC to grant the required permissions to the head pod when you use the auto scaling feature. You can perform the following steps to configure permissions:

   First, you must create the ServiceAccount, Role, and RoleBinding required by the slurmctld pod. Example: Set the name of the Slurm-managed cluster to slurm-job-demo, and set the namespace to default. Create a file named rbac.yaml and copy the following content to the file:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: slurm-job-demo
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: slurm-job-demo
   rules:
   - apiGroups: ["kai.alibabacloud.com"]
     resources: ["slurmclusters"]
     verbs: ["get", "watch", "list", "update", "patch"]
     resourceNames: ["slurm-job-demo"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: slurm-job-demo
   subjects:
   - kind: ServiceAccount
     name: slurm-job-demo
   roleRef:
     kind: Role
     name: slurm-job-demo
     apiGroup: rbac.authorization.k8s.io

   Run the kubectl apply -f rbac.yaml command to submit the resource list.

   Grant permissions to the slurmctld pod. Run the kubectl edit slurmcluster slurm-job-demo command to modify the Slurm-managed cluster. Set the Spec.Slurmctld.Template.Spec.ServiceAccountName parameter to the service account you created.

   apiVersion: kai.alibabacloud.com/v1
   kind: SlurmCluster
   ...
   spec:
     slurmctld:
       template:
         spec:
           serviceAccountName: slurm-job-demo
   ...

   Rebuild the StatefulSet that manages the slurmctld to apply the changes you just made. Run the kubectl get sts slurm-job-demo command to find the StatefulSet that manages the slurmctld pod. Run the kubectl delete sts slurm-job-demo command to delete this StatefulSet. The Slurm Operator rebuilds the StatefulSet and applies the new configurations.

2. Configure the auto scaling file /etc/slurm/slurm.conf.

   Manage ConfigMaps by using a shared volume

   # The following parameters are required if you use on-cloud nodes.  
   # The SuspendProgram and ResumeProgram features are developed by Alibaba Cloud.
   SuspendTimeout=600
   ResumeTimeout=600
   # The interval at which the node is automatically suspended when no job runs on the node. 
   SuspendTime=600
   # Set the number of nodes that can be scaled per minute. 
   ResumeRate=1
   SuspendRate=1
   # You must set the value of the NodeName parameter in the ${cluster_name}-worker-${group_name}- format. You must specify the amount of resources for the node in this line. Otherwise, the slurmctld pod 
   # considers that the node has only one vCPU. Make sure that the resources that you specified on the on-cloud nodes are the same as those declared in the workerGroupSpec parameter. Otherwise, resources may be wasted. 
   NodeName=slurm-job-demo-worker-cpu-[0-10] Feature=cloud State=CLOUD
   # The following configurations are fixed. Keep them unchanged. 
   CommunicationParameters=NoAddrCache
   ReconfigFlags=KeepPowerSaveSettings
   SuspendProgram="/slurm-suspend.sh"
   ResumeProgram="/slurm-resume.sh"

   Manually manage ConfigMaps.

   If slurm.conf is stored in the ConfigMap of slurm-config, you can run the kubectl edit slurm-config command to add the following configurations to the ConfigMap:

   slurm.conf:
   ...
     # The following parameters are required if you use on-cloud nodes.  
     # The SuspendProgram and ResumeProgram features are developed by Alibaba Cloud.
     SuspendTimeout=600
     ResumeTimeout=600
     # The interval at which the node is automatically suspended when no job runs on the node. 
     SuspendTime=600
     # Set the number of nodes that can be scaled per minute. 
     ResumeRate=1
     SuspendRate=1
     # You must set the value of the NodeName parameter in the ${cluster_name}-worker-${group_name}- format. You must specify the amount of resources for the node in this line. Otherwise, the slurmctld pod 
     # considers that the node has only one vCPU. Make sure that the resources that you specified on the on-cloud nodes are the same as those declared in the workerGroupSpec parameter. Otherwise, resources may be wasted. 
     NodeName=slurm-job-demo-worker-cpu-[0-10] Feature=cloud State=CLOUD
     # The following configurations are fixed. Keep them unchanged. 
     CommunicationParameters=NoAddrCache
     ReconfigFlags=KeepPowerSaveSettings
     SuspendProgram="/slurm-suspend.sh"
     ResumeProgram="/slurm-resume.sh"

   Use Helm to manage ConfigMaps

   1. Add the following ConfigMap to the values.yaml file:

      slurm.conf:
      ...
        # The following parameters are required if you use on-cloud nodes.  
        # The SuspendProgram and ResumeProgram features are developed by Alibaba Cloud.
        SuspendTimeout=600
        ResumeTimeout=600
        # The interval at which the node is automatically suspended when no job runs on the node. 
        SuspendTime=600
        # Set the number of nodes that can be scaled per minute. 
        ResumeRate=1
        SuspendRate=1
        # You must set the value of the NodeName parameter in the ${cluster_name}-worker-${group_name}- format. You must specify the amount of resources for the node in this line. Otherwise, the slurmctld pod 
        # considers that the node has only one vCPU. Make sure that the resources that you specified on the on-cloud nodes are the same as those declared in the workerGroupSpec parameter. Otherwise, resources may be wasted. 
        NodeName=slurm-job-demo-worker-cpu-[0-10] Feature=cloud State=CLOUD
        # The following configurations are fixed. Keep them unchanged. 
        CommunicationParameters=NoAddrCache
        ReconfigFlags=KeepPowerSaveSettings
        SuspendProgram="/slurm-suspend.sh"
        ResumeProgram="/slurm-resume.sh"

   2. Run the helm upgrade command to update the Slurm configuration.

3. Apply the new configuration

   If the name of the Slurm-managed cluster is slurm-job-demo, run the kubectl delete sts slurm-job-demo command to apply the new configuration for the slurmctld pod.

4. Change the number of worker node replicas to 0 in the slurmcluster.yaml file to so that you can view node scaling activities in subsequent steps.

   Manual management

   Set the name of the Slurm-managed cluster to slurm-job-demo. Run the kubectl edit slurmcluster slurm-job-demo command to change the value of workerCount to 10 in the Slurm-managed cluster. This operation changes the number of worker node replicas to 0.

   Manage by using Helm

   In the values.yaml file, change the .Values.workerGroup[].workerCount parameter to 0. Then, run the helm upgrade slurm-job-demo . command to update the current Helm chart. This operation sets the number of worker node replicas to 0.

5. Submit a job by using the sbatch command.

   1. Run the following command to create a Shell script:

      cat << EOF > cloudnodedemo.sh

      Enter the following content after the command prompt:

      > #!/bin/bash
      > srun hostname
      > EOF

   2. Run the following command to check whether the content of the script is correct:

      cat cloudnodedemo.sh

      Expected output:

        #!/bin/bash
        srun hostname

      The output indicates that the script content is correct.

   3. Run the following command to submit the script to the Slurm-managed cluster for processing:

      sbatch cloudnodedemo.sh

      Expected output:

      Submitted batch job 1

      The output indicates that the job is submitted and assigned a job ID.

6. View the cluster scaling results.

   1. Run the following command to view the scaling logs of the Slurm-managed cluster:

      cat /var/log/slurm-resume.log

      Expected output:

       namespace: default cluster: slurm-demo
        resume called, args [slurm-demo-worker-cpu-0]
        slurm cluster metadata: default slurm-demo
        get SlurmCluster CR slurm-demo succeed
        hostlists: [slurm-demo-worker-cpu-0]
        resume node slurm-demo-worker-cpu-0
        resume worker -cpu-0
        resume node -cpu-0 end

      The output indicates that the Slurm-managed cluster automatically adds one compute node based on the workloads to execute the submitted job.

   2. Run the following command to view the pods in the cluster:

      kubectl get pod

      Expected output:

      NAME                                          READY   STATUS    RESTARTS        AGE
      slurm-demo-head-9hn67                         1/1     Running   0               21m
      slurm-demo-worker-cpu-0                       1/1     Running   0               43s

      The output indicates that the slurm-demo-worker-cpu-0 pod is added to the cluster. In this case, the cluster is scaled out when the job is submitted.

   3. Run the following command to view the nodes in the cluster:

      sinfo

      Expected output:

      PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
      debug*       up   infinite      10  idle~ slurm-job-demo-worker-cpu-[2-10]
      debug*       up   infinite      1   idle slurm-job-demo-worker-cpu-[0-1]

      The output indicates that slurm-demo-worker-cpu-0 node is the node that is newly started and another 10 on-cloud nodes are available for scale-out.

   4. Run the following command to view the information about the executed job:

      scontrol show job 1

      Expected output:

      JobId=1 JobName=cloudnodedemo.sh
         UserId=root(0) GroupId=root(0) MCS_label=N/A
         Priority=4294901757 Nice=0 Account=(null) QOS=(null)
         JobState=COMPLETED Reason=None Dependency=(null)
         Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
         RunTime=00:00:00 TimeLimit=UNLIMITED TimeMin=N/A
         SubmitTime=2024-05-28T11:37:36 EligibleTime=2024-05-28T11:37:36
         AccrueTime=2024-05-28T11:37:36
         StartTime=2024-05-28T11:37:36 EndTime=2024-05-28T11:37:36 Deadline=N/A
         SuspendTime=None SecsPreSuspend=0 LastSchedEval=2024-05-28T11:37:36 Scheduler=Main
         Partition=debug AllocNode:Sid=slurm-job-demo:93
         ReqNodeList=(null) ExcNodeList=(null)
         NodeList=slurm-job-demo-worker-cpu-0
         BatchHost=slurm-job-demo-worker-cpu-0
         NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
         ReqTRES=cpu=1,mem=1M,node=1,billing=1
         AllocTRES=cpu=1,mem=1M,node=1,billing=1
         Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
         MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
         Features=(null) DelayBoot=00:00:00
         OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
         Command=//cloudnodedemo.sh
         WorkDir=/
         StdErr=//slurm-1.out
         StdIn=/dev/null
         StdOut=//slurm-1.out
         Power=

      In the output, NodeList=slurm-demo-worker-cpu-0 indicates that the job is executed on the newly added node.

   5. After a period of time, run the following command to view the node scale-in results:

      sinfo

      Expected output:

      PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
      debug*       up   infinite     11  idle~ slurm-demo-worker-cpu-[0-10]

      The output indicates that the number of nodes available for scale-out becomes 11. This indicates that the automatic scale-in is complete.