Container Service for Kubernetes:GPU Device Plugin-related operations

Kubernetes 1.32 and later

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, find the one you want to manage and click its name. In the left-side navigation pane, click Add-ons.

3. On the Add-ons page, find the ack-nvidia-device-plugin card and click Upgrade.

4. In the dialog box that appears, click OK.

Kubernetes 1.20 to 1.31

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, click the name of the one you want to change. In the left-side navigation pane, choose Nodes > Nodes.

3. Select the GPU-accelerated nodes requiring batch maintenance. In the lower part of the node list, click Batch Operations. In the Batch Operations dialog box, select Execute Shell Command and click Confirm.

   Important

   We recommend that you first upgrade GPU Device Plugin on a small number of GPU-accelerated nodes. After the upgrade is completed, verify that NVIDIA Device Plugin runs as normal on these nodes. Then, upgrade the remaining GPU-accelerated nodes on a larger scale.

4. In the CloudOps Orchestration Service (OOS) console that appears, select Execution Mode as Failure Pause, and then click Next: Parameter Settings.

5. On the parameter settings page, select Run Shell Script and paste the provided sample script.

   Note

   You must set the RUN_PKG_VERSION parameter in the following script to the minor version run by your Kubernetes cluster, such as 1.30. Do not set the value to a patch version, such as 1.30.1. If you specify a patch version, script errors may occur.

   #!/bin/bash
   set -e
   
   RUN_PKG_VERSION=1.30
   
   function update_device_plugin() {
   	base_dir=/tmp/update_device_plugin
   	rm -rf $base_dir
   	mkdir -p $base_dir
   	cd $base_dir
   	region_id=$(curl -ssL 100.100.100.200/latest/meta-data/region-id  2> /dev/null || echo "")
   	if [[ $region_id == "" ]];then
   		echo "Error: failed to get region id,region id is null"
   		exit 1
   	fi
   	PKG_URL=https://aliacs-k8s-${region_id}.oss-${region_id}.aliyuncs.com/public/pkg/run/run-${RUN_PKG_VERSION}.tar.gz
   	curl -sSL --retry 3 --retry-delay 2 -o run.tar.gz $PKG_URL
   	tar -xf run.tar.gz
   
   	local dir=pkg/run/$RUN_PKG_VERSION/module
   	sed -i "s@registry.cn-hangzhou.aliyuncs.com/acs@registry-${region_id}-vpc.ack.aliyuncs.com/acs@g" $dir/nvidia-device-plugin.yml
   	mkdir -p /etc/kubernetes/device-plugin-backup
   	mkdir -p /etc/kubernetes/manifests
   	mv  /etc/kubernetes/manifests/nvidia-device-plugin.yml /etc/kubernetes/device-plugin-backup/nvidia-device-plugin.yml.$(date +%s)
   	sleep 5
   	cp -a $dir/nvidia-device-plugin.yml /etc/kubernetes/manifests
   	echo "succeeded to update device plugin"
   }
   
   if [ -f /etc/kubernetes/manifests/nvidia-device-plugin.yml ];then
   	update_device_plugin
   else
   	echo "skip to update device plugin"
   fi

6. Click Next: Confirm. After verifying the information, click Create.

   You will be redirected to the Task Execution Management page to monitor the task status. If the Execution Output displays succeeded to update device plugin, the update was successful.

7. Execute the following command to verify the Device Plugin on the GPU-accelerated node is functioning correctly.

   kubectl get nodes <NODE_NAME> -o jsonpath='{.metadata.name} ==> nvidia.com/gpu: {.status.allocatable.nvidia\.com/gpu}'

   Expected output:

   cn-hangzhou.172.16.XXX.XX ==> nvidia.com/gpu: 1

   If the GPU-accelerated node reports the extended resource nvidia.com/gpu as not 0, the Device Plugin is operational.

Kubernetes 1.32 and later

1. Run the following command to query the pod that runs NVIDIA Device Plugin on a node:

   kubectl get pod -n kube-system -l component=nvidia-device-plugin -o wide | grep <NODE>

2. Run the following command to restart the pod that runs NVIDIA Device Plugin:

   $ kubectl delete po <DEVICE_PLUGIN_POD> -n kube-system 

Kubernetes 1.20 to 1.31

1. On the Clusters page, click the name of the one you want to change. In the left-side navigation pane, choose Nodes > Nodes.

2. Select the GPU-accelerated nodes that you want to manage and click Batch Operations. In the Batch Operations dialog box, select Run Shell Scripts and click OK.

   Important

   We recommend that you first upgrade GPU Device Plugin on a small number of GPU-accelerated nodes. After the upgrade is completed, verify that NVIDIA Device Plugin runs as normal on these nodes. Then, upgrade the remaining GPU-accelerated nodes on a larger scale.

3. In the OOS console, select Suspend upon Failure for Execution Mode and click Next Step: Parameter Settings.

4. In the Parameter Settings step, click Run Shell Script and copy the following sample script to the code editor:

   #!/bin/bash
   set -e
   
   if [ -f /etc/kubernetes/manifests/nvidia-device-plugin.yml ];then
   	cp -a /etc/kubernetes/manifests/nvidia-device-plugin.yml /etc/kubernetes
   	rm -rf /etc/kubernetes/manifests/nvidia-device-plugin.yml
   	sleep 5
   	mv /etc/kubernetes/nvidia-device-plugin.yml /etc/kubernetes/manifests
   	echo "the nvidia device is restarted"
   else
   	echo "no need to restart nvidia device plugin"
   fi

5. Click Next Step: OK. Confirm the information and click Create. You are redirected to the Task Execution Management page, on which you can view the task status.

6. Run the following command to check whether NVIDIA Device Plugin runs as normal on the specified GPU-accelerated node:

   kubectl get nodes <NODE_NAME> -o jsonpath='{.metadata.name} ==> nvidia.com/gpu: {.status.allocatable.nvidia\.com/gpu}'

   Expected output:

   cn-hangzhou.172.16.XXX.XX ==> nvidia.com/gpu: 1

   If the value of nvidia.com/gpu in the output is not 0, it indicates NVIDIA Device Plugin runs as normal.

Log on to the ACK console. In the left-side navigation pane, click Clusters.

Kubernetes 1.32 and later

1. Run the following command to modify the environment variables of the static pod:

   kubectl edit pod -n kube-system ack-nvidia-device-plugin-lr7dr -oyaml 

2. Add the following environment variable: CHECKPOINT_DEVICE_ID_STRATEGY.

       env:
         - name: CHECKPOINT_DEVICE_ID_STRATEGY
           value: index

3. Refer to Restart NVIDIA Device Plugin to restart NVIDIA Device Plugin to make the modifications take effect.

Kubernetes 1.20 to 1.31

1. Verify the Device Plugin version in the /etc/kubernetes/manifests/nvidia-device-plugin.yml file on the target node by checking the image tag. If the version is 0.9.3 or higher, no change is needed. Otherwise, update to the latest version v0.9.3-0dd4d5f5-aliyun.

2. Modify the static pod's environment variables in the /etc/kubernetes/manifests/nvidia-device-plugin.yml file. Add the environment variable CHECKPOINT_DEVICE_ID_STRATEGY as shown in the example code.

       env:
         - name: CHECKPOINT_DEVICE_ID_STRATEGY
           value: index

3. Refer to Restart NVIDIA Device Plugin to restart NVIDIA Device Plugin to make the modifications take effect.