Container Service for Kubernetes:GPU FAQ

Why are the GPU ECC configurations in my cluster inconsistent?

Error-Correcting Code (ECC) mode improves GPU stability and reliability by detecting and correcting GPU memory errors. However, enabling this mode slightly reduces the available GPU memory.

ECC mode recommendations:

• Disable ECC: Suitable for cost-sensitive scenarios and low-latency inference, such as online real-time inference.

• Enable ECC: Suitable for applications that require high data consistency and integrity, such as database servers, financial systems, scientific computing, and high-performance computing (HPC).

For these reasons, we do not initialize ECC mode uniformly. This can result in inconsistent ECC configurations across the GPU nodes in a cluster.

Does Container Service for Kubernetes support vGPU-accelerated instances?

vGPU-accelerated instances require a GRID License from NVIDIA to function correctly. To use vGPU-accelerated instances, you must purchase a GRID License from NVIDIA and build your own license server.

Because Alibaba Cloud does not provide license servers, vGPU-accelerated instances cannot be used directly, even in a vGPU-accelerated cluster. Therefore, the Container Service for Kubernetes console no longer supports selecting vGPU-accelerated instances as cluster nodes.

Unsupported vGPU-accelerated instances include ECS instances with prefixes such as ecs.vgn5i, ecs.vgn6i, ecs.vgn7i, and ecs.sgn7i. To use these instances, you must purchase a GRID License from NVIDIA and build your own license server.

If you have already set up your license server, you can follow these steps to add vGPU-accelerated instances to an ACK cluster.

Add vGPU-accelerated instances to an ACK cluster

1. Go to Privilege Quota and request the custom OS image feature.

2. Create a custom OS image based on CentOS 7.x or Alibaba Cloud Linux 2. The image must have the NVIDIA GRID driver installed and the GRID License correctly configured. For more information, see Create a custom image from an instance and Install a GRID driver on a vGPU-accelerated instance (Linux).

3. Create a node pool. For more information, see Create and manage a node pool.

4. Add the vGPU-accelerated instances to the node pool created in Step 3. For more information, see Add existing nodes.

Next step: Update the NVIDIA driver license for vGPU-accelerated instances in an ACK cluster

For information about how to update the NVIDIA driver license for vGPU-accelerated instances in an ACK cluster, see Update the NVIDIA driver license for vGPU-accelerated (vGPU) instances in an ACK cluster.

How to manually upgrade the kernel on a GPU node in an existing cluster

1. Set the GPU node to unschedulable. This example uses the node cn-beijing.i-2ze19qyi8votgjz12345.

   kubectl cordon cn-beijing.i-2ze19qyi8votgjz12345
   
   node/cn-beijing.i-2ze19qyi8votgjz12345 already cordoned

2. Drain the GPU node on which you want to upgrade the driver.

   kubectl drain cn-beijing.i-2ze19qyi8votgjz12345 --grace-period=120 --ignore-daemonsets=true
   
   node/cn-beijing.i-2ze19qyi8votgjz12345 cordoned
   WARNING: Ignoring DaemonSet-managed pods: flexvolume-9scb4, kube-flannel-ds-r2qmh, kube-proxy-worker-l62sf, logtail-ds-f9vbg
   pod/nginx-ingress-controller-78d847fb96-5fkkw evicted

3. Uninstall the current NVIDIA driver.

   Note

   This step uninstalls driver version 384.111. If your driver version is different, download the corresponding driver installation package from the NVIDIA website and replace the version number accordingly.

   1. Log on to the GPU node and run nvidia-smi to check the driver version.

      sudo nvidia-smi -a | grep 'Driver Version'
      Driver Version                      : 384.111

   2. Download the NVIDIA driver installation package.

      sudo cd /tmp/
      sudo curl -O https://cn.download.nvidia.cn/tesla/384.111/NVIDIA-Linux-x86_64-384.111.run

      Note

      You must use the installation package to uninstall the NVIDIA driver.

   3. Uninstall the current NVIDIA driver.

      sudo chmod u+x NVIDIA-Linux-x86_64-384.111.run
      sudo sh ./NVIDIA-Linux-x86_64-384.111.run --uninstall -a -s -q

4. Upgrade the kernel.

5. Restart the GPU instance.

   sudo reboot

6. Log on to the GPU node again and install the corresponding kernel-devel package.

   sudo yum install -y kernel-devel-$(uname -r)

7. Go to the NVIDIA website to download and install the required NVIDIA driver. This example uses version 410.79.

   sudo cd /tmp/
   sudo curl -O https://cn.download.nvidia.cn/tesla/410.79/NVIDIA-Linux-x86_64-410.79.run
   sudo chmod u+x NVIDIA-Linux-x86_64-410.79.run
   sudo sh ./NVIDIA-Linux-x86_64-410.79.run -a -s -q
   
   # warm up GPU
   sudo nvidia-smi -pm 1 || true
   sudo nvidia-smi -acp 0 || true
   sudo nvidia-smi --auto-boost-default=0 || true
   sudo nvidia-smi --auto-boost-permission=0 || true
   sudo nvidia-modprobe -u -c=0 -m || true

8. Check the /etc/rc.d/rc.local file to confirm that it contains the following configuration. If not, add it manually.

   sudo nvidia-smi -pm 1 || true
   sudo nvidia-smi -acp 0 || true
   sudo nvidia-smi --auto-boost-default=0 || true
   sudo nvidia-smi --auto-boost-permission=0 || true
   sudo nvidia-modprobe -u -c=0 -m || true

9. Restart kubelet and Docker.

   sudo service kubelet stop
   sudo service docker restart
   sudo service kubelet start

10. Set the GPU node to schedulable again.

    kubectl uncordon cn-beijing.i-2ze19qyi8votgjz12345
    
    node/cn-beijing.i-2ze19qyi8votgjz12345 already uncordoned

11. Verify the driver version in the device plugin pod on the GPU node.

    kubectl exec -n kube-system -t nvidia-device-plugin-cn-beijing.i-2ze19qyi8votgjz12345 nvidia-smi
    Thu Jan 17 00:33:27 2019
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 410.79       Driver Version: 410.79       CUDA Version: N/A      |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |===============================+======================+======================|
    |   0  Tesla P100-PCIE...  On   | 00000000:00:09.0 Off |                    0 |
    | N/A   27C    P0    28W / 250W |      0MiB / 16280MiB |      0%      Default |
    +-------------------------------+----------------------+----------------------+
    
    +-----------------------------------------------------------------------------+
    | Processes:                                                       GPU Memory |
    |  GPU       PID   Type   Process name                             Usage      |
    |=============================================================================|
    |  No running processes found                                                 |
    +-----------------------------------------------------------------------------+

    Note

    If you run the docker ps command and find that no containers are started on the GPU node, see Fix container startup issues on GPU nodes.

Fix container startup issues on GPU nodes

sudo service kubelet stop
Redirecting to /bin/systemctl stop kubelet.service
sudo service docker stop
Redirecting to /bin/systemctl stop docker.service
sudo service docker start
Redirecting to /bin/systemctl start docker.service
sudo service kubelet start
Redirecting to /bin/systemctl start kubelet.service

sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

sudo docker info | grep -i cgroup
Cgroup Driver: cgroupfs

Follow these steps to fix the issue.

1. Back up the /etc/docker/daemon.json file. Then, run the following command to update /etc/docker/daemon.json.

   sudo cat >/etc/docker/daemon.json <<-EOF
   {
       "default-runtime": "nvidia",
       "runtimes": {
           "nvidia": {
               "path": "/usr/bin/nvidia-container-runtime",
               "runtimeArgs": []
           }
       },
       "exec-opts": ["native.cgroupdriver=systemd"],
       "log-driver": "json-file",
       "log-opts": {
           "max-size": "100m",
           "max-file": "10"
       },
       "oom-score-adjust": -1000,
       "storage-driver": "overlay2",
       "storage-opts": ["overlay2.override_kernel_check=true"],
       "live-restore": true
   }
   EOF

2. Restart Docker and kubelet.

   sudo service kubelet stop
   Redirecting to /bin/systemctl stop kubelet.service
   sudo service docker restart
   Redirecting to /bin/systemctl restart docker.service
   sudo service kubelet start
   Redirecting to /bin/systemctl start kubelet.service

3. Confirm that the Cgroup driver for Docker is systemd.

   sudo docker info | grep -i cgroup
   Cgroup Driver: systemd

What do I do if adding an ECS Bare Metal Instance node fails?

ECS Bare Metal Instances (ecs.ebmgn7) support multi-instance GPU (MIG). To prevent existing MIG settings from affecting node deployment, the system resets any existing MIG settings each time a node of this type is added to a cluster. This reset time is variable. If the reset takes too long, the script for adding the node may time out, causing the operation to fail.

If adding a node of this instance family fails, you can run the following command on the node's host:

sudo cat /var/log/ack-deploy.log

command timeout: timeout 300 nvidia-smi --gpu-reset

If this error is present, the failure was caused by the MIG reset. You can add the node again. For more information, see Add existing nodes.

What do I do if "Failed to initialize NVML: Unknown Error" occurs when running a GPU container on Alibaba Cloud Linux 3?

sudo nvidia-smi

Failed to initialize NVML: Unknown Error

What do I do if a GPU card becomes unavailable due to XID 119 or XID 120 errors?

How to manually isolate a faulty GPU in a cluster?

When you use shared GPU scheduling, a faulty GPU in the cluster can cause a Job to fail. If a Job restarts, it may be scheduled to the same faulty GPU and fail again. To prevent repeated failures, you can manually mark the faulty GPU. The scheduler will no longer use the marked GPU, which provides fault isolation.

To mark a faulty GPU, you can submit a special ConfigMap to the cluster as shown in the following example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: <node-name>-device-status   # Replace <node-name> with the actual node name.
  namespace: kube-system
data:
  devices: |
    - deviceId: 0          # Run nvidia-smi to get the GPU ordinal number.
      deviceType: gpu
      healthy: false

The ConfigMap must be in the kube-system namespace. The name of the ConfigMap must be the name of the node that hosts the faulty GPU, with the -device-status suffix. In the data field, deviceId is the GPU ordinal number that is obtained from the nvidia-smi command, deviceType is gpu, and healthy is false. After you submit the configuration, the scheduler isolates the corresponding GPU.

Resolve the "Failed to initialize NVML: Unknown Error" message in GPU containers

sudo nvidia-smi

Failed to initialize NVML: Unknown Error

#!/bin/bash
set -e

export CONFIG=/etc/nvidia-container-runtime/config.toml
export CONTAINER_ROOT_PATH="/run/containerd/io.containerd.runtime.v2.task/k8s.io"

if [ -f $CONFIG ];then
    # Change the log level in the nvidia-container-runtime configuration from "info" to "error".
	sed -i 's@^log-level = "info"@log-level = "error"@g' $CONFIG
    # Clear the content of the container's log.json file.
	find $CONTAINER_ROOT_PATH -mindepth 2 -maxdepth 2 -name log.json -type f -exec sh -c 'echo "" > "{}"' \;
fi