How to manually update the NVIDIA driver of a node - Container Service for Kubernetes

Prerequisites

Ensure that you have:

kubectl configured with access to the cluster
SSH access to the target GPU-accelerated node
The new NVIDIA driver .run file downloaded from the NVIDIA official site to the node
(Conditional) The matching NVIDIA Fabric Manager .rpm package, if required — see Step 3, substep 5

Step 1: Cordon and drain the node

Mark the GPU-accelerated node as unschedulable:
```
kubectl cordon <NODE_NAME>
```
Replace <NODE_NAME> with the target node name. Expected output:
```
node/<NODE_NAME> cordoned
```
Evict workload pods from the node:
```
kubectl drain <NODE_NAME> --grace-period=120 --ignore-daemonsets=true
```
--grace-period=120 gives pods up to 120 seconds for graceful shutdown. --ignore-daemonsets=true skips DaemonSet-managed pods, which are handled in the next step. Expected output:
```
There are pending nodes to be drained:
 <NODE_NAME>
```

Step 2: Uninstall the current NVIDIA driver

Log in to the node and stop kubelet and containerd. This terminates DaemonSet pods holding GPU references that kubectl drain cannot evict.
```
sudo systemctl stop kubelet containerd
```
Check for processes still using GPU devices:
```
sudo fuser -v /dev/nvidia*
```
No output means no processes hold GPU references — proceed to the next substep. If output appears, terminate each listed process. For example:
```
                     USER        PID ACCESS COMMAND
/dev/nvidia0:        root       3781 F.... dcgm-exporter
/dev/nvidiactl:      root       3781 F...m dcgm-exporter
```
Terminate the process:
```
sudo kill 3781
```
Run sudo fuser -v /dev/nvidia* again until no processes remain.
Uninstall the current NVIDIA driver:
```
sudo nvidia-uninstall
```
(Optional) Uninstall NVIDIA Fabric Manager if installed. Check whether it is installed:
```
sudo rpm -qa | grep ^nvidia-fabric-manager
```
No output means NVIDIA Fabric Manager is not installed — skip this step. Otherwise, uninstall it:
```
yum remove nvidia-fabric-manager
```

Step 3: Install the new NVIDIA driver on the node

Install the new driver. Example with NVIDIA-Linux-x86_64-510.108.03.run:
```
sudo bash NVIDIA-Linux-x86_64-510.108.03.run -a -s -q
```

Verify the installation:

sudo nvidia-smi

The output shows the new driver version. Example for 510.108.03:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.108.03   Driver Version: 510.108.03   CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  Off  | 00000000:00:07.0 Off |                    0 |
| N/A   35C    P0    40W / 300W |      0MiB / 32768MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Apply the required post-install settings:

sudo nvidia-smi -pm 1 || true                            # Enable Persistence mode
sudo nvidia-smi -acp 0 || true                           # Set permission to UNRESTRICTED
sudo nvidia-smi --auto-boost-default=0 || true           # Disable auto boost mode
sudo nvidia-smi --auto-boost-permission=0 || true        # Allow non-admin users to control auto boost mode
sudo nvidia-modprobe -u -c=0 -m || true                  # Load the NVIDIA unified memory kernel module and create device files

(Optional) To load the NVIDIA driver on boot, ensure /etc/rc.d/rc.local contains:

sudo nvidia-smi -pm 1 || true
sudo nvidia-smi -acp 0 || true
sudo nvidia-smi --auto-boost-default=0 || true
sudo nvidia-smi --auto-boost-permission=0 || true
sudo nvidia-modprobe -u -c=0 -m || true

Check whether NVIDIA Fabric Manager is required — nodes with NVSwitch-based multi-GPU interconnects need it.
```
sudo lspci | grep -i 'Bridge:.*NVIDIA'
```
No output means NVIDIA Fabric Manager is not required. If output appears, download the matching NVIDIA Fabric Manager package from the NVIDIA YUM repository. The package version must match the driver version. Install and start NVIDIA Fabric Manager:
```
# Install NVIDIA Fabric Manager
sudo yum localinstall nvidia-fabric-manager-510.108.03-1.x86_64.rpm

# Enable NVIDIA Fabric Manager on boot
systemctl enable nvidia-fabricmanager.service

# Start NVIDIA Fabric Manager
systemctl start nvidia-fabricmanager.service
```

Restart kubelet and containerd:

sudo systemctl restart containerd kubelet

Step 4: Return the node to the cluster

Mark the node as schedulable:

sudo kubectl uncordon <NODE_NAME>

Replace <NODE_NAME> with your node name. Kubernetes resumes scheduling pods on the node.