Elastic GPU Service:Manually install Tesla driver on a GPU instance (Linux)

(Optional) Enable Persistence Mode by using the NVIDIA Persistence Daemon.

After the Tesla driver is installed, Persistence Mode is disabled (off) by default. The Tesla driver performs more stably when Persistence Mode is enabled. To ensure service stability, we recommend that you enable Persistence Mode by using the NVIDIA Persistence Daemon. For more information, see Persistence Daemon.

1. Run the following command to start the NVIDIA Persistence Daemon.

   sudo nvidia-persistenced --user username 
   # Replace username with your username.

2. Run the following command to check the status of Persistence Mode.

   nvidia-smi

   The returned message indicates that Persistence-M is in the enabled (on) state.

   [ecs-usexxx2q9sgg1tZ ~]$ sudo nvidia-persistenced --user ecs-user
   [ecs-usexxx2q9sgg1tZ ~]$ nvidia-smi
   Tue Sep 10 14:02:16 2024
   +-------------------------------+----------------------+----------------------+
   | NVIDIA-SMI 470.161.03   Driver Version: 470.161.03   CUDA Version: 11.4     |
   |-------------------------------+----------------------+----------------------+
   | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
   | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
   |                               |                      |               MIG M. |
   |===============================+======================+======================|
   |   0  NVIDIA A10          On   | 00000000:00:07.0 Off |                    0 |
   |  0%   33C    P8     8W / 150W |      0MiB / 22731MiB |      0%      Default |
   |                               |                      |                  N/A |
   +-------------------------------+----------------------+----------------------+
   +-----------------------------------------------------------------------------+
   | Processes:                                                                  |
   |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
   |        ID   ID                                                   Usage      |
   |=============================================================================|
   |  No running processes found                                                 |
   +-----------------------------------------------------------------------------+

(Optional) Configure Persistence Mode to enable on system reboot.

If the system reboots, the enabled (on) state of Persistence Mode is lost. You can perform the following operations to re-enable Persistence Mode.

The Tesla driver installation package installs NVIDIA's installation scripts, such as example scripts and installer scripts, to /usr/share/doc/NVIDIA_GLX-1.0/samples/nvidia-persistenced-init.tar.bz2.

1. Run the following commands to decompress and install the NVIDIA scripts.

   cd  /usr/share/doc/NVIDIA_GLX-1.0/samples/
   sudo tar xf nvidia-persistenced-init.tar.bz2
   cd  nvidia-persistenced-init
   sudo sh install.sh

2. Run the following command to check if the NVIDIA Persistence Daemon is running.

   sudo systemctl status nvidia-persistenced

   If the output resembles the following, the NVIDIA Persistence Daemon is running.

   [ecs-user@xxx nvidia-persistenced-init]$ sudo systemctl status nvidia-persistenced
   ● nvidia-persistenced.service - NVIDIA Persistence Daemon
      Loaded: loaded (/usr/lib/systemd/system/nvidia-persistenced.service; enabled; vendor preset: disabled)
      Active: active (running) since Tue 2024-09-10 14:13:20 CST; 40s ago
     Process: 13882 ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced (code=exited, status=0/SUCCESS)
    Main PID: 13883 (nvidia-persiste)
       Tasks: 1 (limit: 383833)
      Memory: 196.0K
      CGroup: /system.slice/nvidia-persistenced.service
              └─13883 /usr/bin/nvidia-persistenced --user nvidia-persistenced
   Sep 10 14:13:19 iZbp13orbqqx6m2q9sgg1tZ systemd[1]: Starting NVIDIA Persistence Daemon...
   Sep 10 14:13:19 iZbp13orbqqx6m2q9sgg1tZ nvidia-persistenced[13883]: Started (13883)
   Sep 10 14:13:20 iZbp13orbqqx6m2q9sgg1tZ systemd[1]: Started NVIDIA Persistence Daemon.

   Note

   You can adapt the NVIDIA Persistence Daemon installation script for your operating system to ensure it works correctly.

3. Run the following command to verify that the Persistence Mode is set to on.

   nvidia-smi

4. (Optional) Run the following commands to stop the NVIDIA Persistence Daemon.

   You can disable the NVIDIA Persistence Daemon if it is no longer needed.

   sudo systemctl stop nvidia-persistenced
   sudo systemctl disable nvidia-persistenced

(Optional) Enable Persistence Mode by using the NVIDIA Persistence Daemon.

After the Tesla driver is installed, Persistence Mode is disabled (off) by default. The Tesla driver performs more stably when Persistence Mode is enabled. To ensure service stability, we recommend that you enable Persistence Mode by using the NVIDIA Persistence Daemon. For more information, see Persistence Daemon.

1. Run the following command to start the NVIDIA Persistence Daemon.

   sudo nvidia-persistenced --user username 
   # Replace username with your username.

2. Run the following command to check the status of Persistence Mode.

   nvidia-smi

   The returned message indicates that Persistence-M is in the enabled (on) state.

   [ecs-usexxx2q9sgg1tZ ~]$ sudo nvidia-persistenced --user ecs-user
   [ecs-usexxx2q9sgg1tZ ~]$ nvidia-smi
   Tue Sep 10 14:02:16 2024
   +-------------------------------+----------------------+----------------------+
   | NVIDIA-SMI 470.161.03   Driver Version: 470.161.03   CUDA Version: 11.4     |
   |-------------------------------+----------------------+----------------------+
   | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
   | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
   |                               |                      |               MIG M. |
   |===============================+======================+======================|
   |   0  NVIDIA A10          On   | 00000000:00:07.0 Off |                    0 |
   |  0%   33C    P8     8W / 150W |      0MiB / 22731MiB |      0%      Default |
   |                               |                      |                  N/A |
   +-------------------------------+----------------------+----------------------+
   +-----------------------------------------------------------------------------+
   | Processes:                                                                  |
   |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
   |        ID   ID                                                   Usage      |
   |=============================================================================|
   |  No running processes found                                                 |
   +-----------------------------------------------------------------------------+

(Optional) Configure Persistence Mode to enable on system reboot.

If the system reboots, the enabled (on) state of Persistence Mode is lost. You can perform the following operations to re-enable Persistence Mode.

The Tesla driver installation package installs NVIDIA's installation scripts, such as example scripts and installer scripts, to /usr/share/doc/NVIDIA_GLX-1.0/samples/nvidia-persistenced-init.tar.bz2.

1. Run the following commands to decompress and install the NVIDIA scripts.

   cd  /usr/share/doc/NVIDIA_GLX-1.0/samples/
   sudo tar xf nvidia-persistenced-init.tar.bz2
   cd  nvidia-persistenced-init
   sudo sh install.sh

2. Run the following command to check if the NVIDIA Persistence Daemon is running.

   sudo systemctl status nvidia-persistenced

   If the output resembles the following, the NVIDIA Persistence Daemon is running.

   [ecs-user@xxx nvidia-persistenced-init]$ sudo systemctl status nvidia-persistenced
   ● nvidia-persistenced.service - NVIDIA Persistence Daemon
      Loaded: loaded (/usr/lib/systemd/system/nvidia-persistenced.service; enabled; vendor preset: disabled)
      Active: active (running) since Tue 2024-09-10 14:13:20 CST; 40s ago
     Process: 13882 ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced (code=exited, status=0/SUCCESS)
    Main PID: 13883 (nvidia-persiste)
       Tasks: 1 (limit: 383833)
      Memory: 196.0K
      CGroup: /system.slice/nvidia-persistenced.service
              └─13883 /usr/bin/nvidia-persistenced --user nvidia-persistenced
   Sep 10 14:13:19 iZbp13orbqqx6m2q9sgg1tZ systemd[1]: Starting NVIDIA Persistence Daemon...
   Sep 10 14:13:19 iZbp13orbqqx6m2q9sgg1tZ nvidia-persistenced[13883]: Started (13883)
   Sep 10 14:13:20 iZbp13orbqqx6m2q9sgg1tZ systemd[1]: Started NVIDIA Persistence Daemon.

   Note

   You can adapt the NVIDIA Persistence Daemon installation script for your operating system to ensure it works correctly.

3. Run the following command to verify that the Persistence Mode is set to on.

   nvidia-smi

4. (Optional) Run the following commands to stop the NVIDIA Persistence Daemon.

   You can disable the NVIDIA Persistence Daemon if it is no longer needed.

   sudo systemctl stop nvidia-persistenced
   sudo systemctl disable nvidia-persistenced

(Conditionally required) If your instance belongs to the ebmgn8v, ebmgn7, or ebmgn7e instance family, install the nvidia-fabricmanager service that matches your driver version.

1. Install the nvidia-fabricmanager service.

   You can install the nvidia-fabricmanager service from source code or an installation package. The following command examples are for the Ubuntu 16.04, Ubuntu 18.04, Ubuntu 20.04, Ubuntu 22.04, or Ubuntu 24.04 operating system. In the commands, replace driver_version with the version of the driver that you downloaded in Step 1: Download the NVIDIA Tesla driver.

   Important

   • On Ubuntu 22.04, the nvidia-fabricmanager service requires a Tesla driver version later than 515.48.07. The following example for Ubuntu 22.04 uses driver version 535.154.05.

   • On Ubuntu 24.04, the nvidia-fabricmanager service requires a Tesla driver version later than 550.90.07. The following example for Ubuntu 24.04 uses driver version 570.133.20.

   • Source code

     Ubuntu 16.04, Ubuntu 18.04, or Ubuntu 20.04

     driver_version=460.91.03
     driver_version_main=$(echo $driver_version | awk -F '.' '{print $1}')
     distribution=$(. /etc/os-release;echo $ID$VERSION_ID | sed -e 's/\.//g')
     sudo wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/cuda-$distribution.pin
     sudo mv cuda-$distribution.pin /etc/apt/preferences.d/cuda-repository-pin-600
     sudo wget -qO - https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/3bf863cc.pub | sudo apt-key add -
     sudo echo "deb https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64 /" | tee /etc/apt/sources.list.d/cuda.list
     sudo apt-get update
     sudo apt-get -y install nvidia-fabricmanager-${driver_version_main}=${driver_version}-*

     Ubuntu 22.04

     driver_version=535.154.05
     driver_version_main=$(echo $driver_version | awk -F '.' '{print $1}')
     distribution=$(. /etc/os-release;echo $ID$VERSION_ID | sed -e 's/\.//g')
     sudo wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/cuda-$distribution.pin
     sudo mv cuda-$distribution.pin /etc/apt/preferences.d/cuda-repository-pin-600
     sudo wget -qO - https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/3bf863cc.pub | sudo apt-key add -
     sudo echo "deb https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64 /" | tee /etc/apt/sources.list.d/cuda.list
     sudo apt-get update
     sudo apt-get -y install nvidia-fabricmanager-${driver_version_main}=${driver_version}-*

     Ubuntu 24.04

     driver_version=570.133.20
     driver_version_main=$(echo $driver_version | awk -F '.' '{print $1}')
     distribution=$(. /etc/os-release;echo $ID$VERSION_ID | sed -e 's/\.//g')
     sudo wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/cuda-$distribution.pin
     sudo mv cuda-$distribution.pin /etc/apt/preferences.d/cuda-repository-pin-600
     sudo wget -qO - https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/3bf863cc.pub | sudo apt-key add -
     sudo echo "deb https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64 /" | tee /etc/apt/sources.list.d/cuda.list
     sudo apt-get update
     sudo apt-get -y install nvidia-fabricmanager-${driver_version_main}=${driver_version}-*

   • Installation package

     • Ubuntu 16.04

       driver_version=460.91.03
       driver_version_main=$(echo $driver_version | awk -F '.' '{print $1}')
       sudo wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/nvidia-fabricmanager-${driver_version_main}_${driver_version}-1_amd64.deb
       sudo dpkg -i nvidia-fabricmanager-${driver_version_main}_${driver_version}-1_amd64.deb

     • Ubuntu 18.04

       driver_version=460.91.03
       driver_version_main=$(echo $driver_version | awk -F '.' '{print $1}')
       sudo wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/nvidia-fabricmanager-${driver_version_main}_${driver_version}-1_amd64.deb
       sudo dpkg -i nvidia-fabricmanager-${driver_version_main}_${driver_version}-1_amd64.deb

     • Ubuntu 20.04

       driver_version=460.91.03
       driver_version_main=$(echo $driver_version | awk -F '.' '{print $1}')
       sudo wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/nvidia-fabricmanager-${driver_version_main}_${driver_version}-1_amd64.deb
       sudo dpkg -i nvidia-fabricmanager-${driver_version_main}_${driver_version}-1_amd64.deb

     • Ubuntu 22.04

       driver_version=535.154.05 
       driver_version_main=$(echo $driver_version | awk -F '.' '{print $1}')
       sudo wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/nvidia-fabricmanager-${driver_version_main}_${driver_version}-1_amd64.deb
       sudo dpkg -i nvidia-fabricmanager-${driver_version_main}_${driver_version}-1_amd64.deb

     • Ubuntu 24.04

       driver_version=570.133.20 
       driver_version_main=$(echo $driver_version | awk -F '.' '{print $1}')
       sudo wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/nvidia-fabricmanager-${driver_version_main}_${driver_version}-1_amd64.deb
       sudo dpkg -i nvidia-fabricmanager-${driver_version_main}_${driver_version}-1_amd64.deb

2. Run the following commands to start the nvidia-fabricmanager service.

   sudo systemctl enable nvidia-fabricmanager
   sudo systemctl start nvidia-fabricmanager

3. Run the following command to check the status of the nvidia-fabricmanager service.

   systemctl status nvidia-fabricmanager

   If the following output is returned, the nvidia-fabricmanager service is running.

   nvidia-fabricmanager.service - NVIDIA fabric manager service
      Loaded: loaded (/lib/systemd/system/nvidia-fabricmanager.service; enabled; vendor preset: enabled)
      Active: active (running) since Mon 2021-09-13 19:14:45 CST; 1 weeks 1 days ago
     Process: 1928 ExecStart=/usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg (code=exited, status=0/SUCCESS)
    Main PID: 2140 (nv-fabricmanage)
       Tasks: 18 (limit: 19660)
      CGroup: /system.slice/nvidia-fabricmanager.service
              └─2140 /usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg
   Sep 13 19:14:26 xxx systemd[1]: Starting NVIDIA fabric manager service...
   Sep 13 19:14:45 xxx nv-fabricmanager[2140]: Successfully configured all the available GPUs and NVSwitches.
   Sep 13 19:14:45 xxx systemd[1]: Started NVIDIA fabric manager service.

   Note

   The nvidia-fabricmanager package version must match the Tesla driver version to ensure the GPU works correctly. On Ubuntu, if you install the nvidia-fabricmanager service by using an installation package, the apt-daily service may automatically update the nvidia-fabricmanager package. This can cause a version mismatch with the Tesla driver, which prevents the nvidia-fabricmanager service from starting and makes the GPU unusable. To resolve this issue, see The GPU does not work as expected because the nvidia-fabricmanager version is different from the Tesla driver version.