"undefined symbol: __nvJitLinkAddData_12_1, version libnvJitLink.so.12" error when using PyTorch on a Linux GPU-accelerated instance - Elastic GPU Service

On a Linux GPU-accelerated instance, you might encounter an error when importing PyTorch, caused by an incompatibility between the instance's CUDA version and the PyTorch version. This topic explains how to resolve this issue.

Symptom

When you use PyTorch on a GPU-accelerated instance running a Linux operating system, such as Alibaba Cloud Linux 3, you might see an error similar to the following:

>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/torch/__init__.py", line 235, in <module>
    from torch._C import *  # noqa: F403
ImportError: /usr/local/lib/python3.8/dist-packages/torch/lib/../../nvidia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkAddData_12_1, version libnvJitLink.so.12

Cause

This issue occurs because the CUDA version on the GPU-accelerated instance is incompatible with the PyTorch version. For more information about CUDA and PyTorch version compatibility, see Previous PyTorch Versions.

PyTorch 2.1.2, installed by running thesudo pip3 install torch command, requires CUDA 12.1. However, the CUDA version that is automatically installed when you purchase a GPU-accelerated instance is 12.0. This version mismatch causes the error.

Solution

If you selected the Auto-install GPU Driver option on the Public Image tab of the Image section when you purchased the GPU-accelerated instance, you can use one of the following methods to upgrade to CUDA 12.1.1.

Method 1: Manually install CUDA

Manually install CUDA 12.1.1. For more information, see NVIDIA CUDA Installation Guide for Linux.

Method 2: Install CUDA with a custom script

Release the existing GPU-accelerated instance.

For more information, see Release an instance.

Create a new GPU-accelerated instance.

For more information, see Create a GPU-accelerated instance. Configure the key parameters as follows:

On the Public Image tab of the Image section, ensure that the Auto-install GPU Driver option is cleared.

In the User Data section under Advanced Settings (Optional), enter a custom script to install NVIDIA Tesla driver 535.154.05 and CUDA 12.1.1. The following script is an example:

Sample custom script

#!/bin/sh
#Please input version to install
DRIVER_VERSION="535.154.05"
CUDA_VERSION="12.1.1"
CUDNN_VERSION="8.9.7.29"
IS_INSTALL_eRDMA="FALSE"
IS_INSTALL_RDMA="FALSE"
INSTALL_DIR="/root/auto_install"
#using .run to install driver and cuda
auto_install_script="auto_install_v4.0.sh"
script_download_url=$(curl http://100.100.100.200/latest/meta-data/source-address | head -1)"/opsx/ecs/linux/binary/script/${auto_install_script}"
echo $script_download_url
rm -rf $INSTALL_DIR
mkdir -p $INSTALL_DIR
cd $INSTALL_DIR && wget -t 10 --timeout=10 $script_download_url && bash ${INSTALL_DIR}/${auto_install_script} $DRIVER_VERSION $CUDA_VERSION $CUDNN_VERSION $IS_INSTALL_RDMA $IS_INSTALL_eRDMA

Method 3: Modify the user data and replace the operating system
1. Stop the existing GPU-accelerated instance.
  
  For more information, see Stop instances.
2. In the instance list, find the stopped GPU-accelerated instance. In the Actions column, choose Instance Settings > Set User Data.
3. Modify the user data and click OK.
  
  Change the values of the DRIVER_VERSION, CUDA_VERSION, and CUDNN_VERSION parameters to the following:
```
...
DRIVER_VERSION="535.154.05"
CUDA_VERSION="12.1.1"
CUDNN_VERSION="8.9.7.29"
...
```
4. Replace the operating system of the GPU-accelerated instance.
  
  For more information, see Replace the operating system (system disk) of an instance.
  
  After the GPU-accelerated instance restarts, the system reinstalls the new versions of the NVIDIA Tesla driver, CUDA, and cuDNN.