Manage third-party libraries in DSW - Platform For AI - Alibaba Cloud Documentation Center

In a DSW instance, you can use the Terminal to install, view, uninstall, and update Python third-party libraries. DSW automatically saves any modifications to the image environment.

Install third-party libraries

DSW provides several developer environments, including Python 2, Python 3, PyTorch, and TensorFlow 2.0. By default, third-party libraries are installed in the Python 3 environment. To install a library in a different environment, you must manually switch to that environment before you run the installation command. Use the following command format.

# Install a library in the Python 3 environment.
pip install <yourLibraryName>
# Install a library in the Python 2 environment.
source activate python2
pip install <yourLibraryName>
# Install a library in the TensorFlow 2.0 environment.
source activate tf2
pip install <yourLibraryName>

Replace <yourLibraryName> with the name of the third-party library that you want to install. For example, you can run the pip install bottle command to install the bottle library.

If you run the installation command in a Jupyter Notebook cell, add an exclamation mark (!) before the command. For example: !pip install bottle.

View third-party libraries

You can run the following command to view installed third-party libraries.

pip list

Uninstall third-party libraries

Run the following command to uninstall a third-party library.

pip uninstall <yourLibraryName>

Replace <yourLibraryName> with the name of your third-party library.

Note

You can only uninstall the third-party libraries that you installed.

Update third-party libraries

You can run the following command to update a third-party library.

pip install --upgrade numpy==<versionNumber>

Replace <versionNumber> with the version number of numpy to install.

View or change the pip source

You can run the following command to view the pip configuration file:

pip config list

In the response, the value of the global.index-url parameter is the global pip source URL. The following text is a sample response:

global.index-url='https://mirrors.aliyun.com/pypi/simple/'
global.trusted-host='mirrors.aliyun.com'
install.trusted-host='mirrors.aliyun.com'

To temporarily set the pip source when you install a third-party library, use the following command:

pip install -i https://pypi.tuna.tsinghua.edu.cn/simple <yourLibraryName>
# If an SSL authentication error occurs, try adding the --trusted-host parameter. For example:
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn <yourLibraryName>

To change the global pip source, you can edit the pip configuration file. The default path of the file is ~/.config/pip/pip.conf. Run the following command to edit the configuration file:

vim ~/.config/pip/pip.conf

The following example shows how to change the source to the Tsinghua pip source. Modify the index-url and trusted-host parameters, and then save and exit the file. The modified `pip.conf` file is as follows:

[global]
trusted-host=pypi.tuna.tsinghua.edu.cn
index-url = https://pypi.tuna.tsinghua.edu.cn/simple/

[install]
trusted-host=pypi.tuna.tsinghua.edu.cn

FAQ

Q: Are packages installed using `pip` lost after a DSW instance is stopped?

Your changes will not be lost. DSW automatically saves your modified image environment.

Q: I failed to install a package using `pip install` in DSW due to a dependency conflict or version error. What should I do?

This issue is usually caused by an environment incompatibility. You can troubleshoot the issue by performing the following steps in order:

Recommended solution: Change the image. Stop the current instance, create a new DSW instance, and select a different official image. For example, if the current PyTorch 2.1 image does not work, you can try the PyTorch 2.3 image or an image from the modelscope series. The `modelscope` series images typically offer better compatibility.
Install a specific version. Check the official documentation of the package to find a version that is compatible with your current DSW environment (Python and CUDA versions). Then, run pip install package_name==x.y.z.
Change the download source. Try using a different mirror, such as the Tsinghua University source: pip install -i https://pypi.tuna.tsinghua.edu.cn/simple <yourLibraryName>.

Q: I installed a library in the DSW Terminal, but why can I not find it when I import it in a Jupyter Notebook?

This may occur because the Terminal and Jupyter Notebook are using different Python environments. You can run the which python command to check which Python environment is being used. Alternatively, you can install the required library in the Notebook. For example:

Q: My code reports an error that the CUDA driver version is too low. Do I need to manually upgrade the NVIDIA driver in DSW?

No, you do not need to. The drivers and CUDA in DSW instances are pre-installed and locked. Do not manually modify them. If you do, the instance may be corrupted and cannot be recovered. The correct approach is to change the DSW image. Stop the current instance, create a new instance, and select an official image with higher CUDA and driver versions.

For example, you can use the official image modelscope:1.9.4-pytorch2.0.1tensorflow2.13.0-gpu-py38-cu118-ubuntu20.04. In the image name, `cu118` indicates that the CUDA version is 11.8.

Q: Can I use Docker to deploy applications in DSW?

No, you cannot. DSW is based on container technology and does not support running Docker within a container. This means that nested virtualization is not supported. If your application has a strong dependency on Docker, use Alibaba Cloud ECS instances for deployment.

Q: The `unzip` and `7z` commands are not available in my DSW instance. How do I decompress files?

You can install them using the apt-get command.

To install unzip: In the Terminal, run apt-get update && apt-get install -y unzip. Then, use unzip your_file.zip.
To install p7zip (for 7z): In the Terminal, run apt-get update && apt-get install -y p7zip-full. Then, use 7z x your_file.7z.

Q: The installation is stuck or times out. What is the cause?

This may be a network issue.

Check whether the instance is configured for public network access. If you selected No Public IP Address when you created the instance or configured a VPC without a NAT Gateway, the instance cannot connect to external download sources.
Try to change the download source. For example, you can switch from the default Alibaba Cloud source to the Tsinghua University source: pip install -i https://pypi.tuna.tsinghua.edu.cn/simple <yourLibraryName>.
If the network is unavailable, you can download the installation package in .whl format to your computer and then upload it to DSW for an offline installation.

Q: How do I obtain root permissions in the DSW WebIDE?

Most official DSW images run as the root user by default. If the command prompt is root@... when you open the Terminal, you already have root permission. You can safely ignore the warning message that appears when you use pip to install packages, which suggests not running the command as the root user. If you are not logged on as the root user, this is a setting of the image. You must switch to an image that supports root logon.