Manage third-party libraries in DSW - Platform For AI - Alibaba Cloud Documentation Center

Use the Terminal in your DSW instance to install, view, uninstall, and update Python third-party libraries. DSW automatically saves the changes to your image environment.

Install third-party libraries

DSW provides development environments that include Python 2, Python 3, PyTorch, and TensorFlow 2.0. By default, `pip` installs third-party libraries into the Python 3 environment. To install a library in a different environment, first switch to that environment. Use the following commands:

# Install a library in the Python 3 environment.
pip install <yourLibraryName>
# Install a library in the Python 2 environment.
source activate python2
pip install <yourLibraryName>
# Install a library in the TensorFlow 2.0 environment.
source activate tf2
pip install <yourLibraryName>

Replace <yourLibraryName> with the library's name. For example, run pip install bottle command to install the bottle library.

If you run the installation command in a Jupyter Notebook cell, prefix the command with an exclamation mark (!), for example: !pip install bottle.

Important

If a newly installed library is not found in your notebook, try restarting the Kernel.

View third-party libraries

To view installed libraries, run the following command:

pip list

Uninstall third-party libraries

To uninstall a library, run the following command:

pip uninstall <yourLibraryName>

Replace <yourLibraryName> with the name of the library to uninstall.

Note

You can only uninstall libraries that you installed yourself.

Update third-party libraries

You can run the following command to update an installed third-party library.

pip install --upgrade numpy==<versionNumber>

Replace <versionNumber> with the target version of numpy.

View or change the pip source

To view the current pip configuration, run the following command:

pip config list

In the output, the global.index-url value is the global pip source URL. The following is a sample output:

global.index-url='https://mirrors.aliyun.com/pypi/simple/'
global.trusted-host='mirrors.aliyun.com'
install.trusted-host='mirrors.aliyun.com'

To temporarily set the pip source for an installation, use the `-i` flag with the pip install command:

pip install -i https://pypi.tuna.tsinghua.edu.cn/simple <yourLibraryName>
# If an SSL authentication error occurs, try adding the --trusted-host parameter. For example:
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn <yourLibraryName>

To change the global pip source, edit the configuration file. The default path for the pip configuration file is ~/.config/pip/pip.conf. Run the following command to edit the file:

vim ~/.config/pip/pip.conf

For example, to change the source to the Tsinghua University mirror, modify the index-url and trusted-host parameters, then save and exit. The modified pip.conf file should look like this:

[global]
trusted-host=pypi.tuna.tsinghua.edu.cn
index-url = https://pypi.tuna.tsinghua.edu.cn/simple/

[install]
trusted-host=pypi.tuna.tsinghua.edu.cn

FAQ

Q: Will my pip-installed packages and code in a DSW instance be lost after shutdown?

No, your data will not be lost upon shutdown, provided the instance uses a cloud disk as its system disk. DSW preserves all instance disk data, including environments in /mnt/workspace and /root. All your files and installed packages will be available when you restart the instance. Note: Permanently deleting the instance will erase all its data.

Q: Why do I get an import error in DSW after installing a Python library with pip?

First, try restarting your Jupyter Kernel. If the error persists, it is likely because the library was installed in a different Python environment than the one your notebook is using. By default, DSW installs packages into the Python 3 environment. To install a library for a different environment, you must activate it first:

# Install in the Python 2 environment.
source activate python2
pip install --user xxx
# Install in the TensorFlow 2.0 environment.
source activate tf2
pip install --user xxx

Replace xxx with the name of the library you want to install.

Q: How do I resolve a `pip install` failure due to dependency conflicts or version errors in a DSW environment?

This error typically indicates an incompatibility between the package and your DSW environment's configuration (e.g., Python or CUDA versions). Follow these steps to resolve the issue:

Change the DSW Image (Recommended). The most reliable solution is to use a different official image that meets the library's requirements. Stop your current instance and create a new one, selecting a more suitable image. For example, if a package fails on the PyTorch 2.1 image, try an image with PyTorch 2.3 or one from the modelscope series, which often offer better compatibility.
Install a Specific Version. Check the library's official documentation for a version compatible with your environment's Python and CUDA versions. Then, install it directly using pip install package_name==x.y.z.
Use a Different Pip Mirror. Network issues with the default mirror can sometimes cause installation failures. Try an alternative mirror, such as the Tsinghua University mirror: pip install -i https://pypi.tuna.tsinghua.edu.cn/simple <yourLibraryName>.

Q: Why can't I import a library in a Jupyter Notebook after successfully installing it in the DSW Terminal?

This happens when the Terminal and the Jupyter Notebook are using different Python environments. You can confirm which environment each is using by running which python in the Terminal and !which python in a notebook cell. The simplest fix is to install the library directly from within the notebook, which ensures it is installed in the correct kernel environment. Prefix the pip command with an exclamation mark (!):

Q: How can I fix a "CUDA driver version is too low" error in a DSW instance? Should I upgrade the NVIDIA driver manually?

No, do not attempt to upgrade the NVIDIA driver or CUDA toolkit manually. The driver and CUDA versions within a DSW instance are pre-configured and immutable. Modifying them will likely corrupt the instance and lead to data loss. The correct solution is to replace the DSW image. Stop your current instance and create a new one using an official image that provides the required CUDA version.

For example, the image tag modelscope:1.9.4-pytorch2.0.1tensorflow2.13.0-gpu-py38-cu118-ubuntu20.04 indicates it includes CUDA 11.8 (as shown by cu118).

Q: Is it possible to run Docker commands inside a DSW instance?

It depends on the resource type. To use Docker with Lingjun resources, you must submit a ticket to be added to the whitelist. For all other DSW instances, running Docker is not supported.

Q: How do I extract .zip or .7z files in a DSW instance if `unzip` and `7z` commands are not found?

You can install the necessary tools using apt-get in the Terminal.

To extract .zip files: Install unzip by running apt-get update && apt-get install -y unzip. Then use unzip your_file.zip.
To extract .7z files: Install p7zip by running apt-get update && apt-get install -y p7zip-full. Then use 7z x your_file.7z.

Q: What should I do when `pip install` hangs or times out in a DSW instance?

A pip install command hanging or timing out is almost always a network-related issue. Follow these steps to diagnose and resolve it.

Step 1: Confirm network connectivity

First, verify that the instance can access the public internet by running ping www.aliyun.com in the Terminal. If this fails, your gateway may be misconfigured.

Step 2: Check the gateway configuration

In your DSW instance configuration, check the Internet Access Gateway:

Public Gateway: This is the default setting. It provides internet access with limited bandwidth, which can cause timeouts for large downloads. For better performance, consider using a Dedicated Gateway.
Dedicated Gateway: This offers faster network access but requires manual configuration. You must create an Internet NAT Gateway in your VPC, bind an Elastic IP (EIP), and configure SNAT rules. If not set up correctly, the instance will have no internet access. For more information, see Improve public network access speed with a dedicated gateway.

Step 3: Switch to a different pip mirror

The default mirror may be slow or temporarily unavailable. Use the -i flag to specify a different mirror for the installation.

# Install using the Tsinghua source (recommended)
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn <yourLibraryName>

# Install using the USTC source
pip install -i https://pypi.mirrors.ustc.edu.cn/simple --trusted-host pypi.mirrors.ustc.edu.cn <yourLibraryName>

# Install using the Douban source
pip install -i https://pypi.doubanio.com/simple --trusted-host pypi.doubanio.com <yourLibraryName>

To change the mirror permanently, see View or change the pip source.

Step 4: Use offline installation

If the network is unavailable or unreliable, you can install the package offline:

Download the package locally. On a machine with a stable connection, download the package's wheel (.whl) file:
```
# Run on your local computer
pip download <yourLibraryName> -d ./packages
```
Upload to DSW. Upload the downloaded .whl file to your DSW instance. For instructions, see Upload and download files.
Install from the file. In the DSW terminal, install from the uploaded file:
```
pip install /path/to/your-package.whl
```

Q: How do I get root permissions in a DSW WebIDE terminal?

You likely already have root permissions. Most official DSW images provide a root shell by default. To check, see if your terminal prompt is root@.... The "pip running as root" warning can be safely ignored. If your image does not provide root access, this is a characteristic of the image itself, and the only solution is to create a new instance using an image that does.

Q: Is there a way to start an xserver or run GUI applications in a DSW instance?

No, DSW environments do not support starting an xserver or running graphical applications.