Manage third-party libraries in DSW - Platform For AI - Alibaba Cloud Documentation Center

In a Data Science Workshop (DSW) instance, you can install, view, uninstall, and update third-party Python libraries from a terminal. DSW automatically saves changes to the image environment.

Install a third-party library

DSW includes development environments such as Python 2, Python 3, PyTorch, and TensorFlow 2.0. By default, packages are installed into the Python 3 environment. To install a package into a different environment, switch to that environment first. Use the following command format.

# Install into the Python 3 environment.
pip install <yourLibraryName>
# Install into the Python 2 environment.
source activate python2
pip install <yourLibraryName>
# Install into the TensorFlow 2.0 environment.
source activate tf2
pip install <yourLibraryName>

Replace <yourLibraryName> with the name of the library to install. For example, run pip install bottle to install the bottle library.

If you run the install command from a Jupyter Notebook cell, prefix the command with an exclamation mark !. For example: !pip install bottle.

Important

If a package was installed successfully but cannot be imported in a notebook, reload the kernel.

Note

When you install large frameworks such as TensorFlow or PyTorch, the process of pip unpacking the wheel package consumes a large amount of memory. If the install process is Killed by the system, it usually indicates insufficient instance memory. We recommend that you switch the DSW instance to an instance type with 30 GiB or more memory, or select a PAI official image that has the target framework preinstalled when you create the instance, to avoid manual installation failures.

View installed third-party libraries

Run the following command to list installed third-party libraries.

pip list

Uninstall a third-party library

Run the following command to uninstall an installed third-party library.

pip uninstall <yourLibraryName>

Replace <yourLibraryName> with the name of the library to uninstall.

Note

You can only uninstall libraries that you installed yourself.

Update a third-party library

Run the following command to update an installed third-party library.

pip install --upgrade numpy==<versionNumber>

Replace <versionNumber> with the target version of numpy.

View or change the pip index URL

Run the following command to view the pip configuration.

pip config list

In the output, the value of the global.index-url parameter is the global pip index URL. Sample output:

global.index-url='https://mirrors.aliyun.com/pypi/simple/'
global.trusted-host='mirrors.aliyun.com'
install.trusted-host='mirrors.aliyun.com'

To set the pip index URL temporarily for a single install command, use the -i option:

pip install -i https://pypi.tuna.tsinghua.edu.cn/simple <yourLibraryName>
# If an SSL verification error occurs, add the --trusted-host parameter:
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn <yourLibraryName>

To change the global pip index URL, edit the pip configuration file. The default path is ~/.config/pip/pip.conf. Run the following command to edit the file:

vim ~/.config/pip/pip.conf

The following example uses the Tsinghua University PyPI mirror. Update the index-url and trusted-host parameters, and then save and exit. The updated pip.conf file:

[global]
trusted-host=pypi.tuna.tsinghua.edu.cn
index-url = https://pypi.tuna.tsinghua.edu.cn/simple/

[install]
trusted-host=pypi.tuna.tsinghua.edu.cn

FAQ

Are my pip-installed packages and code retained after my DSW instance is stopped?

Yes, if the instance uses a cloud disk as the system disk. The disk data of the instance, including the environments under /mnt/workspace and /root, is retained. When the instance is started again, all environments and files are still available. Data is permanently cleared only when the instance is deleted.

Why does an installed package not work?

After you install a package with pip but cannot import it with import, restart the service or the kernel first. If the error persists, verify the current environment. When you install a package, DSW installs it into the Python 3 environment by default. To install the package into a different environment, switch to that environment first. Example:

# Install into the Python 2 environment.
source activate python2
pip install --user xxx
# Install into the TensorFlow 2.0 environment.
source activate tf2
pip install --user xxx

Replace xxx with the name of the package to install.

What should I do if `pip install` fails with a dependency conflict or version error in DSW?

Such errors are usually caused by environment incompatibility. Troubleshoot in the following order:

Preferred solution: change the image. Stop the current instance, create a new DSW instance, and select a different official image. For example, if the current PyTorch 2.1 image does not work, try the PyTorch 2.3 image or a modelscope series image, which generally provides better compatibility.
Install a specific version. Check the package documentation for a version that supports the Python and CUDA versions of your DSW environment, and then run pip install package_name==x.y.z.
Use a different PyPI mirror. Try a PyPI mirror hosted in China such as the Tsinghua mirror: pip install -i https://pypi.tuna.tsinghua.edu.cn/simple <yourLibraryName>.

A library is installed in the DSW terminal but `import` cannot find it in Jupyter Notebook. What should I do?

This is usually because the terminal and Jupyter use different Python environments. Run which python to confirm the current Python environment, or install the required library from a notebook cell. Example:

!which python
/usr/local/bin/python

!pip install bottle
Looking in indexes: https://mirrors.aliyun.com/pypi/simple/
Collecting bottle
  Downloading https://mirrors.aliyun.com/pypi/packages/83/f6/b55...
                                            103.8/103.8 kB 6.7 ...
Installing collected packages: bottle
Successfully installed bottle-0.13.4
WARNING: Running pip as the 'root' user can result in broken per...

[notice] A new release of pip is available: 23.3.2 -> 25.1.1
[notice] To update

My code reports that the CUDA driver version is too low. Should I manually upgrade the NVIDIA driver in DSW?

Do not upgrade the driver. The driver and CUDA stack on a DSW instance are preinstalled and locked. Manual modification is not allowed and not recommended because it can corrupt the instance and make it unrecoverable. The correct approach is to change the DSW image: stop the current instance, create a new instance, and select an official image that ships with a newer version of CUDA and the driver.

For example, the official image modelscope:1.9.4-pytorch2.0.1tensorflow2.13.0-gpu-py38-cu118-ubuntu20.04 includes CUDA 11.8 (the cu118 tag).

VS Code reports an ETIMEDOUT error when I install an extension in DSW. How do I fix this?

Installing a VS Code extension requires access to the VS Code Marketplace (marketplace.visualstudio.com), which is hosted outside mainland China. If the DSW instance uses a public gateway (shared bandwidth), access to resources outside mainland China may be very slow or time out.

Solutions:

Verify network connectivity. In the terminal, run curl -I https://marketplace.visualstudio.com and check whether the request returns successfully.
If access to resources outside mainland China times out, see Improve the public network download speed of DSW to configure a public NAT gateway and an EIP for the VPC where the instance resides, and then use a dedicated gateway.
After a dedicated gateway is configured, if resources outside mainland China still cannot be accessed, see Accelerate access to resources outside mainland China from DSW to enable Global Accelerator (GA).

Can I use Docker in DSW to deploy my application?

Nested containers (sub-containers) are supported only on DSW instances that were created by a Lingjun resource group or a general-purpose resource group of version 1.0. For details, see Sub-container management (DockerBoard) and Use Docker in DSW.

There is no `unzip` or `7z` command in the DSW instance. How do I decompress files?

You can install them by using apt-get.

Install unzip: in the terminal, run apt-get update && apt-get install -y unzip, and then use unzip your_file.zip.
Install p7zip (for 7z): in the terminal, run apt-get update && apt-get install -y p7zip-full, and then use 7z x your_file.7z.

Package installation hangs or times out. What should I do?

If a package installation hangs, times out, or is extremely slow, the cause is usually a network issue. Troubleshoot in the following order:

Step 1: Verify network connectivity

In the terminal, run ping www.aliyun.com to check whether the instance can reach the Internet. If the network is unreachable, proceed to Step 2 to check the gateway configuration.

Step 2: Check the gateway configuration

On the instance configuration page, check the type of Internet Access Gateway:

Public gateway: By default, DSW uses a Public Gateway to access the Internet. You can confirm the gateway type on the DSW instance configuration page. When you use a public gateway, the bandwidth is limited and download speeds may be insufficient for large files. In this case, select a dedicated gateway.
Dedicated gateway: A dedicated gateway provides higher network speeds. After you select a dedicated gateway, you must create a public NAT gateway in a virtual private cloud (VPC), bind an Elastic IP (EIP), and configure SNAT rules. Otherwise, Internet access is unavailable. For details, see Improve internet access with a private gateway.

Step 3: Try a different pip index URL

DSW uses the Alibaba Cloud PyPI mirror by default, which may experience issues during peak hours or network fluctuations. Try switching to another PyPI mirror hosted in China:

# Install by using the Tsinghua PyPI mirror (recommended)
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn <yourLibraryName>

# Install by using the USTC PyPI mirror
pip install -i https://pypi.mirrors.ustc.edu.cn/simple --trusted-host pypi.mirrors.ustc.edu.cn <yourLibraryName>

# Install by using the Douban PyPI mirror
pip install -i https://pypi.doubanio.com/simple --trusted-host pypi.doubanio.com <yourLibraryName>

You can also change the default pip index URL permanently. See View or change the pip index URL.

Step 4: Use offline installation

If the network is unreachable or highly unstable, install the package offline:

On your local computer (with good network connectivity), download the .whl package:
```
# Run on your local computer
pip download <yourLibraryName> -d ./packages
```
Upload the downloaded .whl file to the DSW instance. For details, see Upload and download files.
Install the package offline in DSW:
```
pip install /path/to/your-package.whl
```

Access to the aiext-pypi mirror (for example, aiext-pypi.mirrors.aliyuncs.com) fails. What should I do?

aiext-pypi.mirrors.aliyuncs.com is an internal pip mirror configured exclusively for PAI-PPU images. It can be accessed only from DSW instances that use a PAI-PPU series image. If your DSW instance uses a different image (such as a PyTorch or TensorFlow official image), this mirror is unreachable. This is expected and is not a network failure.

Solutions:

To use this mirror, create a new instance by using a PAI-PPU series image.
If you do not need a PAI-PPU image, switch to another public pip mirror hosted in China (such as the Alibaba Cloud PyPI mirror or the Tsinghua PyPI mirror). See View or change the pip index URL.

How do I obtain root privileges in the browser-based IDE of DSW?

Most official DSW images run as the root user by default. If the terminal prompt starts with root@, you are already root. The "running pip as the root user is not recommended" warning can be ignored. If your image does not log in as root, the behavior is determined by the image itself. Switch to an image that supports root login.

How do I start X server in DSW?

DSW does not support X server.

Can I use Vulkan for graphics rendering in DSW?

No. DSW runs in a containerized environment. The GPU driver is preinstalled and locked by the official image. You cannot load the nvidia-drm kernel module or switch to the NVIDIA proprietary driver. Because of this platform limitation, the Vulkan ICD cannot be initialized. This issue cannot be resolved by environment variables or by changing the image. To run graphics-intensive applications that depend on Vulkan, such as SAPIEN, use an ECS GPU instance or deploy with a PAI-EAS custom image.

Install and use Miniconda

To use Miniconda to manage Python runtime environments and third-party packages, follow these steps to install it.

Open a terminal and run the following commands to download and install Miniconda.

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b -p /root/miniconda3
/root/miniconda3/bin/conda init

Stop the DSW instance and restart it. After the restart, you can use the conda Python environment. Run the following command to check the current Python environment.
```
# Run in a terminal
which python 

# Run in a notebook cell
!which python
```
If the conda Python environment is still not available in the terminal, update the PATH in the .bashrc file as follows:
1. In the terminal, run cd /root/ to go to the directory that contains the .bashrc file.
2. Use vim to edit the .bashrc file. Add export PATH="/root/miniconda3/bin:$PATH" at the top of the file, and then save and exit.
3. In the terminal, run source .bashrc to apply the changes.
4. Run which python again to verify the current Python environment.