ACCL high-performance collective communication library - Platform For AI

ACCL (Alibaba Collective Communication Library) extends NVIDIA NCCL with Alibaba Cloud network optimizations to deliver higher communication performance for distributed training, with built-in fault diagnosis and self-recovery.

Key features

ACCL provides the following capabilities, configurable through environment variables:

Fixes bugs found in the corresponding open-source NCCL version.
Optimizes collective communication operators and message sizes for higher performance than open-source NCCL.
Collects communication statistics to diagnose slowdowns and hangs from device faults. Integrates with PAI AIMaster: An elastic and automatic fault tolerance engine and C4D: Model training job diagnosis tool for rapid anomaly detection and automatic fault tolerance.
Supports multi-path transmission and load balancing to reduce congestion from uneven hashing and improve training throughput.

Limits

ACCL is required for DLC jobs that use custom images with Lingjun resources in regions where Lingjun resources are available.

Install the ACCL library

Official Lingjun images already include ACCL. Skip this section if you use them.

Step 1: Verify NCCL dynamic linking in PyTorch

In your custom image container:

Determine the location of the PyTorch library.

For example, if PyTorch is installed in /usr/local/lib:

find /usr/local/lib -name "libtorch*"
# Example results:
/usr/local/lib/python3.10/dist-packages/torch/lib/libtorchcuda.so
/usr/local/lib/python3.10/dist-packages/torch/lib/libtorch.so
/usr/local/lib/python3.10/dist-packages/torch/lib/libtorchbindtest.so

Check PyTorch's NCCL library dependency using the ldd command:
```
ldd libtorch.so | grep nccl
```
- If output appears, NCCL is dynamically linked. Proceed to install ACCL:
```
libnccl.so.2=>/usr/lib/x86_64-linux-gnu/libnccl.so.2(0x00007feab3b27000)
```
- If no output appears, NCCL is statically linked and ACCL cannot replace it. Use a PyTorch version with dynamic NCCL linking, or create a custom image based on the official NVIDIA NGC image.

Step 2: Check the CUDA version

Verify your CUDA version:

nvidia-smi

Note the CUDA version in your output (12.2 in this example).

Step 3: Download ACCL for your CUDA version

ACCL download links by CUDA version:

CUDA version	ACCL download link
12.8	http://accl-n.oss-cn-beijing.aliyuncs.com/cuda12.8/lib/libnccl.so.2
12.6	http://accl-n.oss-cn-beijing.aliyuncs.com/cuda12.6/lib/libnccl.so.2
12.5	http://accl-n.oss-cn-beijing.aliyuncs.com/cuda12.5/lib/libnccl.so.2
12.4	http://accl-n.oss-cn-beijing.aliyuncs.com/cuda12.4/lib/libnccl.so.2
12.3	http://accl-n.oss-cn-beijing.aliyuncs.com/cuda12.3/lib/libnccl.so.2
12.2	http://accl-n.oss-cn-beijing.aliyuncs.com/cuda12.2/lib/libnccl.so.2
12.1	http://accl-n.oss-cn-beijing.aliyuncs.com/cuda12.1/lib/libnccl.so.2
11.8	http://accl-n.oss-cn-beijing.aliyuncs.com/cuda11.8/lib/libnccl.so.2
11.7	http://accl-n.oss-cn-beijing.aliyuncs.com/cuda11.7/lib/libnccl.so.2
11.4	http://accl-n.oss-cn-beijing.aliyuncs.com/cuda11.4/lib/libnccl.so.2

Download ACCL for your CUDA version. Example for CUDA 12.3:

wget http://accl-n.oss-cn-beijing.aliyuncs.com/cuda12.3/lib/libnccl.so.2

Step 4: Install ACCL

Check whether libnccl.so.2 exists on your system:

sudo find / -name "libnccl.so.2"

Then take one of the following actions:

If libnccl.so.2 is not found or is in /usr/lib64 or /lib64, copy the downloaded file to the system directory:
```
sudo cp -f ./libnccl.so.2 /usr/lib64
```
If libnccl.so.2 is in a non-standard directory (e.g. /opt/xxx/), overwrite it with the downloaded file:
```
sudo cp -f libnccl.so.2 /opt/xxx/
```

Step 5: Refresh dynamic library cache

Refresh the dynamic library cache:

sudo ldconfig

Step 6: Verify ACCL is loaded

Submit a DLC job by using a custom image. For more information, see Create a training job.
Check the job log. If the startup log displays ACCL version information, the library is loaded. For instructions on viewing job logs, see Training job details.

Note
Ensure the log includes the accl-n identifier. Otherwise, ACCL is not loaded.
```
NCCL version 2.20.5.7-accl-n+cuda12.4, COMMIT_ID Zeaa6674c2f1f896e3a6bbd77e85231e0700****, BUILD_TIME 2024-05-10 15:40:56
```

Recommended environment variables

Set these environment variables to optimize communication throughput:

export NCCL_IB_TC=136
export NCCL_IB_SL=5
export NCCL_IB_GID_INDEX=3
export NCCL_SOCKET_IFNAME=eth
export NCCL_DEBUG=INFO
export NCCL_IB_HCA=mlx5
export NCCL_IB_TIMEOUT=22
export NCCL_IB_QPS_PER_CONNECTION=8
export NCCL_MIN_NCHANNELS=4
export NCCL_NET_PLUGIN=none
export ACCL_C4_STATS_MODE=CONN
export ACCL_IB_SPLIT_DATA_NUM=4
export ACCL_IB_QPS_LOAD_BALANCE=1
export ACCL_IB_GID_INDEX_FIX=1
export ACCL_LOG_TIME=1

Key environment variables:

Environment variable	Description
NCCL_IB_TC	Network traffic class mapping for Alibaba Cloud. Incorrect values degrade performance.
NCCL_IB_GID_INDEX	GID index for RDMA. Incorrect values cause NCCL errors.
NCCL_SOCKET_IFNAME	Network interface for NCCL connections. Varies by instance type. Incorrect values cause connection failures.
NCCL_DEBUG	NCCL log level. Set to INFO for detailed troubleshooting logs.
NCCL_IB_HCA	RDMA network interface card. Incorrect values degrade performance.
NCCL_IB_TIMEOUT	RDMA connection timeout. Improves fault tolerance. Incorrect values cause training interruptions.
NCCL_IB_QPS_PER_CONNECTION	Queue pairs per connection. Higher values boost network throughput.
NCCL_NET_PLUGIN	NCCL network plugin. Set to none to prevent other plugins from loading.
ACCL_C4_STATS_MODE	ACCL statistics granularity. CONN aggregates by connection.
ACCL_IB_SPLIT_DATA_NUM	Number of parts to split data across multiple queue pairs for transmission.
ACCL_IB_QPS_LOAD_BALANCE	Enables or disables load balancing across queue pairs.
ACCL_IB_GID_INDEX_FIX	Detects and bypasses GID anomalies before job start. Set to 1 to enable.
ACCL_LOG_TIME	Prepends timestamps to log entries for troubleshooting. Set to 1 to enable.