Container Service for Kubernetes:Submit eRDMA-accelerated PyTorch distributed training jobs using Arena

Step 1: Install ACK eRDMA Controller

You can perform the following steps to install ACK eRDMA Controller.

1. On the Clusters page, find the one you want to manage and click its name. In the left navigation pane, click Add-ons.

2. On the Add-ons page, click the Networking tab, find ACK eRDMA Controller, follow the instructions on the page to configure and install the component.

   Parameter	Description
   preferDriver Driver type	Select the type of the eRDMA driver used on the cluster nodes. Valid values: default: The default driver mode. compat: The driver mode that is compatible with RDMA over Converged Ethernet (RoCE). ofed: The ofed-based driver mode, which is applicable to GPU models. For more information about the types of drivers, see Enable eRDMA.
   Specifies whether to assign all eRDMA devices of nodes to pods	Valid values: True: If you select this check box, all eRDMA devices on the node are assigned to the pod. False: If you do not select this check box, the pod is assigned an eRDMA device based on the non-uniform memory access (NUMA) topology. You must enable the static CPU policy for the node to ensure that NUMA can be allocated to pods and devices. For more information about how to configure CPU policies, see Create and manage a node pool.

   In the left-side navigation pane, choose Workloads > Pods. On the Pods page, select the ack-erdma-controller namespace to view the status of pods and ensure that the component runs as expected.

Why doesn't the log show NET/IB : Using erdma_0?

Possible reasons include the following:

• The --device=aliyun/erdma=1 parameter was not added to the submission command.

• The node does not support eRDMA, or the eRDMA Controller component is not installed correctly.

• The container image is missing RDMA user mode libraries, such as `libibverbs` and `librdmacm`.

Solutions:

1. Run kubectl get node <node_name> -o jsonpath='{.status.allocatable}' to confirm that the node has the aliyun/erdma resource.

2. Log on to the node and run ibv_devinfo to check if the eRDMA device is functioning correctly.

3. Verify that the container image contains the necessary RDMA libraries.

What should I do if an NCCL timeout error occurs during training?

NCCL timeouts are usually caused by network instability or incorrect configuration. To resolve this issue, you can try the following:

1. Increase the timeout period: Set --env NCCL_IB_TIMEOUT=23 (the default is 22).

2. Increase the number of retries: Set --env NCCL_IB_RETRY_CNT=10.

3. Check the network connectivity between nodes and the status of the eRDMA devices.

How can I determine if eRDMA has improved training performance?

You can compare the performance by following these steps:

1. Submit two training jobs: one that uses eRDMA and one that does not. You can differentiate them by adding or omitting the --device=aliyun/erdma=1 parameter.

2. Compare the completion time for the same number of training steps.

3. Use the AI Profiling tool to analyze the proportion of time spent on communication.

Typically, the performance improvement from eRDMA is most significant in scenarios with multiple nodes (more than two) and large models (billions of parameters or more).

What should I do if an Arena job submission fails with an insufficient resources error?

Possible reasons include the following:

• Insufficient GPU or eRDMA resources.

• The node does not meet the scheduling conditions, such as node selectors or taint tolerations.

Solutions:

1. Run kubectl get nodes to view the node status and available resources.

2. Run kubectl describe pod <pod_name> to view the detailed reason for the pod scheduling failure.

3. Adjust the resource request or add more nodes to meet the requirements.

Core job parameters

NCCL environment variable configuration (for optimizing distributed communication):

For more information about NCCL environment variables, see the official NCCL documentation.

How do I build a container image that supports eRDMA for a production environment?

To use eRDMA, you must install the RDMA user mode libraries and drivers in the container. The following Dockerfile example is based on the official PyTorch image and integrates the eRDMA driver and the MNIST training example.

Build a production container image that supports eRDMA. For more information, see Enable eRDMA in a container (Docker).

ARG PYTORCH_IMAGE=docker.io/pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime

FROM docker.io/library/alpine:3.22.1 AS downloader

WORKDIR /workspace

RUN apk add git wget gzip

RUN mkdir -p /workspace/MNIST/raw && \
    cat <<EOF > /workspace/MNIST/raw/checksums.md5
f68b3c2dcbeaaa9fbdd348bbdeb94873 train-images-idx3-ubyte.gz
d53e105ee54ea40749a09fcbcd1e9432 train-labels-idx1-ubyte.gz
9fb629c4189551a2d022fa330f9573f3 t10k-images-idx3-ubyte.gz
ec29112dd5afa0611ce80d1b7f02629c t10k-labels-idx1-ubyte.gz
EOF

RUN cd /workspace/MNIST/raw && \
    wget https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz && \
    wget https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz && \
    wget https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz && \
    wget https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz && \
    md5sum -c checksums.md5 && \
    rm checksums.md5 && \
    gzip -d *.gz

RUN git clone https://github.com/kubeflow/arena.git -b v0.15.2

FROM ${PYTORCH_IMAGE}

WORKDIR /workspace

COPY --from=downloader /workspace .

RUN set -eux && \
    apt update && \
    apt install -y wget gpg && \
    wget -qO - https://mirrors.aliyun.com/erdma/GPGKEY | gpg --dearmour -o /etc/apt/trusted.gpg.d/erdma.gpg && \
    echo "deb [ ] https://mirrors.aliyun.com/erdma/apt/ubuntu jammy/erdma main" > /etc/apt/sources.list.d/erdma.list && \
    apt update && \
    apt install -y libibverbs1 ibverbs-providers ibverbs-utils librdmacm1 && \
    pip install --no-cache-dir -r /workspace/arena/examples/pytorch/mnist/requirements.txt && \
    rm -rf /var/lib/apt/lists/*