Container Service for Kubernetes:Submit PyTorch distributed training jobs with eRDMA acceleration using Arena

Step 1: Install the ACK eRDMA Controller component

Install the ACK eRDMA Controller component.

1. On the Clusters page, click the name of your cluster. In the navigation pane on the left, click Add-ons.

2. On the Component Management page, click the Network tab, find the ACK eRDMA Controller component, and then follow the prompts to configure and install the component.

   Configuration Item	Description
   preferDriver Driver Type	Select the eRDMA driver type used on the cluster nodes. Valid values: default: The default driver mode. compat: The RoCE-compatible driver mode. ofed: The OFED-based driver mode. This mode is suitable for GPU-accelerated instance types. For more information about driver types, see Enable eRDMA.
   Assign all eRDMA devices on a node to a pod	Valid values: True (selected): Assigns all eRDMA devices on the node to the pod. False (cleared): Assigns one eRDMA device to the pod based on the NUMA topology. The node must have the CPU Static Policy enabled to ensure fixed NUMA allocation for pods and devices. For information about how to configure the CPU Policy, see Create and manage node pools.

   After the installation is complete, in the navigation pane on the left, choose Workloads > Pods. Then, set the namespace to ack-erdma-controller and check the pod status to confirm that the component is running as expected.

Why does the log not show NET/IB : Using erdma_0?

Possible causes include the following:

• You did not add the --device=aliyun/erdma=1 parameter in the submit command.

• The node does not support eRDMA or the eRDMA Controller add-on is not installed correctly.

• The container image lacks RDMA user-mode libraries (such as libibverbs and librdmacm).

Solutions:

1. Run kubectl get node <node-name> -o jsonpath='{.status.allocatable}' to confirm the node has aliyun/erdma resources.

2. Log on to the node and run ibv_devinfo to check if the eRDMA device works.

3. Check whether the container image includes required RDMA libraries.

What do I do if NCCL timeout errors occur during training?

NCCL timeouts usually result from unstable networks or incorrect configurations. We recommend:

1. Increase the timeout value: Set --env NCCL_IB_TIMEOUT=23 (default is 22).

2. Increase retry attempts: Set --env NCCL_IB_RETRY_CNT=10.

3. Check network connectivity between nodes and eRDMA device status.

How do I determine whether eRDMA actually improves training performance?

Compare results as follows:

1. Submit two training jobs: one with eRDMA (add --device=aliyun/erdma=1) and one without.

2. Compare completion times for the same number of training steps.

3. Analyze communication time percentage using the AI Profiling tool.

eRDMA delivers the most noticeable performance gains in multi-node (two or more nodes) scenarios with large models (billions of parameters or more).

What do I do if Arena fails to submit a job and reports insufficient resources?

Possible causes include the following:

• Insufficient GPU or eRDMA resources.

• Nodes do not meet scheduling requirements (such as node selectors or taint tolerations).

Solutions:

1. Run kubectl get nodes to view node status and available resources.

2. Run kubectl describe pod <pod-name> to view detailed reasons for pod scheduling failure.

3. Adjust resource requests or add nodes as needed.

Key Arena job parameters

NCCL environment variables (to optimize distributed communication):

For more NCCL environment variables, see the NCCL documentation.

How do I build a container image that supports eRDMA in production?

To use eRDMA, install RDMA user-mode libraries and drivers in the container. The following Dockerfile example builds on the official PyTorch image and integrates eRDMA drivers and the MNIST training example.

For more information about building container images that support eRDMA in production, see Enable eRDMA in containers (Docker).

ARG PYTORCH_IMAGE=docker.io/pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime

FROM docker.io/library/alpine:3.22.1 AS downloader

WORKDIR /workspace

RUN apk add git wget gzip

RUN mkdir -p /workspace/MNIST/raw && \
    cat <<EOF > /workspace/MNIST/raw/checksums.md5
f68b3c2dcbeaaa9fbdd348bbdeb94873 train-images-idx3-ubyte.gz
d53e105ee54ea40749a09fcbcd1e9432 train-labels-idx1-ubyte.gz
9fb629c4189551a2d022fa330f9573f3 t10k-images-idx3-ubyte.gz
ec29112dd5afa0611ce80d1b7f02629c t10k-labels-idx1-ubyte.gz
EOF

RUN cd /workspace/MNIST/raw && \
    wget https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz && \
    wget https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz && \
    wget https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz && \
    wget https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz && \
    md5sum -c checksums.md5 && \
    rm checksums.md5 && \
    gzip -d *.gz

RUN git clone https://github.com/kubeflow/arena.git -b v0.15.2

FROM ${PYTORCH_IMAGE}

WORKDIR /workspace

COPY --from=downloader /workspace .

RUN set -eux && \
    apt update && \
    apt install -y wget gpg && \
    wget -qO - https://mirrors.aliyun.com/erdma/GPGKEY | gpg --dearmour -o /etc/apt/trusted.gpg.d/erdma.gpg && \
    echo "deb [ ] https://mirrors.aliyun.com/erdma/apt/ubuntu jammy/erdma main" > /etc/apt/sources.list.d/erdma.list && \
    apt update && \
    apt install -y libibverbs1 ibverbs-providers ibverbs-utils librdmacm1 && \
    pip install --no-cache-dir -r /workspace/arena/examples/pytorch/mnist/requirements.txt && \
    rm -rf /var/lib/apt/lists/*