Container Service for Kubernetes:Use eRDMA to accelerate container networks in ACK clusters

Enable eRDMA device access

Declare the aliyun/erdma resource in the container's resource limits:

spec:
  containers:
  - name: erdma-container
    resources:
      limits:
        aliyun/erdma: 1

After the pod starts, verify that the eRDMA devices are available:

/# ls /dev/infiniband/
rdma_cm  uverbs0

Enable transparent TCP acceleration with SMC-R

SMC-R transparently accelerates TCP connections using RDMA — no application code changes required. After enabling eRDMA device access, add the following annotation to the pod:

metadata:
  annotations:
    network.alibabacloud.com/erdma-smcr: "true"

Scenario 1: Accelerate NCCL communication for GPU distributed training

Use this scenario for GPU-accelerated instance types running distributed training workloads. All pods use the ofed driver and request both GPU and eRDMA resources.

1. Install the ACK eRDMA Controller with preferDriver set to ofed.

2. Add GPU nodes to the node pool. See Create and manage node pools.

3. Install eRDMA-related packages in your container image at build time. For Debian or Ubuntu (replace {OS|ubuntu} and {Version|focal} with your OS name and version):

   wget -qO - https://mirrors.aliyun.com/erdma/GPGKEY | apt-key add - \
     && echo "deb [ arch=amd64 ] https://mirrors.aliyun.com/erdma/apt/{OS|ubuntu} {Version|focal}/erdma main" \
       | tee /etc/apt/sources.list.d/erdma.list \
     && apt update \
     && apt install -y libibverbs1 ibverbs-providers ibverbs-utils librdmacm1

   For Alibaba Cloud Linux or RHEL:

   cat > /etc/yum.repos.d/erdma.repo <<EOF
   [erdma]
   name = ERDMA Repository
   baseurl = http://mirrors.aliyun.com/erdma/yum/redhat/7/erdma/x86_64/
   gpgcheck = 0
   enabled = 1
   EOF
   yum install --disablerepo=* --enablerepo erdma -y libibverbs ibverbs-providers ibverbs-utils librdmacm

4. Deploy the GPU application. The following StatefulSet template runs nccl-test across two replicas, each with 8 GPUs and one eRDMA device:

   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: nccltest
   spec:
     selector:
       matchLabels:
         app: nccltest
     serviceName: "nccltest"
     replicas: 2
     template:
       metadata:
         labels:
           app: nccltest
       spec:
         hostNetwork: true
         dnsPolicy: ClusterFirstWithHostNet
         containers:
         - env:
           - name: NCCL_SOCKET_IFNAME
             value: "eth0"
           - name: NCCL_DEBUG
             value: "INFO"
           - name: NCCL_IB_GID_INDEX
             value: "1"
           image: <nccl-test-image-with-erdma>
           imagePullPolicy: Always
           name: nccltest
           securityContext:
             privileged: true
           resources:
             limits:
               nvidia.com/gpu: "8"
               aliyun/erdma: "1"
             requests:
               nvidia.com/gpu: "8"
               aliyun/erdma: "1"

5. Verify that NCCL is using eRDMA. Check the application logs — the expected output lists the eRDMA devices NCCL is using for communication. The log confirms that erdma_0 and erdma_1 are active, indicating eRDMA is accelerating the NCCL communication.

   

Scenario 2: Transparently accelerate application networks with SMC-R

Use this scenario for standard TCP-based workloads where you want eRDMA acceleration without modifying application code. Set preferDriver to default.

1. Install the ACK eRDMA Controller with preferDriver set to default.

2. Deploy the application with the SMC-R annotation. The following Deployment enables both eRDMA device access and transparent SMC-R acceleration:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: app-with-erdma
     name: app-with-erdma
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: app-with-erdma
     template:
       metadata:
         labels:
           app: app-with-erdma
         annotations:
           network.alibabacloud.com/erdma-smcr: "true"
       spec:
         containers:
         - image: <application image>
           imagePullPolicy: Always
           name: app-with-erdma
           resources:
             limits:
               aliyun/erdma: 1

3. Verify the acceleration status. Install smc-tools in the container and run smcss:

   /# smcss
   State          UID   Inode   Local Address           Peer Address            Intf Mode
   ACTIVE         00000 0059964 172.17.192.73:47772     172.17.192.10:80        0000 SMCR

   SMCR in the Mode column confirms the connection is using eRDMA acceleration. If the column shows TCP, check that both the client and server pods have the SMC-R annotation set to "true".