Platform For AI:Instance interconnection for distributed training

Prerequisites

• You have multiple instances created from a general computing resource group or a Lingjun resource group. All instances must be in the same VPC.

• The public access gateway for the resource group containing the instances must be set to Private Gateway.

• All instances must be in the same cluster (for example, instances from a Lingjun resource group cannot interconnect with instances from a general computing resource group).

• Only specific instance types support Remote Direct Memory Access (RDMA) or enhanced RDMA (eRDMA). For details, see Default variables (pre-configured by the platform) and Limitations.

  DSW and DLC provide the same features for RDMA and eRDMA. Refer to the DLC documentation for more information.

Supported features

• Pre-configured, high-performance network environment variables tailored for different resource types and network architectures.

  • For new DSW instances in a Lingjun resource group, see Default variables (pre-configured by the platform).

  • For new DSW instances in a general computing resource group, see Platform-preconfigured environment variables.

• On nodes that support RDMA, instances can interconnect over RDMA/eRDMA.

• Instances can interconnect using their instance ID, which functions as a DNS hostname.

  root@dsw-1194983-84bb46b99b-jqqpm:/mnt/workspace# ping dsw-l28wnjxxx
  PING dsw-l28wnjdlyzf332fjj5.svc.cluster.local.c5333f3848f1f46328bed0fb647781d21.t1157703270994901 (10.244.48.2) 56(84) bytes of data.
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=1 ttl=64 time=0.018 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=2 ttl=64 time=0.019 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=3 ttl=64 time=0.018 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=4 ttl=64 time=0.021 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=5 ttl=64 time=0.016 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=6 ttl=64 time=0.021 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=7 ttl=64 time=0.020 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=8 ttl=64 time=0.020 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=9 ttl=64 time=0.021 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=10 ttl=64 time=0.021 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=11 ttl=64 time=0.021 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=12 ttl=64 time=0.021 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=13 ttl=64 time=0.019 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=14 ttl=64 time=0.039 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=15 ttl=64 time=0.020 ms

These capabilities let you develop and debug multi-node, multi-GPU tasks in DSW.

Procedure

1. Use the DSW instance cloning feature to launch the required number of instances with identical environments.

2. (Optional) Install the RDMA or eRDMA library on your instances.

   1. For a Lingjun resource group: Use an image that includes the RDMA library. For details, see Configure an image.

   2. For a general computing resource group: Install eRDMA library.

3. From instance A, ping the instance ID of instance B to verify the network connectivity. For example: ping dsw-l28wnjdlyzj*********.

4. Configure and debug your multi-node tasks based on your distributed training framework.