Platform For AI:Instance interconnection for distributed training

Prerequisites

• You have multiple instances created from a general computing resource group or a Lingjun resource group. The instances must be in the same VPC.

• The resource group's public network access gateway must be set to Private Gateway.

• All instances must be in the same cluster. For example, Lingjun instances and general computing instances cannot interconnect.

• Only some instance types support RDMA/eRDMA. For details, see Default variables (pre-configured by the platform) and Limitations.

  DSW and DLC have the same RDMA/eRDMA functionality, so you can refer to the DLC documentation for details.

Supported features

• DSW pre-configures high-performance network environment variables optimized for different resource types and network architectures.

  • For new DSW instances created from a Lingjun resource group, see Default variables (pre-configured by the platform).

  • For new DSW instances created from a general computing resource group, see Platform-preconfigured environment variables.

• Instances of RDMA-capable instance types can interconnect over RDMA/eRDMA.

• Instances can interconnect by using their instance ID, which functions as a DNS domain name.

  root@dsw-1194983-84bb46b99b-jqqpm:/mnt/workspace# ping dsw-l28wnjxxx
  PING dsw-l28wnjdlyzf332fjj5.svc.cluster.local.c5333f3848f1f46328bed0fb647781d21.t1157703270994901 (10.244.48.2) 56(84) bytes of data.
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=1 ttl=64 time=0.018 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=2 ttl=64 time=0.019 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=3 ttl=64 time=0.018 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=4 ttl=64 time=0.021 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=5 ttl=64 time=0.016 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=6 ttl=64 time=0.021 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=7 ttl=64 time=0.020 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=8 ttl=64 time=0.020 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=9 ttl=64 time=0.021 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=10 ttl=64 time=0.021 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=11 ttl=64 time=0.021 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=12 ttl=64 time=0.021 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=13 ttl=64 time=0.019 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=14 ttl=64 time=0.039 ms
  64 bytes from 10.244.48.2 (10.244.48.2): icmp_seq=15 ttl=64 time=0.020 ms

These capabilities allow you to develop and debug multi-node, multi-GPU tasks.

Procedure

1. Use the DSW instance cloning feature to launch multiple instances with identical environments.

2. (Optional) Install the RDMA/eRDMA library on your instances.

   1. Lingjun resources: Use an image that includes the RDMA library. For more information, see Configure an image.

   2. General computing resources: See Install eRDMA library.

3. From one instance, ping the instance ID of another to verify network connectivity. For example: ping dsw-l28wnjdlyzj*********.

4. Configure and debug the multi-node job based on the distributed framework you use.