When you submit a training task in Deep Learning Containers (DLC) of Platform for AI (PAI), DLC automatically injects environment variables that you can use directly in your code.
Common environment variables
These environment variables build on Lingjun Intelligent Computing. You can find the description of each variable in RDMA: high-performance networks for distributed training.
PyTorch environment variables
In distributed PyTorch training, the master node and the worker nodes must establish a connection to communicate. DLC synchronizes key information, such as the address and port of the master node, through the following environment variables:
| Variable | Description |
|---|---|
MASTER_ADDR | Address of the master node. For example, dlc18isgeayd****-master-0. This value is usually the Service address of the master node. |
MASTER_PORT | Port of the master node. For example, 23456. |
WORLD_SIZE | Total number of nodes in the distributed task. For example, if you submit a task that contains one master node and one worker node, WORLD_SIZE is set to 2. |
RANK | Index of the node. For example, if you submit a task that contains one master node and two worker nodes, RANK is 0 on the master node, 1 on worker-0, and 2 on worker-1. |
NPROC_PER_NODE | Number of GPUs on each worker node. For example, if the worker node uses the 8 GPUs * GU7E specification, NPROC_PER_NODE is 8 on that worker node. |
TensorFlow environment variables
A distributed TensorFlow task builds its distributed network topology from the TF_CONFIG environment variable. DLC sets the following environment variables for TensorFlow training:
| Variable | Description |
|---|---|
TF_CONFIG | Distributed network topology of TensorFlow. Example: |