The whale.cluster operation supports a variety of grouping strategies for you to group hardware resources and map the hardware resources to virtual devices. This topic describes the syntax and parameters of the whale.cluster operation. This topic also describes the grouping strategies and workers involved in the operation. In addition, this topic provides the sample code to show you how to call the operation to group and map resources.

Background information

A distributed training framework is used to distribute different parts of a model to distributed hardware resources by using different parallelism strategies. This way, data can be computed in parallel. In Whale, this process consists of two parts: group and map resources, and divide the model. The whale.cluster operation is a tool that Whale provides to group distributed computing resources and map the computing resources to logical resources.

This operation involves the following important concepts:
  • cluster

    The cluster class is used to manage the mappings between hardware resources and logical resources. It supports a variety of grouping strategies for you to group hardware resources and map the hardware resources to virtual devices.

  • layout

    The layout parameter is contained in the cluster class. The layout parameter specifies how to group hardware resources and map the hardware resources to logical resources.

Syntax

cluster(worker_hosts=None,
        ps_hosts=None,
        job_name="worker",
        rank=0,
        layout="all")

Parameters

  • worker_hosts: the host information about workers. The value is of the STRING type. The default value is None, which indicates to obtain the host and ranking information from the TF_CONFIG environment variable.
  • ps_hosts: the host information about parameter servers. The value is of the STRING type.
  • job_name: the name of the job allocated to the current worker. The value is of the STRING type. The default value is worker.
  • rank: the ranking of the current worker. The value is of the INTEGER type.
  • layout: the grouping strategy for resources. The value is of the DICT type. The default value is all. The whale.cluster operation supports the following resource grouping strategies:
    • all

      Place all GPUs in a group.

      This section describes how to implement the grouping strategy for the resources shown in the following figure. This section also provides a sample response to show you the effect of the grouping strategy.allYou can run the following code to implement this grouping strategy:
      import whale as wh
      cluster = wh.cluster(layout="all")
      The system returns a cluster.slices object similar to the following example:
      [
          [ 
              [Worker0:GPU0], 
              [Worker0:GPU1], 
              [Worker0:GPU2], 
              [Worker0:GPU3], 
              [Worker1:GPU0], 
              [Worker1:GPU1], 
              [Worker1:GPU2], 
              [Worker1:GPU3] 
          ] 
      ]
    • average

      Place a specified number of GPUs in each group.

      Arrange GPUs by row for each worker, as shown in the following figure.Arrange by rowYou can run the following code to implement this grouping strategy:
      import whale as wh
      cluster = wh.cluster(layout = {'average' : 4 } )
      The system returns a cluster.slices object similar to the following example:
      [
          [ 
              [Worker0:GPU0], 
              [Worker0:GPU1],
              [Worker0:GPU2], 
              [Worker0:GPU3]
          ],
          [
              [Worker1:GPU0],
              [Worker1:GPU1],
              [Worker1:GPU2],
              [Worker1:GPU3]
          ],
      ]
      The following figure shows the resource grouping result of the average strategy.average
    • block

      Group GPUs by block.

      In this example, resources are divided into two groups. One group contains six GPUs and the other contains two GPUs, as shown in the following figure. Each group is further divided into two subgroups. The slices attribute provides information about each subgroup.blockYou can run the following code to implement this grouping strategy:
      import whale as wh
      cluster = wh.cluster(layout={"block" : "3,1"})
      The system returns a cluster.slices object similar to the following example:
      [
          [ 
              [
                  Worker0:GPU0, 
                  Worker0:GPU1, 
                  Worker0:GPU2
              ],
              [
                  Worker1:GPU0, 
                  Worker1:GPU1, 
                  Worker1:GPU2
              ]
          ],
          [
              [Worker0:GPU3], 
              [Worker1:GPU3] 
          ]
      ]
    • row

      Arrange GPUs by row for each worker.

      The row strategy horizontally groups resources, as shown in the following figure.rowYou can run the following code to implement this grouping strategy:
      import whale as wh
      cluster = wh.cluster(layout = {"row" : 4 })
      The system returns a cluster.slices object similar to the following example:
      [
          [ 
              [Worker0:GPU0], 
              [Worker0:GPU1], 
              [Worker0:GPU2], 
              [Worker0:GPU3]
          ],
          [
              [Worker1:GPU0], 
              [Worker1:GPU1],
              [Worker1:GPU2], 
              [Worker1:GPU3] 
          ]
      ]
    • column

      Arrange GPUs by row for each worker and vertically group the GPUs.

      The following figure shows the resource grouping result of the column strategy.columnYou can run the following code to implement this grouping strategy:
      import whale as wh
      cluster = wh.cluster(layout = {"column" : 2 })
      The system returns a cluster.slices object similar to the following example:
      [
          [ 
              [Worker0:GPU0], 
              [Worker1:GPU0]
          ],
          [
              [Worker0:GPU1], 
              [Worker1:GPU1]
          ],
          [
              [Worker0:GPU2], 
              [Worker1:GPU2]
          ],
          [
              [Worker0:GPU3], 
              [Worker1:GPU3] 
          ]
      ]
      If more than two workers are involved, you can group the resources by using the column strategy. The following figure shows the resource grouping result when the layout={"column" : 2} setting is used.Multi-workerThe system returns a cluster.slices object similar to the following example:
      [
          [ 
              [Worker0:GPU0], 
              [Worker1:GPU0]
          ],
          [
              [Worker2:GPU0],
              [Worker3:GPU0]
          ],
          [
              [Worker0:GPU1], 
              [Worker1:GPU1]
          ],
          [
              [Worker2:GPU1],
              [Worker3:GPU1]
          ],
          [
              [Worker0:GPU2], 
              [Worker1:GPU2]
          ],
          [
              [Worker2:GPU2],
              [Worker3:GPU2]
          ],
          [
              [Worker0:GPU3], 
              [Worker1:GPU3]
          ],
          [
              [Worker2:GPU3], 
              [Worker3:GPU3],
          ]
      ]
    • specific

      Group resources by configuring specific device information.

      Arrange GPUs by row for each worker, as shown in the following figure.Arrange by rowYou can run the following code to implement this grouping strategy:
      import whale as wh
      cluster = wh.cluster(
          layout = {'specific' : [ [['worker:0/gpu:0'], ['worker:0/gpu:1']], [['worker:1/gpu:0'], ['worker:1/gpu:1']] ] } )
      The system returns a cluster.slices object similar to the following example:
      [
          [ 
              [Worker0:GPU0], 
              [Worker0:GPU1]
          ],
          [
              [Worker1:GPU0],
              [Worker1:GPU1]
          ],
      ]
      The following figure shows the resource grouping result of the specific strategy.Resource grouping result

Return value

After the grouping strategy specified by the layout parameter is implemented, a whale.Cluster object is returned, which contains the following attributes:
  • rank
    You can use the following syntax to reference this attribute:
    cluster.rank
    Returns the ranking of the current worker.
  • worker_num
    You can use the following syntax to reference this attribute:
    cluster.worker_num
    Returns the number of workers.
  • gpu_num_per_worker
    You can use the following syntax to reference this attribute:
    cluster.gpu_num_per_worker
    Returns the number of GPUs per worker.
  • slices
    You can use the following syntax to reference this attribute:
    cluster.slices
    Returns information about the groups that are divided by using the grouping strategy specified by the layout parameter. The value is a nested list.
  • total_gpu_num
    You can use the following syntax to reference this attribute:
    cluster.total_gpu_num
    Returns the total number of GPUs.

Examples

You can use one of the following approaches to map different parts of a model to the virtual device slices that are grouped by using the whale.cluster operation:
  • Automatic mapping based on the with keyword
    Add the with keyword at the beginning of the model definition. This way, the Whale framework automatically maps different parts of the model to virtual device slices. The following code provides an example on how this approach works:
    import whale as wh
    with wh.cluster(layout = {"row" : 4 }):
        # Assume that resources are grouped into two slices.
        # [
        #    [ [Worker0:GPU0], [Worker0:GPU1], [Worker0:GPU2], [Worker0:GPU3] ],
        #    [ [Worker1:GPU0], [Worker1:GPU1], [Worker1:GPU2], [Worker1:GPU3] ]
        # ]
        with wh.stage():
            model_part_1()
        with wh.stage():
            model_part_2()
  • Manual mapping based on the slice information
    After the whale.cluster operation is called, resources are grouped based on the strategy specified by the layout parameter. Then, you can reference the groups by using the cluster.slices method. Whale allows you to configure the device information when you specify scopes. Therefore, you can use the cluster.slices method to manually map different parts of the model to virtual device slices. The following code provides an example on how this approach works:
    import whale as wh
    cluster = wh.cluster(layout = {"row" : 4 })
    # Assume that resources are grouped into two slices.
    # [
    #    [ [Worker0:GPU0], [Worker0:GPU1], [Worker0:GPU2], [Worker0:GPU3] ],
    #    [ [Worker1:GPU0], [Worker1:GPU1], [Worker1:GPU2], [Worker1:GPU3] ]
    # ]
    with wh.stage(cluster.slices[0]):
        model_part_1()
    with wh.stage(cluster.slices[1]):
        model_part_2()
    The manual mapping approach allows you to configure the device information for each scope more flexibly. For example, you can map multiple scopes to the same slice.
Note Do not mix the use of the with keyword in automatic mapping and manual mapping.

Workers in Whale

Based on the definition of the whale.cluster operation, the resources of all available workers are grouped based on the strategy specified by the layout parameter to generate a cluster.slices object. The cluster.slices object is a nested list. In the Whale framework, the workers corresponding to all devices in the slices[0] group are called master workers. These workers are used to construct the original graph and rewrite the graph for parallel training. The rest of the workers do not construct graphs. Instead, they directly call the server.join method and wait for the master workers to allocate tasks. These workers are called slave workers.

Assume that three servers are available and two GPUs are available for each server. If the layout=wh.cluster(layout={"block" : "3,3" }) setting is used when you call the whale.cluster operation, the system returns a cluster.slices object similar to the following example:
[
    [
        ["/job:worker/replica:0/worker:0/device:GPU:0"],
        ["/job:worker/replica:0/worker:0/device:GPU:1"],
        ["/job:worker/replica:0/worker:1/device:GPU:0"]
    ],
    [
        ["/job:worker/replica:0/worker:1/device:GPU:1"],
        ["/job:worker/replica:0/worker:2/device:GPU:0"],
        ["/job:worker/replica:0/worker:2/device:GPU:1"]
    ]
]
In this example, the cluster.slices[0] method returns the following device information:
[
  ["/job:worker/replica:0/worker:0/device:GPU:0"],
  ["/job:worker/replica:0/worker:0/device:GPU:1"],
  ["/job:worker/replica:0/worker:1/device:GPU:0"]
]
Therefore, the workers whose value of task_index is 0 or 1 are master workers, and the workers whose value of task_index is 2 are slave workers.