All Products
Search

This Product

    DataWorks:Understand the logic of the for-each node

    Document Center

    DataWorks:Understand the logic of the for-each node

    Last Updated:Apr 15, 2026

    DataWorks provides the for-each node, which you can use to loop through the result set from an assignment node and orchestrate a business flow. This topic describes the components and logic of the for-each node.

    Usage notes

    The following table describes how to use the for-each node.

    Description

    References

    Learn about the scenarios where you can use the for-each node.

    Scenarios

    Note

    This is used only to traverse the result set returned by the assignment node.

    Learn about the limits and notes for the for-each node, such as the maximum number of traversals, how to test the node, and how to view logs.

    Limits, Notes

    Learn how to customize the business flow within a for-each node. The flow must start with the built-in Start node and end with the built-in End node.

    Node components and flow orchestration

    Learn that the number of traversals for a for-each node is determined by the output of its upstream assignment node.

    Number of traversals

    Learn that the for-each node provides built-in variables to retrieve relevant values for each traversal.

    Built-in variables

    View examples of how to retrieve variable values and how the number of traversals is calculated.

    Examples of retrieving values when the upstream assignment node is a Shell or ODPS SQL node

    Scenarios

    The DataWorks for-each node is primarily used for scenarios that require loops. It must be used with an upstream assignment node. The for-each node receives the output from the assignment node and then loops through the results.

    for-each

    Limits

    • The for-each node is supported only in DataWorks Standard Edition and later. For more information, see Feature Details by Version.

    • A for-each node can loop a maximum of 1,024 times. The actual number of traversals is determined by the result set that is output by the assignment node.

    • Concurrent execution is not supported. A new loop can start only after the previous loop is complete.

    Notes

    Dimension

    Category

    Description

    Upstream and downstream dependencies

    Dependency settings

    The for-each node traverses the values passed from an assignment node. Therefore, the assignment node must be an upstream node of the for-each node.

    Traversal support

    Maximum number of traversals

    A for-each node supports a maximum of 1,024 loops. If this limit is exceeded, a runtime error occurs. The actual number of loops is controlled by the output of the upstream assignment node.

    Number of traversals

    Determined by the result set that is output by the assignment node.

    Inner nodes

    Flow orchestration

    • You can delete the dependencies between the inner nodes of a for-each node and re-orchestrate the internal business flow. However, the Start node and End node must be the first and last nodes of the internal business flow, respectively.

    • When you use a branch node inside a for-each node for logical judgments or result traversal, you must also use a merge node.

    Value retrieval

    Built-in variables are provided to retrieve specified values from the upstream assignment node.

    Debugging and running

    Task debugging

    • In standard mode, DataWorks does not support testing and running for-each nodes directly in the DataStudio interface.

      To test and verify the running results of a for-each node, you must commit and publish the task that contains the for-each node to the development environment's Operation Center. Then, run the for-each node task in the Operation Center of the development environment.

    • Execute the assignment node and the for-each node at the same time. When you perform a data backfill operation for an auto triggered task in the Operation Center, start the data backfill from the assignment node. This ensures both the assignment node and the for-each node are executed. Do not execute the for-each node separately.

    View logs

    When you view the execution log of a for-each node in the Operation Center, right-click the instance and click View Inner Nodes to view the execution logs of the inner nodes.

    Node components and flow orchestration

    The for-each node in DataWorks is a special type of node that contains inner nodes. When you create a for-each node, three inner nodes are automatically created: a Start node (loop start), a Shell node (loop task), and an End node (loop end). These inner nodes form an internal flow that loops through the output of the upstream assignment node.for-each内部节点As shown in the preceding figure:

    • Shell node

      By default, DataWorks creates a Shell inner node to run tasks. You can also delete the default Shell node and customize the node for the loop traversal task.

      • If the loop traversal task is a Shell task, you can double-click the Shell node to open the code editor and develop the task code.

      • If the loop traversal task is complex, you can create other task nodes in the internal flow and rebuild the node's running process as required.

        Note

        When you customize a loop task node, you can delete the dependencies between the inner nodes and re-orchestrate the internal flow. However, the Start node and End node must be the first and last nodes in the internal flow.

    • Start node and End node

      These are the start and end nodes for each loop in the internal flow. They do not contain any specific task code.

      Note

      The End node of a for-each node does not control the number of loops. The number of loops is determined by the output of the upstream assignment node.

    Number of traversals

    Note

    A for-each node can loop a maximum of 1,024 times. The actual number of traversals is determined by the result set that is output by the assignment node.

    The number of traversals is determined by the result set that is output by the assignment node:

    • Shell and Python traversals: The output is a one-dimensional array. The number of loops is the number of elements in the array. The elements are separated by commas.

      For example, if the assignment language of the assignment node is Shell or Python (Python 2) and the output is the one-dimensional array 2021-03-28,2021-03-29,2021-03-30,2021-03-31,2021-04-01, the for-each node loops five times.

    • SQL traversals: The output is a two-dimensional array. The number of loops is the number of rows in the array.

      For example, if the assignment language of the assignment node is ODPS SQL and the output is a two-dimensional array:

      +----------------------------------------------+
| uid            | region         | age_range       | zodiac |
+----------------------------------------------+
| 0016359810821  | Hubei Province | 30-40 years old | Cancer |
| 0016359814159  | Unknown        | 30-40 years old | Cancer |
+----------------------------------------------+

      The for-each node loops two times.

    Built-in variables

    You can use the following methods in the inner nodes of a for-each node to retrieve the result set from the upstream assignment node. If an assignment node exists in the internal flow of the for-each node, you can retrieve values using the default value retrieval method for downstream nodes. For more information about the value retrieval method, see Assignment node.

    When a DataWorks for-each node loops through the output of an assignment node, you can use built-in variables to retrieve the current loop number and the offset.

    Built-in variable

    Description

    Comparison with a for loop

    ${dag.loopDataArray}

    Gets the dataset of the assignment node.

    Equivalent to the following code in a for loop:

    data=[]

    ${dag.foreach.current}

    Gets the current traversal value.

    Take the following for loop as an example:

    for(int i=0;i<data.length;i++) {
   print(data[i]);
}

    • data[i] is equivalent to ${dag.foreach.current}.

    • i is equivalent to ${dag.offset}.

    ${dag.offset}

    The current offset (the offset of each traversal relative to the first traversal).

    ${dag.loopTimes}

    Gets the current number of traversals.

    -

    If you know the table schema of your output, you can use the following variables to retrieve other values.

    Other variable

    Description

    ${dag.foreach.current[n]}

    When the output of the upstream assignment node is a two-dimensional array, this gets the data of a specific column in the current data row during each traversal.

    ${dag.loopDataArray[i][j]}

    When the output of the upstream assignment node is a two-dimensional array, this gets the data in row i and column j of the dataset.

    ${dag.foreach.current[n]}

    When the output of the upstream assignment node is a one-dimensional array, this gets the data at a specific index.

    Examples of using built-in variables

    Example 1: The upstream assignment node is a Shell node

    • Assignment node output

      The upstream assignment node is a Shell node. The final output is 2021-03-28,2021-03-29,2021-03-30,2021-03-31,2021-04-01.

    • Value retrieval in the for-each node

      Note

      Because the output is a one-dimensional array with five comma-separated elements, the for-each node loops five times.

      Built-in variable

      Value in the 1st loop

      Value in the 2nd loop

      ${dag.loopDataArray}

      2021-03-28,2021-03-29,2021-03-30,2021-03-31,2021-04-01

      ${dag.foreach.current}

      2021-03-28

      2021-03-29

      ${dag.offset}

      0

      1

      ${dag.loopTimes}

      1

      2

      ${dag.foreach.current[3]}

      2021-03-30

    Example 2: The upstream assignment node is an ODPS SQL node

    • Output from the assignment node

      The upstream assignment node is an ODPS SQL node. The last select statement returns two records:

      +----------------------------------------------+
| uid            | region         | age_range       | zodiac |
+----------------------------------------------+
| 0016359810821  | Hubei Province | 30-40 years old | Cancer |
| 0016359814159  | Unknown        | 30-40 years old | Cancer |
+----------------------------------------------+

    • Retrieving values in the for-each node

      Note

      Because the output is a two-dimensional array with two rows, the for-each node traverses twice.

      Built-in variable

      Value in the 1st loop

      Value in the 2nd loop

      ${dag.loopDataArray}

      +----------------------------------------------+
| uid            | region         | age_range       | zodiac |
+----------------------------------------------+
| 0016359810821  | Hubei Province | 30-40 years old | Cancer |
| 0016359814159  | Unknown        | 30-40 years old | Cancer |
+----------------------------------------------+

      ${dag.foreach.current}

      0016359810821,Hubei Province,30-40 years old,Cancer

      0016359814159,Unknown,30-40 years old,Cancer

      ${dag.offset}

      0

      1

      ${dag.loopTimes}

      1

      2

      ${dag.foreach.current[0]}

      0016359810821

      0016359814159

      ${dag.loopDataArray[1][0]}

      0016359814159