DataWorks:Logic of for-each nodes

Composition of a for-each node

When you create a for-each node, DataWorks automatically creates three inner nodes organized into an inner workflow:

The inner workflow must always start with the start node and end with the end node. The number of loops is determined by the assignment node's output, not the end node.

Customizing the inner workflow

Delete the dependencies between existing inner nodes and wire up your own. For complex loop tasks, add multiple inner nodes and connect them as needed. If any inner node uses a branch node for logical decisions, include a merge node as well.

How the for-each node relates to its assignment node

The relationship between a for-each node and its assignment node works like a for loop in code:

# Assignment node output (the dataset)
data = [element_0, element_1, element_2, ...]

# For-each node execution (one iteration per element/row)
for i in range(len(data)):
    process(data[i])   # inner workflow runs here

The assignment node produces the dataset (data). The for-each node iterates over it, exposing built-in variables you can reference inside the inner workflow.

Relationship to the workflow

The assignment node must be configured as the ancestor node of the for-each node. After the assignment node runs and produces its output, the for-each node receives that output and begins iterating.

Shell assignment node

Assignment node output (one-dimensional array, 5 elements):

2021-03-28,2021-03-29,2021-03-30,2021-03-31,2021-04-01

Loop count: 5

Note: ${dag.foreach.current[3]} returns the element at index 3 (the fourth element) of the full array, regardless of which loop is currently running.

ODPS SQL assignment node

Assignment node output (two-dimensional array, 2 rows):

+----------------------------------------------+
| uid            | region         | age_range          | zodiac |
+----------------------------------------------+
| 0016359810821  | Hubei Province | 30 to 40 years old | Cancer |
| 0016359814159  | Unknown        | 30 to 40 years old | Cancer |
+----------------------------------------------+

Loop count: 2

Note: ${dag.foreach.current[0]} returns the value in column 0 (the uid column) of the current row. ${dag.loopDataArray[1][0]} returns the value at row 1, column 0 of the full dataset — always 0016359814159, regardless of the current loop.

Dependencies

A for-each node must depend on the assignment node. Configure the assignment node as the ancestor node of the for-each node so that the for-each node receives the assignment node's output.

Debugging

For-each nodes cannot be tested directly in DataStudio when using a standard mode workspace. To verify a for-each node's behavior:

1. Commit and deploy both the assignment node and the for-each node to Operation Center in the development environment.

2. Run both nodes together — running only the for-each node does not produce the assignment node's output.

To verify that the assignment node correctly passes its output to the for-each node, use the data backfill feature to backfill data for both nodes at the same time.

View logs

To view the operational logs of a for-each node:

1. Open the Cycle Task page in Operation Center and find the for-each node.

2. Open the directed acyclic graph (DAG) of the node.

3. Right-click the node name and select View Internal Nodes.