Automate Batch Iteration Using For-Each Nodes & Assignment Outputs - DataWorks

A for-each node iterates over a result set passed by an upstream assignment node and runs an inner workflow for each entry. This topic walks through a complete example: an assignment node outputs two entries, and the for-each node prints the current loop count for each iteration.

Key concepts

Term	Description
for-each node	A node that iterates over an input dataset and runs an inner workflow for each entry
assignment node	The upstream node that produces the result set the for-each node iterates over; its built-in output parameter is `outputs`
inner node	A node inside the for-each node's internal workflow; by default, a start node, a Shell node, and an end node
loopDataArray	The for-each node's built-in input parameter that receives the dataset to iterate over
loop count	The number of the current iteration, exposed via `${dag.loopTimes}`

Prerequisites

Before you begin, ensure that you have:

A DataWorks workspace with access to DataStudio
Familiarity with for-each node logic. See Logic of for-each nodes

Limitations

Limitation	Workaround
For-each nodes cannot be tested directly in DataStudio	Use data backfill in Operation Center: select both the assignment node and the for-each node, then run them together
Running only the for-each node does not produce assignment node output	Always use data backfill and select both nodes to get the full output
The start and end nodes inside a for-each node have fixed logic	Replace only the inner Shell node; the start and end nodes cannot be edited
Unsaved changes to inner node code are not included at commit time	Save the inner Shell node manually before committing the for-each node

Built-in variables

The following variables are available inside a for-each node's inner workflow.

Variable	Description
`${dag.foreach.current}`	The current data entry being processed
`${dag.loopDataArray}`	The full input dataset
`${dag.offset}`	The zero-based index of the current iteration (offset from 1)
`${dag.loopTimes}`	The current loop count; equals `${dag.offset}` + 1

For more details and examples, see Built-in variables and Examples of variable values.

Set up the workflow

Create a workflow with an assignment node as the ancestor and a for-each node as the descendant.

Go to the DataStudio page. Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose Data Development and Governance > Data Development. On the page that appears, select the desired workspace and click Go to Data Development.
Create an auto triggered workflow.
Create a for-each node.
1. In the Scheduled Workflow pane, move the pointer over the icon and choose Create Node > General > for-each. Alternatively, right-click the workflow in the Business Flow section and choose Create Node > General > for-each.
2. In the Create Node dialog box, set the Name and Path parameters.
3. Click Confirm.
Create an assignment node.
1. Double-click the workflow to open its configuration tab. Click + Create Node and drag Assignment Node from the General section onto the canvas. For more information about assignment nodes, see Configure an assignment node.
2. In the Create Node dialog box, set the Name and Path parameters. The assignment node is placed in the current workflow by default.
3. Click Confirm.
Drag a directed line from the assignment node to the for-each node to configure the assignment node as the ancestor.

Configure the assignment node

On the workflow configuration tab, double-click the assignment node. The node's configuration tab opens.
Select SHELL from the Language drop-down list.
Enter the following code in the code editor:
```
echo 'this is name,ok';
```
Click the Properties tab in the right-side navigation pane. In the Input and Output Parameters section, check the Output Parameters table. The outputs parameter is the assignment node's built-in output parameter.
Click the icon to save the node.
Click the icon to commit the node. In the Submit dialog box, set the Change description parameter. Determine whether to enable code review based on your requirements. If your workspace is in standard mode, click Deploy in the upper-right corner after committing to deploy the node to the production environment. See Deploy nodes.
Important
Configure the Rerun and Parent Nodes parameters on the Properties tab before committing. If code review is enabled, the node can be deployed only after the code passes review. See Code review.

Configure the for-each node

Double-click the for-each node. The configuration tab opens, showing the default start, Shell, and end nodes. Replace the Shell node with another node type if needed: This example uses the default Shell node.
- To use a Shell node, configure it directly.
- To use a different node type, delete the default Shell node and create the required type.
Configure the Shell node.
1. Double-click the Shell node. The node's configuration tab opens.
2. Enter the following code in the code editor:
```
echo ${dag.loopTimes} ----Display the current number of loops.
```
  Note
  Save the Shell node manually after editing. No save reminder appears when you commit the for-each node. Unsaved changes are not included in the committed version.
Bind the assignment node's output to the for-each node's input.
1. On the for-each node's configuration tab, click the Properties tab in the right-side navigation pane.
2. In the Input and Output Parameters section, find loopDataArray in the Input Parameters table and click Change in the Actions column.
3. In the Value Source column, select the outputs parameter of the assignment node from the drop-down list, then click Save.
  Note
  Skipping this step causes an error when you commit the for-each node.
Click the icon to save the for-each node.
Click the icon to commit the for-each node. In the Commit dialog box, select the inner nodes to commit, enter a description, and click Commit. If your workspace is in standard mode, click Deploy after committing. See Deploy nodes.
Important
Configure the Rerun and Parent Nodes parameters on the Properties tab before committing.

Test the for-each node and view results

For-each nodes cannot be tested in DataStudio. Use data backfill in Operation Center to run both the assignment node and the for-each node together.

On the node configuration tab, click Operation Center in the upper-right corner.
In the left-side navigation pane, choose Cycle Task Maintenance > Cycle Task.
Find the for-each node and click DAG in the Actions column to open its directed acyclic graph (DAG). Right-click the assignment node and choose Run > Current and Descendent Nodes Retroactively. Configure the parameters in the Patch Data dialog box and click OK.
Refresh the Patch Data page. After the data backfill instance finishes, click DAG in the Actions column of the instance.
Right-click the assignment node in the DAG and select View Runtime Log to verify its output.
On the Patch Data page, right-click the for-each node in the DAG and select View Internal Nodes.
In the middle pane, click Loop 1. Right-click the Shell node in the DAG and select View Runtime Log.
The log shows the output of the Shell node for the first loop.
Repeat for Loop 2 to verify the second iteration.

What's next

Logic of for-each nodes — understand the internal execution model and built-in variable behavior
Configure an assignment node — set up the upstream node that produces the result set
Deploy nodes — deploy committed nodes to the production environment