DataWorks:Logic of do-while nodes

Loop state variables

These variables are always available, regardless of whether you use an assignment node.

Assignment node variables

If a do-while node depends on an assignment node, you can pass the assignment node's output parameters to the do-while node's Input Parameters. Input parameters follow the ${dag.<parameter-name>} format, where <parameter-name> matches the name you set in Input Parameters for the do-while node.

In the table below, input is a placeholder for the actual parameter name you configure.

Add the assignment node's output parameters to Output Parameters for the assignment node, then add them to Input Parameters for the inner shell node of the do-while node (not the do-while node itself). The inner shell node reads data using these variables in its code.

Variable values by assignment node type

The data format depends on the language used by the assignment node.

For more details on the output format, see Output format of the outputs parameter.

+----------------------------------------------+
| uid            | region         | age_range          | zodiac |
+----------------------------------------------+
| 0016359810821  | Hubei Province | 30 to 40 years old | Cancer |
| 0016359814159  | Unknown        | 30 to 40 years old | Cancer |
+----------------------------------------------+

ODPS SQL

Exit after 10 loops or when a table is empty:

SELECT CASE
    WHEN COUNT(1) > 0 AND ${dag.offset} <= 9
    THEN true
    ELSE false
  END
FROM xc_dpe_e2.xc_rpt_user_info_d
WHERE dt = '20200101';

The query compares the row count and the current offset against fixed values. When ${dag.offset} exceeds 9 (after 10 loops) or the table returns no rows, the end node returns false.

Shell

Exit after 5 loops:

if [ ${dag.loopTimes} -lt 5 ]; then
    echo "True"
else
    echo "False"
fi

${dag.loopTimes} is 1 on the first loop and increments by 1 each iteration. When the fifth loop finishes, ${dag.loopTimes} equals 5, the condition is no longer true, and the end node outputs False.

Python 2

Exit when all rows in the assignment node's result set have been processed:

if ${dag.loopTimes} < ${dag.input.length}:
    print True
else:
    print False

${dag.input.length} is the number of entries in the dataset. The loop continues until ${dag.loopTimes} reaches the dataset length, at which point every entry has been processed.

Iterate over a result set with an assignment node

Use an assignment node to query or compute a dataset, then use the do-while node to process each entry in a loop.

Setup:

1. Create an assignment node (for example, assign_node) that outputs the dataset.

2. Create a do-while node that depends on assign_node.

   The dependency must be on the do-while node itself, not on the inner shell node.

3. Add the output parameters of assign_node to Output Parameters for assign_node.

4. Add those same parameters to Input Parameters for the inner task node (shell node) of the do-while node.

   Configure input parameters on the inner shell node, not on the do-while node.

5. In the end node code, use ${dag.loopTimes} and ${dag.input.length} to exit when all entries are processed.

To verify that assign_node passes its output correctly, use the data backfill feature in Operation Center to backfill data for both nodes together. Running the do-while node alone skips the assignment node's output.

Combine with a branch node and merge node

When the inner workflow needs conditional logic or result traversal, add a branch node and merge node inside the do-while node.

Setup:

• Inside the do-while node, connect the branch node (branch_node) and merge node (merge_node) as part of the inner workflow.

• The branch node and merge node must always be used together inside a do-while node.