DataWorks:Configure a do-while node

Limitations

• The do-while node is available only in DataWorks Standard Edition and later editions. For more information, see Features of DataWorks editions.

• The maximum number of loops for a do-while node is 1,024.

• Concurrent execution is not supported. A new loop can start only after the previous one is complete.

Create a do-while node

1. Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.

2. Create a do-while node.

   1. On the Data Studio page, move the pointer over the icon and choose Create Node > General > do-while.

      Alternatively, you can open a workflow, right-click General, and select Create Node > do-while.

   2. In the Create Node dialog box, specify the node name and path.

   3. Click OK.

Examples

Edit the node code

1. Double-click the Shell node to open its editing page.

2. Enter the following code.

   echo ${dag.loopTimes} ----Print the loop count.

   • The ${dag.loopTimes} variable is a system reserved variable that represents the current loop count, starting from 1. Internal nodes of a do-while node can directly reference this variable. For more information about built-in variables, see Built-in variables and Context-dependent variables.

   • Make sure to save the code after you modify it in the Shell node. No prompt is displayed when you submit the node. If you do not save the code, the latest changes are not applied.

Define the loop exit condition

Define the exit condition so that the loop exits on the 5th iteration.

1. Double-click the End node to open its editing page.

2. In the Select Language drop-down list, select Python.

3. Enter the following code to define the exit condition of the do-while node.

   if ${dag.loopTimes}<5: 
    print True; 
   else: 
    print False;

   • The ${dag.loopTimes} variable is a system reserved variable that represents the current loop count, starting from 1. Internal nodes of a do-while node can directly reference this variable. For more information about built-in variables, see Built-in variables and Context-dependent variables.

   • The code compares dag.loopTimes with 5 to limit the total number of loop iterations. In the first iteration, dag.loopTimes is 1, in the second it is 2, and so on until the fifth iteration when it is 5. At that point, the expression ${dag.loopTimes}<5 evaluates to False, and the loop exits.

Submit the do-while node

1. Click the icon in the toolbar to save the node.

2. Click the icon in the toolbar to submit the node.

   When you submit the node, enter a Change Description in the Submission dialog box and specify whether to initiate a code review after the node is submitted.

   Important

   • You must configure the Rerun attribute and Parent Nodes before you can submit the node.

   • Code review helps ensure the quality of task code and prevents errors caused by deploying unreviewed code directly to the production environment. If code review is enabled, the submitted node code must be approved by reviewers before it can be deployed. For more information, see Code review.

   If you use a workspace in standard mode, you must click Deploy in the upper-right corner of the node editing page to deploy the task to the production environment after it is submitted. For more information, see Deploy nodes.

Test and view execution logs

The submission and deployment process for a do-while node is the same as that for a regular node. The online execution process is also the same, but testing from the Data Studio interface is not supported.

1. Click O&M Personnel in the upper-right corner of the page to go to Operation Center.

2. In the left-side navigation pane, click Auto Triggered Task O&M > Periodic tasks.

3. Select the desired node. In the DAG on the right side, right-click the assignment node and select Retroactive Instance > Current and Descendant Nodes Retroactively.

4. Refresh the Data Backfill Instance page. After the backfill instance runs successfully, click DAG next to the instance.

5. View the execution logs of the do-while node.

   1. Right-click the do-while node and select View Inner Nodes.

      For composite nodes such as do-while nodes, you must view the inner nodes to see the detailed execution logs. In this example, the loop ran 5 times and all iterations completed successfully.

      The internal loop body of a do-while node consists of the following three parts:

      • The left side of the view shows the rerun history list of the do-while node. Each time the do-while instance runs as a whole, a corresponding record is added to the history list.

      • The middle section shows the loop record list, which displays the total number of loop iterations and the status of each iteration.

      • The right side of the view shows the details of each iteration. Click an iteration in the loop record list to display the running status of each instance in that iteration.

   2. On the inner nodes page, click an iteration number on the left, right-click the desired node, and select View Runtime Log.

   3. View the detailed execution logs of the Nth iteration.

      On the inner nodes page, click Iteration 5 on the left to view the logs of the Shell node in the 5th iteration.

      The runtime log shows that in the 5th iteration, the End node executed the evaluation code if 5 < 5: print True else: print False, and the output was False (Output Result: False). The loop terminated and the task completed normally.

      As shown in this example, the workflow of a do-while node is as follows:

      1. The execution starts from the Start node.

      2. Tasks are executed sequentially based on the defined dependency relationships.

      3. The exit condition is defined in the End node.

      4. After a set of tasks completes, the exit condition statement in the End node is executed.

      5. If the evaluation statement in the End node prints True in the logs, the loop continues from step 1.

      6. If the evaluation statement in the End node prints False in the logs, the entire loop exits and the do-while node completes.

Summary

• The following table compares the do-while node with the while, for-each, and do-while loop types:

  • The do-while node supports the do...while loop pattern, which executes the loop body first and then evaluates the condition. It can also indirectly implement the foreach pattern by using the system variable dag.offset in combination with node context.

  • The do-while node does not support the while loop pattern, which evaluates the condition before executing the loop body.

• Execution flow of a do-while node:

  1. Starting from the Start node, tasks in the loop body are executed sequentially based on the defined dependency relationships.

  2. The code defined by the user in the End node is executed.

     • If the End node outputs True, the next iteration begins.

     • If the End node outputs False, the loop terminates.

• Using context dependencies: Internal nodes of a do-while node can reference the node context defined for the do-while node by using ${dag.context variable name}.

• System parameters: DataWorks automatically provides two system variables for internal nodes of a do-while node.

  • dag.loopTimes: The loop count starting from 1.

  • dag.offset: The offset of the current iteration relative to the first iteration, starting from 0.