DataWorks:Logic of for-each nodes

The following table describes the usage notes of for-each nodes.

A for-each node is a type of special node that contains inner nodes. When you create a for-each node, the following three inner nodes are automatically created: the start node (loop start node), the Shell node (loop task node), and the end node (loop end node). The inner nodes are organized into an inner node workflow to loop through the output of an assignment node. The preceding figure shows the following information:

• Shell node

  DataWorks automatically creates an inner Shell task node. You can delete the default Shell node and configure an inner loop task node based on your business requirements.

  • If you use a Shell node as an inner loop task node, double-click the Shell node and edit the node code on the node configuration tab.

  • If your loop task is complicated, you can create more inner nodes in the inner workflow for the for-each node to process the loop task and connect the inner nodes based on your business requirements.

    Note

    When you customize inner loop task nodes of a for-each node, you can delete the dependencies between the existing inner nodes of the for-each node, and configure an inner workflow for the for-each node based on your business requirements. When you configure an inner workflow for a for-each node, make sure that the inner workflow starts with the start node and ends with the end node.

• start and end nodes

  The start node marks the startup of a loop, and the end node marks the end of a loop. The two nodes are not used to process a loop task.

  Note

  The number of loops for a for-each node is determined by the output of an assignment node rather than the end node of the for-each node. The assignment node is configured as the ancestor node of the for-each node.

The number of loops for a for-each node is determined by the result set passed by an assignment node:

• If the assignment node that you configure as the ancestor node of the for-each node uses Shell or Python, the number of loops for the for-each node is determined by the generated one-dimensional array. The number of loops is equal to the number of elements in the one-dimensional array. The elements are separated by commas (,).

  For example, if the assignment node uses Shell or Python (Python 2), the output of the assignment node is a one-dimensional array such as 2021-03-28,2021-03-29,2021-03-30,2021-03-31,2021-04-01. The for-each node loops through the output of the assignment node for five times.

• If you use an SQL node as an inner loop task node of the for-each node, the number of loops for the for-each node is determined by the generated two-dimensional array. The number of loops is equal to the number of rows in the two-dimensional array.

  For example, if ODPS SQL is used by the assignment node, the output of the assignment node is a two-dimensional array:

  +----------------------------------------------+
  | uid            | region | age_range | zodiac |
  +----------------------------------------------+
  | 0016359810821  | Hubei Province | 30 to 40 years old   | Cancer |
  | 0016359814159  | Unknown   | 30 to 40 years old   | Cancer |
  +----------------------------------------------+

  The output indicates that the for-each node loops through the output of the assignment node for twice.

You can use the built-in variables provided by a for-each node in DataWorks to obtain the number of loops that are finished and the offset between the current loop and the first loop when the for-each node is used to loop through the output of the assignment node.

data=[]

for(int i=0;i<data.length;i++) {
   print(data[i]);
}

If you are familiar with the schema of the output table, you can also use the variables in the following table to obtain the related values in each loop.