Logic of scheduling dependencies

Reasons for configuring scheduling dependencies

Scheduling dependencies define the relationships between nodes. After you configure scheduling dependencies for a node, the node is run only after the running of its ancestor node is successful.

After the running of a node is successful, the node generates the latest table data. Then, its descendant node extracts data from the node. This mechanism ensures that a node can obtain valid data from its ancestor node.

The scheduling dependencies of a node are configured on the Properties panel of the node. You must specify Parent Nodes and Outputs.

Automatic parsing

For an SQL node, the system can automatically determine the scheduling dependencies of the node based on the task code developed for it. Then, the system automatically adds the required output or dependent ancestor node to the scheduling dependencies of the node.

Principles for adding scheduling dependencies based on automatic parsing

• If the code developed for a node contains output commands such as INSERT and CREATE, the system automatically parses the commands and adds the table generated by the node to Outputs for the node.
• If the code developed for a node contains input commands such as SELECT, the system automatically parses the commands and adds the input table to Parent Nodes for the node.
• The output of a node is configured as the input of its descendant node. This way, a scheduling dependency is established between the nodes based on their data lineages.

In principle, automatic parsing results are consistent with data lineages. When you commit a node, the system checks whether the scheduling dependencies configured for the node are consistent with data lineages. If they are inconsistent, the system displays an error message. In this case, you can use one of the following methods to resolve the issue:

• If the scheduling dependencies of a node do not contain a table that is not generated by an auto triggered node, you must check whether the scheduling dependencies are correctly configured for the node.
• If the scheduling dependencies of a node contain a table that is not generated by an auto triggered node, you must manually delete the dependency for the node.

Requirements and principles for code development

Manual configuration

DataWorks allows you to manually modify Parent Nodes and Outputs for a node during the code development for the node. If the scheduling dependencies automatically generated by the system for your node do not meet your business requirements, you can manually modify the dependencies.

Scenarios

Configuration methods

• Delete a scheduling dependency in the code editor of a nodeIf the SELECT statement in the code of a node specifies a table that is not generated by an auto triggered node, you can delete the dependency for the node. Specifically, you can go to the code editor of the node, right-click the table name that you want to delete, and then click Delete Input to perform the operation. The preceding figure shows the process. You can also add a rule as a comment at the top of the code. This way, the system does not automatically parse the dependency based on the rule.
• Delete a scheduling dependency on the Properties panel of a node
  If the SELECT statement in the code of a node specifies a table that is not generated by an auto triggered node, you can manually delete the dependent ancestor node for the node. Specifically, you can go to the Properties panel of the node, set Auto Parse to No, and then manually perform the operation.

Configuration by drawing lines to connect nodes

DataWorks allows you to specify the relationships between nodes by drawing lines to connect nodes on the editing pages of workflows. After the nodes are connected, the system automatically adds scheduling dependencies for each node based on the connections.After all nodes are created, the system automatically adds an output whose name is suffixed with _out for each node. When you connect nodes by drawing lines, the system adds an output whose name is suffixed with _out to the input of each descendant node.

Scenarios

After you create a workflow, you can connect nodes by drawing lines on the editing page of the workflow to configure scheduling dependencies for each node based on your business requirements. During subsequent code development, you can add or modify scheduling dependencies for each node manually or by using the automatic parsing feature. This way, all nodes in the workflow can be configured with correct scheduling dependencies.

Cases

In this section, an example is provided to demonstrate how scheduling dependencies work.The preceding figure shows the following information:

• The table generated by a node must be added to Outputs for the node. If the code developed for the node contains output commands such as INSERT, the system automatically parses the commands and adds the table to Outputs for the node.
• The input of a node must be added to Parent Nodes for the node. If the code developed for the node contains input commands such as SELECT, the system automatically parses the commands and adds the table to Parent Nodes for the node.
• The output of a node is configured as the input of its descendant node. This way, a scheduling dependency is established between the nodes based on their data lineages.

Then, the system runs the ancestor node first based on the established scheduling dependency. After the running of the ancestor node is successful, the system begins to run its descendant node.

Instructions on configuring scheduling dependencies

For more information about how to configure scheduling dependencies in common scenarios, see Dependencies.

FAQ

• When I commit Node A, the system reports an error that the output name of the dependent ancestor node of Node A does not exist. What do I do?
• When I commit a node, the system reports an error that the input and output of the node are not consistent with the data lineage in the code developed for the node. What do I do?