When you configure scheduling dependencies, you link nodes based on their output names. Learn how to configure the inputs and outputs that control node scheduling.
Configure node inputs
You can configure node inputs in two ways:
-
Use the automatic parsing feature to derive dependencies from your code.
-
Manually add a dependency by entering the Output Name of Current Node of a parent node.
On the Scheduling panel, set Auto Parse to Yes. The parsed dependencies then appear in the Parent Nodes table.
When you manually add a parent node, you must provide its Output Name of Current Node. If the parent node's name differs from its output, you must use the correct Output Name of Current Node.
To validate a dependency, check that a value appears in the Parent Node ID column for the parsed parent node.
A dependency creates a relationship between two nodes. You can only set dependencies for existing nodes.
Invalid parent node dependencies
An invalid parent node dependency typically has one of two causes:
-
The parent node does not exist. In the Parent Nodes table of the Scheduling panel, the Parent Node ID column displays a hyphen (
-) to indicate this. -
The parent node output does not exist. Attempting to commit the node triggers an error message stating that the parent node output is missing, which blocks the commit. You must ensure that the parent node with the specified output has been committed. A pre-commit validation warning might also appear, indicating that the parent node's status is invalid. You can click Go to Scheduling to check the dependency configuration.
An invalid dependency typically occurs when no node generates the upstream table, or when the node that generates the table has an incorrect output configured.
You can resolve this issue in two ways:
-
Confirm whether a node generates the required table.
-
Find the node that generates the table, identify its node output, and then manually enter the node output in the Parent Nodes section.
When you manually add a parent node, you must provide its Output Name of Current Node. If the parent node's name differs from its output, you must use the correct Output Name of Current Node.
For example, if parent node A has a node output of A1 and descendant node B depends on node A, you must enter A1 in the input field for parent nodes and click the Add button to add the dependency.
Configure parent node dependencies
If your table does not have any parent nodes, you can add a parent node by clicking Add Root Node. In the Parent Nodes section of the Scheduling configuration page, select node output from the drop-down list, enter the output name or output table name of the parent node, and then click Add. Select Use workspace root node to add the workspace root node as a parent node. In the cross-cycle dependency (formerly Previous Cycle) section, you can select This Node, first-level child node, or other nodes as needed.
Configure the node output
For efficient configuration, use the same name for the node, its node output, and its output table.
-
Quickly identify which table the node operates on.
-
Quickly determine the impact of a node failure.
-
Improve the accuracy of automatic parsing.
Automatic parsing
Auto Parse: Automatically parses scheduling dependencies from your code.
How it works: DataWorks reads table names from the code and then identifies the nodes that produce those tables.
For example, consider this node code:
INSERT OVERWRITE TABLE pm_table_a SELECT * FROM project_b_name.pm_table_b ;The parsed dependencies are as follows:
-
pm_table_a: Is automatically parsed as the output of the current node. -
project_b_name.pm_table_b: Is automatically parsed as the output of a parent node.
DataWorks determines that the current node depends on a node in the project_b_name workspace that produces pm_table_b, and that the current node produces pm_table_a. Therefore, the parent node output is project_b_name.pm_table_b, and this node's output is test_pm_01.pm_table_a because the current workspace is named test_pm_01.
-
To ignore the parsed dependencies, select No.
-
If the code contains temporary tables, such as tables whose names start with t_, they are not parsed as scheduling dependencies. You can define the prefix for temporary tables in your Workspace Settings.
-
If a table in the code is both an output table and a referenced input table, it is parsed only as an output table.
-
If a table is referenced or produced multiple times in the code, only one scheduling dependency is created.
By default, tables with names that begin with t_ are treated as temporary tables and are automatically parsed as such. If a table with a name that begins with t_ is not a temporary table, contact your project administrator to change this setting on the Workspace Settings page. On the Table Management tab, set Temporary table prefix to t_, and click Save Configuration.
Delete an input for a table
During data development, you may use static tables that are populated by uploading local files rather than generated by nodes.
When you configure dependencies, you may need to remove a static table's input. If a static table's name does not match the temporary table prefix, such as t_, it is not automatically ignored and you must manually remove it.
To do this, right-click the table name in the code editor and select Delete Input from the context menu.
If you upgraded from DataWorks V1.0 to DataWorks V2.0, the default node output for your migrated nodes is in the format <WorkspaceName>.<NodeName>.