Periodic scheduling can lead to special cases, such as node dependency loops and node isolation. Both can disrupt task scheduling. This topic describes these scenarios and explains how to resolve them.
Scenario 1: Node dependency loop
Causes and solutions for a node dependency loop
A node dependency loop occurs when an ancestor node also depends on one of its own descendant nodes, creating a circular dependency. You must analyze your business workflow and promptly remove the circular dependency.
For example, a descendant node may depend on table data produced by an ancestor node from the previous cycle and write the processed results back to that same table. If the descendant node is configured to depend on the ancestor node from the current cycle instead of the previous one, a node dependency loop is formed. After you clarify the business logic, reconfigure the descendant node to depend on its ancestor node's previous cycle.
-
Example: In the following scenario, Node A reads Table C and outputs data to Table A. A descendant node, Node B, cleanses the data from Table A and writes it to Table B. Subsequently, Node C cleanses data from Table B and writes it back to Table C. This process creates a dependency loop, which DataWorks automatically detects.
-
Solution: Analyze your business workflow and remove the circular dependency. In this example, if the descendant node needs to process data from the ancestor node of the previous cycle, configure a cross-cycle dependency where Node A depends on Node C from the previous cycle.
Monitoring and alerts for node dependency loops
To ensure that auto triggered tasks can generate auto triggered instances and run automatically, DataWorks provides built-in alert rules to periodically scan for issues. If a node dependency loop is detected, DataWorks automatically sends an alert. If you receive such an alert, resolve the issue promptly.
-
DataWorks scans the status of auto triggered tasks at 09:00, 12:00, 16:00, 20:00, and 22:00 every day. If DataWorks finds an issue, it automatically sends an alert. Issues that occur within 10 minutes before a scan are not included in the current scan and are detected in the next one.
-
The node dependency loop alert is a built-in alert rule and requires no manual creation. By default, DataWorks sends alerts to the node owner by SMS and email. However, you can change the alert recipients for global rules on the Rule Management page. For more information, see Rule Management.
Scenario 2: Node isolation
Node isolation
Node isolation occurs when a node has no ancestor nodes upon expanding its dependencies in Auto Triggered Task or Cycle Examples. This type of node cannot be triggered by the scheduling system and will not run. If an isolated node has many descendant nodes, the consequences can be severe. DataWorks automatically sends an alert when node isolation occurs. If you receive such an alert, resolve the issue promptly.
In DataWorks, every custom auto triggered task, except for the workspace root node, must depend on an ancestor node to be scheduled and run correctly.
Resolving node isolation
|
Scenario |
Description |
Solution |
|
Inconsistent instance generation modes between ancestor and descendant nodes. |
When both the ancestor and descendant nodes are new, if the ancestor's instance generation mode is Next Day (T+1) and the descendant's mode is Immediately After Deployment, the descendant node becomes isolated if it depends only on this ancestor. This occurs because the ancestor node does not generate an instance on the current day, leaving the descendant node without a dependency to run. |
Adjust the instance generation mode for the ancestor node and redeploy it. For more information about this scenario, see Inconsistent Instance Generation Modes Between Upstream and Downstream Tasks. |
|
The ancestor node is outside its effective time range. |
In DataWorks, nodes that are outside their effective time range do not generate instances. If a descendant node depends solely on an ancestor node that is outside its effective time range, the descendant node becomes isolated. |
Go to the ancestor node's scheduling configuration page and modify its Effective Period. |
|
The output of the ancestor node has changed. |
Changes to the business logic of an ancestor node can alter its output, which may break downstream dependencies. If a descendant node depends only on this ancestor, it becomes isolated. |
Reconfigure the node dependencies for the descendant node. |
|
Scheduling is disabled for a cross-workspace dependency. |
If a node has a cross-workspace dependency on an ancestor node, and scheduling is disabled in the ancestor node's workspace, the node becomes isolated if it has no other dependencies. |
Contact the workspace owner to enable scheduling, or remove the cross-workspace dependency. |
|
An intermediate task is outside the specified data backfill range. |
When you perform a data backfill, if an intermediate task in the dependency chain (a task that has both an ancestor and a descendant) is not included in the backfill's time range, any descendant nodes that depend on it may become isolated. Example:
If you backfill data for tasks A and C for the time range |
Include the intermediate task in the data backfill for the same time range. For more information, see Perform a data backfill. In this example, you must also backfill data for Task B for the |
Monitoring and alerts for node isolation
Every night, DataWorks generates Cycle Examples for the next day based on Auto Triggered Task. To ensure these instances are generated and run correctly, DataWorks uses built-in alert rules to scan for issues. DataWorks automatically sends an alert when node isolation is detected. If you receive such an alert, resolve the issue promptly.
-
DataWorks scans the status of auto triggered tasks at 09:00, 12:00, and 16:00 every day. If DataWorks finds an issue, it automatically sends an alert. Issues that occur within 10 minutes before a scan are not included in the current scan and are detected in the next one.
-
The node isolation alert is a built-in alert rule and requires no manual creation. By default, DataWorks sends alerts to the node owner by SMS and email. However, you can change the alert recipients for global rules on the Rule Management page. For more information, see Rule Management.
Locate isolated nodes
Go to Operation and Maintenance Center. On the O&M Dashboard page, check the Focus On section to view the number and details of isolated nodes.