Isolated nodes lack dependencies on ancestor nodes, which prevents automatic scheduling and can cause downstream pipeline failures. Learn how to identify and resolve them.
What is an isolated node?
An isolated node is a node that lacks a dependency on any ancestor node. As a result, an Periodic tasks or Auto Triggered Instances with this status is not triggered by the automatic scheduling system and will not run. If an isolated node has many downstream dependencies, it can cause serious pipeline failures. DataWorks automatically sends an alert when an isolated node is detected. If you receive an isolated node alert, resolve the issue immediately. For more information about the alerting mechanism, see Monitoring and alerting for isolated nodes.
In DataWorks, every custom auto triggered node must have a dependency on an ancestor node to be scheduled and run.
Resolve isolated nodes
|
Scenario |
Description |
Solution |
|
The instance generation modes of the upstream and downstream nodes are inconsistent. |
When both the upstream and downstream nodes are new, the downstream node can become isolated if its only dependency is on an upstream node with a different instance generation mode. For example, if the upstream node is set to the T+1 (Next Day) mode and the downstream node is set to generate an instance immediately after deployment, the upstream node does not generate an instance on the current day, which isolates the downstream node. |
Adjust the upstream node's instance generation mode and redeploy it. For more information about this scenario, see Scenario 3: Inconsistent instance generation modes. |
|
The upstream node is outside its validity period. |
DataWorks does not generate instances for nodes that are outside their specified validity period. If a downstream node's only dependency is on an upstream node that is outside its validity period, the downstream node becomes isolated. |
Go to the upstream node's scheduling configuration and update its validity period. For more information, see Validity Period. |
|
The upstream node output has changed. |
Changes to an upstream node's business logic can alter its node output, breaking the downstream node's dependency. If this is the downstream node's only dependency, it becomes isolated. |
Reconfigure the dependency for the downstream node. For more information, see Impacts of deleting or changing a node output. |
|
A cross-workspace dependency exists, and scheduling is disabled in the upstream node's workspace. |
A node with a single cross-workspace dependency becomes isolated if scheduling is disabled in the upstream node's workspace. |
Contact the owner of the upstream node's workspace to enable scheduling, or remove the cross-workspace dependency. For more information about how to enable scheduling, see Scheduling settings. |
|
An intermediate task is outside the specified time range for data backfill. |
When you perform a data backfill, if an intermediate task in the scheduling dependency chain falls outside the backfill time range, downstream nodes that depend on this task may become isolated. An intermediate task has both upstream and downstream dependencies. Example:
If you backfill data for nodes A and C for the time range |
Run a data backfill for the intermediate task within the same time range. For more information, see Backfill data. In this example, you must also perform a data backfill for Node B for the time range |
Monitoring and alerting
Every night, DataWorks generates the next day's Auto Triggered Instances based on the configured Periodic tasks. DataWorks uses built-in alert rules to periodically scan auto triggered nodes for issues such as isolated nodes. When an issue is detected, an alert is triggered automatically. Handle these alerts immediately to prevent pipeline failures.
-
DataWorks scans the status of auto triggered nodes at 09:00, 12:00, and 16:00 every day. If an issue occurs within 10 minutes before a scan, it will not be detected until the next scan.
-
The isolated node alert is a built-in system rule, so you do not need to create it manually. By default, DataWorks sends alerts to the node owner by SMS and email. You can change the alert recipients for this global rule on the Rule Management page. For more information, see Rule Management.
Locate isolated nodes
Go to the Operation Center and open the Overview page. The Highlights section displays the number and details of isolated nodes.