Configure Cross-Cycle Dependency Check Nodes in DataWorks - DataWorks - Alibaba Cloud - DataWorks

When standard upstream and downstream dependencies cannot meet your complex business scenarios—for example, when a daily node must wait for all hourly nodes from the previous day to complete, or when you need to combine conditions from multiple tasks with AND/OR logic—you need a more flexible tool. This topic describes how to use the dependency check node to build complex dependency checks that span scheduling cycles, workflows, and workspaces. This allows you to precisely control when your nodes run, ensuring the accuracy and timeliness of data processing.

Feature overview

The dependency check node is a flexible tool for handling complex, cross-cycle dependency scenarios. For example, when a daily node needs to wait for all hourly nodes from the previous day to succeed before it can run, or when you need to satisfy conditions from multiple nodes in different workspaces simultaneously.

Key capabilities

Cross-entity dependencies: Set up dependencies across workspaces and workflows.
Cross-cycle dependencies: Precisely specify the time range of upstream instances for cross-cycle dependencies.
Complex logical combinations: Combine multiple dependency check conditions by using AND and OR operators.
Custom check policies: Flexibly set the check interval and max duration. The node fails if the duration is exceeded.

Requirements

Edition requirements: This feature is available only in DataWorks Enterprise Edition.

Configuration UI

The dependency check node configuration has two main sections: Check dependencies and Check settings.

In the Dependency Check area, you can add multiple check items. For each check item, you must select an upstream node to depend on and set a time period. Multiple check items can be combined by using AND/OR logical operators. The check settings area contains two parameters: check interval (default: 1 minute/check) and max duration (default: 60 minutes).

Configure a dependency check node

The core of configuring a dependency check node is building a dependency-checking logical expression. You can define multiple check items and check groups and combine them by using AND/OR relationships. Each check item points to one or more instances of an upstream node within a specific time range. When the dependency check node runs, it evaluates the entire logical expression and returns a boolean value (True/False). This result determines whether the downstream node runs or remains blocked.

Configure check items

You can set up multiple check groups in a single dependency check node. Each group can contain multiple Check item, which allows you to build a complex dependency network.

Add check groups and check items:
- Click + Add Item to create a new dependency check condition.
- Multiple check items can be organized within a check group.

Configure the content of a check item:

Task: Search for and select the node to depend on by its Output Name, Name, or ID.
- You can select nodes from the same workflow, a different workflow in the current workspace, or another workspace to which you have permissions.
  
  Important
  For simple cross-workspace or cross-workflow dependencies, see Configure cross-workflow and cross-workspace scheduling dependencies.
- If you select a node that has not been published to the production environment, the system displays a warning: The selected node is not published to the production environment. The check may fail at runtime.

Period: Determines which historical instance of the upstream node to check at runtime. Select a period type based on your needs. The following table provides more details.

Note

Conditions within a single check item have an AND relationship. For example, if you select all hourly nodes for a given day, all of them must have succeeded for the item to return True.

Period type	Options	Description
Minute	`Previous X minutes` (X is 0-59)	Checks the instance from `X` minutes before the current time. `Current minute` corresponds to `Previous 0 minutes`.
Hour	`Previous X hours` (X is 0-23)	Checks the instance from `X` hours before the current time. `Current hour` corresponds to `Previous 0 hours`.
Day	`Previous X days` (X is 0-7)	Checks the instances from `X` calendar days ago. `Today` corresponds to `Previous 0 days`.
Week	`Monday to Sunday of this week`, `Monday to Sunday of last week`, `Last Monday`, `Last Tuesday`... `Last Sunday`.	Checks instances within the specified week range.
Month	`First day to last day of this month`, `First day of this month`, `Last day of last month`, and more.	Checks instances within the specified month range.

Set logical relationships (AND/OR):
- On the left side of the UI, you can set the logical relationship between check groups and between check items within a group.
  - AND: All associated check items or check groups must succeed.
  - OR: At least one of the associated check items or check groups must succeed.

Configure check settings

At the bottom of the page, you can define the node's checking behavior:

Check interval: Sets how often to check the status of all dependencies. The minimum value is 1 minute/check.
Max duration: Sets the maximum amount of time the node will wait. If the dependencies are not met after this duration, the dependency check node itself fails. The maximum value is 1440 minutes (24 hours).

Runtime logic and statuses

Running: Within the Max duration, if any dependency condition is not met, the instance remains in the Running state and retries periodically according to the check interval.
Succeeded: Within the Max duration, if all dependency conditions are met according to your configured AND/OR logic, the node status changes to Running Succeeded.
Failed:
1. The Max duration is exceeded, and the dependency conditions are still not met.
2. During the check, if a required dependency (such as a dependency in an AND relationship) points to a non-existent task, the node terminates early and its status changes to Failed.

Use case: Cross-day hourly dependencies

Let's return to the classic scenario from the introduction. Assume you have a daily summary report node (a daily node named rpt_daily_summary) that must wait for an hourly node named ods_hourly_log. Specifically, all 24 instances of the hourly node from the previous day (00:00 to 23:59) must succeed before the daily node can start.

Add a dependency check node
In your workflow, add a dependency check node upstream of your daily node rpt_daily_summary and connect them.
Configure the check item
Double-click the dependency check node to configure its dependency check items.
- Add the dependent task: Click + Add Item. In the Task input box, search for and select the hourly node to depend on, ods_hourly_log.
- Define the time range: Set the Period by selecting Day and then selecting Previous 1 day.
  
  The system will check whether all instances (24 in total) of this hourly node from the previous calendar day have succeeded.
- Set the logical relationship: Because there is only one check item, you can leave the logical relationship for the check group and its items as the default AND.
Configure the check policy
- Check interval: For example, set this to 5 minutes/check. This means if the dependency conditions are not met, the system waits 5 minutes before checking again.
- Max duration: For example, set this to 180 minutes. This means the node will wait for a maximum of 3 hours from the daily node's scheduled start time. If not all of the hourly nodes from the previous day have completed after 3 hours, the dependency check node will fail, and the downstream daily node will not start. This prevents indefinite waiting and helps expose issues promptly.

After you complete the configuration, save and publish it. Now, your daily report node will wait until all the necessary hourly data is ready before it runs.

Appendix: Differences from scheduling dependencies

Attribute	Dependency check	Scheduling dependency
Trigger method	The dependency check node actively polls the status of check items. The nodes referenced by check items do not trigger or modify the status of the dependency check node.	When the target node reaches its scheduled run time, it checks if the upstream scheduling dependency node has succeeded.
DAG visualization	The dependency relationship is defined in the node's configuration and does not appear on the DAG visualization.	The DAG visualization displays the dependency as an upstream-to-downstream connection line.
Backfill and rerun behavior	Rerunning or backfilling a node in a check item does not automatically trigger the target node.	When you backfill a scheduling dependency node, you can choose to trigger the target node.