Dataphin:Configure offline task dependencies

Dataphin uses scheduling dependency configurations to run nodes in your business process in a specific order. This ensures that your business data is generated correctly and on time. This topic describes the process and key principles for configuring scheduling dependencies.

Background

A scheduling dependency defines the upstream-downstream relationship between nodes. In Dataphin, a downstream node runs only after its upstream nodes have run successfully. By configuring these dependencies, you ensure that scheduled tasks access the correct data. When an upstream node succeeds, Dataphin makes its output data available for the downstream node to retrieve. This prevents the downstream node from trying to access data before it is generated.

Procedure

1. In the top navigation bar of the Dataphin homepage, choose Develop > Data Development.

2. On the Develop page, select a project from the top navigation bar. If you are using the Dev-Prod model, you must also select an environment.

3. In the left-side navigation pane, choose Data Processing > Compute Task.

4. In the compute task list, click the target compute task to open its tab.

5. In the right sidebar, click Properties to open the Properties panel. In the Scheduling Dependency section, configure the following parameters.

   1. Upstream Dependency

      • Auto Parse

        • If the node's task type is SQL, you can click Auto Parse. Dataphin auto-parses the task code to identify upstream tasks and output tables. After parsing, all detected dependency tables are added to the upstream dependency list. You can view, edit, or delete the parsed dependencies. For more information, see Auto-parse process.

        Note

        • If your code contains an UPDATE, DELETE, UPSERT, or MERGE INTO statement, the target table of the statement is identified as both an input and an output table. Dataphin automatically identifies the task that produces this table as an upstream dependency, so you do not need to configure it manually.

        • If an input table detected by the auto-parse feature is produced by multiple tasks, all of these tasks are added as upstream dependencies by default.

        • For all parsed dependency tables, the dependency period defaults to This Cycle (Today).

        • If your code references a project variable or does not specify a project, the system uses the production project name by default to ensure scheduling stability. For example, if the development project name is onedata_dev:

          • If your code is select * from s_order, the parsed scheduling dependency is onedata.s_order.

          • If your code is select * from ${onedata}.s_order, the parsed scheduling dependency is onedata.s_order.

          • If your code is select * from onedata.s_order, the parsed scheduling dependency is onedata.s_order.

          • If your code is select * from onedata_dev.s_order, the parsed scheduling dependency is onedata_dev.s_order.

      • Add Root Node

        If a task has no upstream dependencies, click Add Root Node to use the root node as the upstream dependency for the current task.

        Note

        Each tenant is initialized with a virtual root node whose name starts with virtual_root_node.

      • Add Previous Cycle of This Node

        This option sets a dependency where the current task instance depends on the successful completion of the previous instance of the same node (for example, from the previous day or N days prior).

      • Add Dependency

        If Auto-parse cannot resolve scheduling dependencies or the upstream dependency configuration generated by Auto-parse does not match your actual application, you can manually click +Add Dependency to add an upstream dependency for the node.

        • Add Dependency - Physical Node

          You can select one or more physical nodes from the node list. To find a specific physical table node, search by This Project, Project, Node Type, Node Name, or Output Table Name.

        • Add Dependency - Logical Table Node

          You can select one or more logical table nodes from the node list. To find a specific logical table node, search by Logical Table Type, Module, and Logical Table Name.

          If you need to depend on specific fields in a logical table instead of the entire table, click the icon in the Dependent Fields column to view and select the table's fields.

        • Add Dependency - Cross-Tenant Node

          You can select one or more cross-tenant nodes from the node list. To find a specific cross-tenant node, search by Tenant, Node Type, and Node Name.

        When you add a dependency, its Dependency Period and Dependency Policy default to recommended values. To modify these settings, click the icon in the dependency list to edit the Dependency Period and Dependency Policy for an individual dependency.

        • Dependency Period: The time range during which the upstream task instance is scheduled to run. Typically, this is the current day, which corresponds to the interval [00:00, 24:00).

        • Dependency Policy: If multiple instances can run within a single dependency period, you must specify a dependency policy. You must also select a completion status to depend on: Run Successfully, Run Failed, or Run Completed (Success or Failure). By default, the downstream task depends on the successful completion of the upstream task. For information about default policies for cross-cycle dependencies, see Appendix 2: Default policies for cross-cycle dependencies.

          Important

          If there is only one instance, you can select any policy option. To ensure compatibility with potential changes to upstream task scheduling, only relative path policies are supported.

          • Run Successfully: The downstream task can run only after the upstream task runs successfully. If the upstream task fails, the downstream task does not run.

          • Run Failed: The downstream task can run only after the upstream task ultimately fails (fails after the final auto-retry attempt). If the upstream task runs successfully, the downstream task does not run.

          • Run Completed (Success or Failure): The downstream task can run after the upstream task either runs successfully or ultimately fails (fails after the final auto-retry attempt).

   2. Output of This Node

      The system automatically generates an output name for the node you create. To add more output names, click Auto-generate Output Name.

      Important

      The system uses output names to build the scheduling dependency graph. Because the system generates these names automatically, avoid changing them manually.

6. Click OK to save the scheduling dependency configuration.

Dependency period and dependency policy

1. Click Properties for the target offline compute task, and then navigate to the Scheduling Dependency section in the Properties panel.

2. In the Scheduling Dependency section, click the icon in the Actions column for the target dependency in the Upstream Dependency list.

3. In the Edit Dependency dialog box, you can view the node name, dependency period, dependency policy, and a preview of the node dependency period.

   • Dependency Period: The time range during which the upstream task instance is scheduled to run. Typically, this is the current day, which corresponds to the interval [00:00, 24:00).

   • Dependency Policy: If multiple instances can run within a single dependency period, you must specify a dependency policy. If there is only one instance, you can select any policy option. To ensure compatibility with potential changes to upstream task scheduling, only relative path policies are supported.

   • Node Dependency Period Preview: View the instance list for the current node and the selected upstream node for a specified business date.

     Section	Description
     ① Instance list of the selected upstream node	Business Date: Determined by the Dependency Period and the selected Current Node Business Date. If the dependency period is This Cycle (Today), the business date is the same as the selected Current Node Business Date. If the dependency period is Previous Cycle (Previous Day), the business date is one day prior to the selected Current Node Business Date. If the dependency period is Previous N Days, the business date is N days prior to the selected Current Node Business Date. If the dependency period is Last 24 Hours and the instances span two business dates, the business date is displayed as a range, such as {yyyy-MM-dd ~ yyyy-MM-dd}. Instance List: Displays the total number of instances for the selected upstream node on a specific business date. If there are five or fewer instances on the business date, all instances are displayed. If there are more than five instances on the business date, you can click Expand All to view all instances. If an instance on the left (the upstream node instance list) is a dependency for the currently selected instance on the right (the current node instance list) and it is the first instance or last instance in the list on the left, then the first instance and the last instance are displayed. If an upstream instance is a dependency of the currently selected instance but is not the first or last instance, then the first instance, the dependent instance, and the last instance are displayed. Instances are listed in the format Instance n ({Scheduled Time}), where n increments from 1.
     ②Instance list of the current node	The total number of instances for the current node on the selected business date. If there are five or fewer instances on the business date, all instances are displayed. If there are more than five instances, only the first instance and the last instance are displayed. You can click Expand All to view all instances. The first instance (Instance 1) is selected by default. You can click another instance to switch the selection. Instances are listed in the format Instance n ({Scheduled Time}), where n increments from 1.
     ③Connection between the selected instance and its dependencies	When the Dependency Policy is set to First Instance, Last Instance, Nearest Subsequent Instance, or Nearest Previous Instance, a single line connects the corresponding upstream instance to the selected downstream instance. When the Dependency Policy is set to All Instances, all instances on the left (the upstream node instance list) are selected. The connection line indicates that all instances on the left are the dependencies of the selected instance on the right.

Appendix 1: Default Dependency Period and Dependency Policy

Appendix 2: Default Policies for Cross-Cycle Dependencies

In the following table, - indicates that the item is not applicable.