Dataphin:Configure Offline Pipeline Scheduling Dependencies

Procedure

1. On the Dataphin homepage, in the top menu bar, select Development > Data Integration.

2. On the Integration page, in the top menu bar, you can select Project. If you are in Dev-Prod mode, you can select an environment.

3. In the left navigation pane, click Offline Integration , and then click the offline pipeline for which you want to configure a channel.

4. In the right navigation pane, click Property to open the Property panel.

5. In the Schedule Dependency area, configure the Schedule Dependency parameters for the integration task.

   1. Upstream Dependency

      • Automatic Parsing

        For upstream dependencies of an integration task, click Automatic Parsing. Dataphin automatically parses upstream nodes and output tables based on the integration task. After parsing, all parsed dependencies are added to the upstream dependencies list. You can view the details of parsed dependencies or edit or delete them.

        Note

        • If an input table parsed automatically has multiple output tasks, all output tasks are used as upstream dependencies by default.

        • For all parsed dependency tables, the dependency cycle defaults to Current Cycle.

      • Add Root Vertex

        If a task has no corresponding upstream dependency, click Add Root Vertex to use it as the upstream dependency for the current task.

        Note

        Each tenant or enterprise has a virtual root vertex starting with virtual_root_node upon initialization.

      • Add Previous Cycle of This Node

        This means the scheduling of this node's task depends on the successful execution of the node's previous cycle (1 day or n hours ago).

      • Add Dependency

        If Automatic Parsing cannot parse scheduling dependencies or the upstream dependency configuration generated by Automatic Parsing does not reflect the actual application, click + Add Dependency to manually add the node's upstream dependencies.

        Important

        When adding dependencies, the system automatically uses recommended optimal settings for Dependency Cycle and Dependency Policy. To modify these, click the dependency list to edit the Dependency Cycle and Dependency Policy for individual dependencies.

        • Dependency Cycle: The scheduled runtime (start time) range of the upstream task instance. This is usually the current day, meaning the range is [00:00~24:00) for the current day.

        • Dependency Policy: If multiple instances exist within a dependency cycle, specify a dependency policy. If only one instance exists, set the dependency policy to any option. To ensure compatibility with potential changes to upstream task scheduling settings, only relative path policies are supported.

        For the default cross-cycle dependency policy, see Appendix: Default cross-cycle dependency policy.

        • Add Dependency - Physical Node

          Select one or more physical nodes from the node list. Search and filter the required physical table nodes based on keywords such as This Project, Project, Node Type, Node Name, or Output Table Name.

        • Add Dependency - Logical Table Node

          Select one or more logical table nodes from the node list. Search and filter the required logical table nodes based on Logical Table Type, Owning Module, and Logical Table Name.

          To depend on certain fields in a logical table instead of the entire logical table, click the icon in the Dependency Fields column of the node list to view the table fields available in the logical table, and then select the fields that meet your requirements.

        • Add Dependency - Cross-Tenant Node

          Select one or more cross-tenant nodes from the node list. Search and filter the required cross-tenant nodes based on keywords such as Tenant, Node Type, and Node Name.

   2. Node Outputs

      The system automatically generates output names for the nodes you create. To add multiple output names, click Auto Generate Output Name.

      Important

      The system uses output names to build the scheduling dependency graph. The system automatically generates output names. Manual intervention is not recommended.

6. Click OK to complete the scheduling dependency configuration.

Appendix: Default Policies for Cross-Cycle Dependencies