Dataphin:Configure scheduling dependencies for an offline pipeline

Procedure

1. On the Dataphin homepage, choose Develop > Data Integration from the top navigation bar.

2. From the top menu bar on the integration page, select a Project. (In Dev-Prod mode, you also need to select an environment.)

3. In the navigation pane on the left, click Offline Integration, and then click the offline pipeline that you want to configure.

4. In the right navigation bar, click Properties to open the Properties panel.

5. In the Scheduling dependency area, configure the scheduling dependency parameters for the integration task.

   1. Upstream Dependencies

      • Automatic Parsing

        Click Automatic Parsing to let Dataphin parse the integration task and retrieve upstream tasks and output tables. All retrieved dependency tables are added to the upstream dependency list. You can view, edit, or delete the parsed dependency tables.

        Note

        • If an input table has multiple source tasks, all source tasks are added as upstream dependencies by default.

        • The dependency period for all parsed dependencies defaults to the current cycle.

      • Add Root Node

        If a task lacks upstream dependencies, click Add Root Node to set a root node as its dependency.

        Note

        Each tenant or enterprise is initialized with a virtual root node whose name starts with virtual_root_node.

      • Add Previous Cycle of This Node

        Creates a self-dependency where the current task instance depends on the successful completion of the previous instance (for example, from the previous day or hour).

      • Add Dependency

        If Automatic Parsing cannot parse the scheduling dependencies, or the parsed result does not match your needs, click +Add Dependency to manually add upstream dependencies.

        • Add Dependency-Physical Node

          Select one or more physical nodes from the node list. You can filter this list by Current Project, Project, Node Type, Node Name, or Output Table Name.

        • Add Dependency-Logical Table Node

          Select one or more logical table nodes from the node list. You can filter this list by Logical Table Type, Business Category, and Logical Table Name.

          To depend on specific fields instead of the entire logical table, click the icon in the Dependent Fields column to select the required fields.

        • Add Dependency-Cross-tenant Node

          Select one or more cross-tenant nodes from the node list. You can filter this list by Tenant, Node Type, or Node Name.

        When you add a dependency, recommended settings for Dependency Period and Dependency Policy are applied automatically. To modify these settings for a specific dependency, click the edit icon in the dependency list.

        • Dependency Period: The time range for the scheduled run of the upstream task instance. Typically, this is the current day, which corresponds to the time range [00:00, 24:00).

        • Dependency Policy: Specify a dependency policy to determine which upstream task status satisfies the dependency when multiple instances exist within a dependency period. Select Succeeded, Failed, or Finished (Succeeded or Failed). The default is Succeeded. For information about default cross-cycle dependency policies, see Appendix: Default cross-cycle dependency policies.

          Important

          If there is only one instance, the dependency policy can be set to any option. To remain compatible with possible changes to the scheduling settings of upstream tasks, only relative-path policies are supported.

          • Succeeded: The upstream task must succeed before the downstream task can run. If the upstream task fails, the downstream task does not run.

          • Failed: The upstream task must ultimately fail (fails after the final automatic retry) before the downstream task can run. If the upstream task succeeds, the downstream task does not run.

          • Finished (Succeeded or Failed): The downstream task can run after the upstream task either succeeds or ultimately fails (fails after the final automatic retry).

   2. Output of This Node

      An output name is automatically generated for the node. To add more output names, click Auto-generate Output Name.

      Important

      Do not manually change the automatically generated output names.

6. Click OK to complete the scheduling dependency configuration.

Appendix: Default cross-cycle dependency policies