, - Dataphin - Alibaba Cloud Documentation Center

Background

Periodic modeling task instances include dimension, fact, and aggregate logical table instances. Scheduling a logical table task creates a corresponding logical table instance node. You can manage logical table instances requiring maintenance in the table or field view.

Usage

Standard modeling uses dynamic materialization logic. You can perform operations and maintenance (O&M) from the following two perspectives:

From the physical O&M perspective, you may need to check the code or runtime log of the latest materialized node to determine which fields or logical table task nodes are affected by each operation. For example, to backfill a logical table task node, you must identify the materialized node it belongs to and then backfill each materialized node individually. This process is complex and cannot guarantee data consistency.
From the logical O&M perspective, you only need to select the fields or logical table instance nodes that need maintenance. The system automatically calculates the corresponding materialized nodes using the current materialization strategy. To ensure data consistency and correctness, the system also alerts you to related fields that require linked operations. You can ignore the physical implementation. This simplifies your workflow, reduces O&M costs, and improves O&M efficiency.

Scheduled modeling task instances: Access

On the Dataphin homepage, in the top menu bar, select Development > Task O&M.
In the left navigation pane, select Instance O&M > Scheduled Instances.
In the top menu bar, select the production or development environment.
On the Scheduled Instances page, click the Modeling Task tab.

Modeling task instance operations

After a recurring task generates a modeling task instance, the instance appears in the Recurring Instance > Modeling Task list. The modeling task instance list shows the instance object name, run status, scheduling cycle, business date, scheduled time, start time, end time, duration, priority, operator, project, related baseline instance, tags, and supported operations.

Instance Object: A recurring task generates an instance object each time it runs. It includes the table name and table ID, and indicates the task's scheduling method. You can click the icon next to the column name to sort by object name in ascending or descending order. For more information, see Description of recurring instance prompt markers.
Run Status: The run status of a modeling task instance consists of the field run status and the node run status. You can click the icon next to a field to view the details for each run status.

The field run status indicates the run statuses of all fields within a logical table. For example, the icon indicates that fields in the logical table have both Succeeded and Not Run statuses. The number next to the field icon is the number of fields in the current logical table node. The number next to the node icon is the number of materialized nodes for the current logical table node. You can also click the icon to view the unified run status of each field. For a description of run statuses, see Recurring instance run status description.

A field's unified run status is determined by the statuses of all materialized nodes it belongs to. The rules are as follows:
- A field's unified run status is Succeeded if any of its materialized nodes has a Succeeded status.
- A field's unified run status is Failed only if all of its materialized nodes have a Failed status.
- A field's unified run status is Running if any of its materialized nodes has a Running status.
- A field's unified run status is Waiting if some of its materialized nodes are in the Waiting status while others are in the Not Run or Failed status.
- A field's unified run status is Not Run if all of its materialized nodes have a Not Run status.
Start Time: The time when the instance starts running. Click the icon next to the column name to sort by start time in ascending or descending order.

Note
For a logical table node, the start time is the start time of the earliest-running materialized node within the instance object.
End Time: The time when the instance finishes running. Click the icon next to the column name to sort by end time in ascending or descending order.

Note
For a logical table node, the end time is the end time of the latest-running materialized node within the instance object.
Duration: The elapsed time from the start to the end of the instance run. Click the icon next to the column name to sort by duration in ascending or descending order.

Note
For a logical table node, the duration is the time difference between the start time of the earliest-running materialized node and the end time of the latest-running materialized node within the instance object.
Priority: The priority level of the instance.

Note
If the baseline feature is enabled, the system sets the priority of a baseline task to the highest priority among all its baselines, which overrides the original task priority configuration.
Project: The project to which the task belongs, shown in the format Project English Name (Project Chinese Name).
Related Baseline Instance: Shows the baseline that protects this instance as a terminal node, along with related baselines that protect this instance as an upstream node.

Note
This field does not appear if the baseline feature is not enabled.

The following table describes the operations supported for modeling task instances.

Actions	Description
DAG	Click the icon to view the DAG for the recurring instance. For more information, see Recurring instance DAG.
View Run Log	Click the icon to view the run log for the recurring instance.
Rerun	Click the icon to rerun the recurring instance. You can only rerun script instances with a Succeeded or Failed status. When you perform a rerun, the system checks whether all upstream nodes of the current instance have a Succeeded status. If this check fails, the system immediately sets the instance's status to Not Run. If your business scenario requires a rerun, you can perform a force rerun. For more information, see Force rerun the current instance.
View Gantt Chart	Note You can view the Gantt chart only after you purchase the intelligent O&M value-added service and enable the intelligent O&M module for the current tenant. Click the icon and select View Gantt Chart to open the View Gantt Chart dialog box, where you can configure the following parameters. Select node as: Select whether to use the current node as the Start Node or End Node. If you select it as the start node, the system automatically selects the current node as the start node below, and you cannot change it. The same applies if you select it as the end node. Start Node: Select the start node for the Gantt chart. You can search for nodes by keyword. Nodes in the list appear in the format `{node name}({node ID}, {Instance Scheduled Time})`. If you select the current node as the end node, the start node defaults to the tenant root node. Note Hourly and minute-level tasks have multiple nodes (one for each instance), which are distinguished by their scheduled time. End Node: Select the end node for the Gantt chart. You can search for nodes by keyword. Nodes in the list appear in the format `{node name}({node ID}, {Instance Scheduled Time})`. Note If you select a logical table as the end node, you cannot select specific fields because the logical table is treated as a single unit. For more information about Gantt charts, see View the critical path Gantt chart.
Download Upstream and Downstream Nodes	Downloads a list of the upstream and downstream nodes for the current node. The list includes all columns, even those not displayed in the UI. After you click Download Upstream and Downstream Nodes, in the Download Upstream and Downstream Nodes dialog box, you can select the levels for upstream and downstream nodes. You can choose from Level 1~Level 10 or All Levels. The default is Level 1 for both. After selecting the levels, click OK to start downloading the Excel file. The file is named `{task_name}_upstream_downstream_nodes_{timestamp}.xlsx`.
Field and Batch Operations	Click the icon and select Field and Batch Operations to perform batch O&M operations on fields within the modeling task instance. For more information, see Logical table fields and batch operations.
View Production Lineage	Click the icon and select View Production Lineage to view the materialization lineage of fields in the logical table instance and the field groups of materialized nodes. For more information, see Logical table fields and batch operations.
View Consumption Lineage	Click the icon and select View Consumption Lineage to see how the fields of a logical table instance node are used by downstream dependencies. This helps you analyze the impact on downstream nodes when you perform O&M on the fields. For more information, see Logical table fields and batch operations.
View Materialization Code	Click the icon and select View Materialization Code to view the materialization code for the logical table instance.
Recurring Task	Click the icon and select Recurring Task to view the logical table task corresponding to the logical table instance.
Edit Development Node	Click the icon and select Edit Development Node to go to the development page for this instance in the development environment, where you can modify the instance. Note This operation is supported only for script instances in projects that use the Dev-Prod mode.
View Production Node	Click the icon and select View Production Node to go to the production page for this instance in the production environment. Note This operation is supported only for script instances in projects that use the Dev-Prod mode.
Edit Node	Click the icon and select Edit Node in a Basic project to go to the edit page of the script task that generated this instance. Note This operation is supported only for script instances in projects that use the Basic mode.
Rerun Upstream and Downstream	Click the icon and select Rerun Upstream and Downstream to rerun the upstream and downstream nodes of the instance. You can only rerun instances with a Succeeded or Failed status. After you confirm the operation, the system first resets the status of the selected instance to Not Run and then reruns it based on its dependencies. Instances with a Waiting or Running status cannot be rerun, which may cause data inconsistency. For more information, see Rerun upstream and downstream. If you need to rerun the entire lineage, we recommend using force rerun upstream and downstream instances. For more information, see Force rerun upstream and downstream instances. Use the Rerun Upstream and Downstream operation in the following scenarios: After the data of the current instance node is updated, you need to synchronize the data in downstream nodes. An upstream or downstream node of an instance fails and needs to be rerun.
Set as Succeeded	Click the icon and select Set as Succeeded to manually set the run status of a Failed or Not Run script instance node to Succeeded, allowing normal scheduling to resume. For more information, see Set as succeeded.
Terminate	Click the icon and select Terminate to stop the currently running instance. After termination, the node status of the instance changes to Failed. Terminating the current instance node prevents its downstream nodes from running. Note You cannot terminate an instance whose node status is Succeeded, Failed, or Not Run. Instances with any other node status can be terminated. Use the Terminate operation in the following scenarios: An instance needs to be terminated due to a Dataphin anomaly or an operational error. The script shows that it is running, but no data is being generated. You can manually terminate the run and check the run log to troubleshoot the issue.
Force Rerun	Click the icon, select Force Rerun, and then choose either Force Rerun Current Instance or Force Rerun Upstream and Downstream Instances. Force Rerun Current Instance: A force rerun does not check the run status of upstream nodes. If the upstream nodes have not all succeeded, the current instance may fail to rerun or produce incorrect data. Proceed with caution. After you force rerun the current script instance, its data is updated. For more information, see Force rerun the current instance. Force Rerun Upstream and Downstream Instances: A force rerun does not check the status of upstream nodes. Instances in any run status can be forcibly rerun. After you confirm the operation, the system first resets the status of the selected instances to Not Run and then forcibly reruns them based on their dependencies. This prevents data corruption that could occur if a downstream node runs while an upstream node is being rerun. For a logical table node, the actual rerun scope is calculated based on the relationships of the selected fields. Only the starting point ignores upstream dependencies and scheduled run times; downstream nodes still adhere to dependency relationships and instance start conditions. A full force rerun may consume significant resources and affect other running instances. For more information, see Force rerun upstream and downstream instances. For more information about instance start conditions, see Dependency relationships and run description. Important A force rerun does not check whether all upstream nodes have succeeded or whether the current instance has reached its scheduled time. This can lead to run failures or data quality issues. Ensure that the operation does not affect downstream data before proceeding.
Remove Upstream Dependencies	Click the icon and select Remove Upstream Dependencies to remove the upstream dependencies of a script instance, regardless of its run status. Important At least one upstream instance must be retained.
Pause	Click the icon and select Pause to pause the current script instance. This does not affect other instances of the same script task. After you pause an instance, the scheduler will not run it or its downstream dependencies. Note If the instance status is Not Run, pausing the instance takes effect immediately and applies to the current scheduling cycle. For other statuses such as Waiting, Running, Succeeded, or Failed, pausing does not change the current status. It takes effect on the next rerun or in the next cycle. The Pause operation is often used when resources are constrained and you need to pause low-priority instances to allow high-priority tasks to run first.
Resume	Click the icon and select Resume to resume a Paused script instance node. After you resume the instance, the scheduler runs it and its downstream dependencies normally.
Modify Priority	Click the icon and select Modify Priority. In the Modify Priority dialog box, change the priority of the current task. You can select Highest, High, Medium, Low, or Lowest. Note An aggregate logical table can have multiple priorities.

Operations on DAG nodes

The DAG provides a clear visualization of the upstream and downstream dependencies between instance nodes and lets you perform O&M on them. By default, the DAG displays the Main node (the selected node) along with its immediate upstream and downstream nodes. You can select a modeling task instance node to perform O&M operations.

Dataphin supports O&M on cross-project instance nodes. To perform O&M on a modeling task instance node in another project, you must have view and operation permissions for the instance's project. For summary table instances, you can only operate on metrics for which you have permissions.

DAG operations

Actions	Description
Expand parent node	Expands dependency nodes at different levels for the Main node in the DAG.
Expand child node
View task	Opens the DAG for the task that generated the current instance node. You can view task node details, upstream and downstream nodes, and perform O&M on the task node. For more information, see Manage modeling tasks.
Expand materialized node	Expands the logical table to show its internal materialized nodes. Hover over a materialized node, and then right-click it or click the icon to view its code, fields, and run log.
View operation log	Displays the operation log for the instance.

DAG node operations

Hover over a DAG node to view its name, type, scheduling cycle, O&M owner, and description. The operations for a modeling task instance DAG node are the same as those in the modeling task instance list. For more information, see Supported operations for modeling task instances.

Batch operations

Actions	Description
Rerun all failed instances	After you select multiple modeling task instances, select Rerun all failed instances, Rerun all final state instances, or Force rerun all instances at the bottom of the list. Rerun all failed instances: Reruns all failed instance nodes in a batch. Rerun all final state instances: Reruns all instance nodes that are in a final state in a batch. Force rerun all instances: Forcibly reruns the selected instance nodes and their downstream nodes that have not run in a batch.
Terminate	You can terminate instances unless their node status is Succeeded, Failed, or Not Run. The batch terminate operation is not supported if you select only instances with a node status of Succeeded, Failed, or Not Run.
Set to Succeeded and continue scheduling	You can manually set the status of Failed or Not Run instance nodes to Succeeded in a batch so that they can continue the scheduling process.
Suspend	You can use a batch operation to suspend instances in any run status. If an instance has a status of Not Run, the suspend operation takes effect immediately and applies to the current scheduling cycle. For instances with other statuses, including Waiting, Running, Succeeded, or Failed, the suspend operation does not change the current status but will take effect on the next rerun or in the next scheduling cycle.
Resume	Resumes periodic instances that are Suspended.
Modify priority	Modifies the priority of selected instances in a batch. You can choose from Highest, High, Medium, Low, or Lowest. Note Aggregate logical tables can have multiple priorities.
Download all	Downloads the data of all periodic instances (including integration, computing, and modeling task instances) to your local machine. The downloaded file is in .xlsx format and named `Dataphin_periodic_instance_{timestamp}.xlsx`. The spreadsheet contains the following columns: Instance Object, Instance ID, Run Status, scheduling cycle, business date, Priority, Owner, Project (for aggregate logical tables with multiple projects, project names are separated by commas), Scheduled Time, Start Time, End Time, Duration, Retries/Auto Retries, Related Baseline instances (baseline names are separated by commas), and scheduling resource group (this field is empty for modeling task instances).

Rerun upstream and downstream

In the Rerun Upstream and Downstream dialog box, configure the parameters.

Note

You can only rerun the upstream and downstream nodes for a modeling task if the nodes have a running state of Succeeded or Failed, and their associated fields have a normalization status of Succeeded or Failed. After you confirm the action, the system first resets the selected instances to the Not Run state and then reruns them based on their dependencies. For more information about the running state of logical tables and the normalization status of fields, see View the periodic modeling list.
If you cannot rerun upstream and downstream instances due to these status limitations, but still need to, you can force a rerun. For more information, see Forcibly Rerun the Current Instance.
Instances that are Waiting or Running cannot be rerun, as this could cause data inconsistency. If you need to rerun the entire pipeline, we recommend forcibly rerunning the upstream and downstream instances. For more information, see Forcibly Rerun Upstream and Downstream Instances.

Basic Information

You can filter fields by their running state or search for and select target fields by name.

The Prioritize required fields option highlights fields that Dataphin identifies as required for the rerun scope to ensure data quality, based on field logic and data changes in upstream nodes.

Note

To ensure data consistency and correctness, the rerun operation reruns all fields in the materialized node that contains the selected field. It also reruns all fields from other materialized nodes that contain dependent fields.

Parameter

Description

Start node

Specifies the starting point for the rerun. The following options are available:

Entire Table: All fields in the logical table instance are included in the rerun of upstream and downstream instances.
Specific Fields: Lets you select specific fields to rerun, including their upstream and downstream nodes.

You can filter fields by their running state or search for and select target fields by name.

Start mode

Defines how the start node runs. The following options are available:

Dry run: For a dry run, the instance status is set to Succeeded (simulating a normal run), but no data is processed. The execution log remains empty and no run duration is recorded.
Normal run: Schedules and runs the instance normally.

Rerun scope

Parameter

Description

Rerun scope

Specifies the scope of instances to rerun. The following options are available:

All Failed Instances: The system automatically selects and reruns all upstream and downstream instances with a running state of Failed. A list of these nodes is not displayed.
Custom configuration: Lets you specify which upstream and downstream instances to rerun. This option requires configuring the Custom Rerun Scope.

Custom Rerun Scope

Specifies the direction of nodes to rerun. This parameter is available only when Rerun scope is set to Custom configuration. For each selected direction, you must specify at least one instance from the corresponding node list.

Upstream Instances: When selected, this option displays the Upstream Node List, which lets you specify which upstream instances to rerun.
Downstream Instances: When selected, this option displays the Downstream Node List, which lets you specify which downstream instances to rerun.

In the upstream or downstream node lists, you can search for nodes by name or ID. You can also filter them by running state, O&M engineer, project, level, or use the Prioritize required fields option.

Click Submit.

After the rerun, the system updates the data in the selected upstream and downstream instance nodes.

Force rerun the current instance

In the Force Rerun Current Instance-Logical Table dialog box, configure the rerun parameters.

Force rerun scope

Description

Entire table

Reruns all fields in the logical table.

Specify fields

You can select specific fields to rerun based on your business needs. All fields in the current logical table instance support a forced rerun, regardless of their normalized run status.

You can filter fields by run status or search for them by name.

After you click OK, the Cascading Required Fields dialog box appears. Review the list of automatically included fields and click OK to confirm.

To ensure data consistency and correctness, the force rerun automatically includes all fields in the same materialized node as your selection, as well as any dependent fields from other materialized nodes.

Force rerun upstream and downstream instances

In the Force rerun upstream and downstream instances - logical table dialog box, select a configuration method and set its parameters.
Note
- The force rerun operation does not check the status of upstream nodes. You can forcibly rerun instances regardless of their running status.
- After you confirm the operation, Dataphin resets the selected instance to the Not Started status and reruns it based on its dependencies. The system then calculates the actual rerun scope for the logical table node based on the cascading dependencies of the selected fields.
- A full forced rerun can cause resource contention and affect other running instances.

You can force rerun upstream and downstream instances by using either Quick configuration or Custom configuration.

Quick configuration

Parameter	Description
Start node	In quick configuration, the start node defaults to the entire table, which includes all fields of the logical table instance.
Start node run mode	Specifies the run mode for the start node. You can select one of the following modes: Dry Run: The instance status is set to Succeeded, but the run log is empty, the execution duration is zero, and no data is processed. Normal Run: The instance is scheduled and runs normally.
Upstream and downstream forced rerun scope	The rerun scope is fixed to all upstream and downstream instances that depend on the current logical table instance. You cannot specify the rerun direction.

Custom configuration

Basic information

Parameter

Description

Start node

Specifies the start node for the rerun. Dataphin supports the following options:

Entire table: All fields in the logical table instance are included in the forced rerun of upstream and downstream instances.
Specified fields: You can select specific fields to serve as the start node.

You can filter fields by their running status, or search for and select them by name.

Start node run mode

Specifies the run mode for the start node. You can select one of the following modes:

Dry Run: The instance status is set to Succeeded, but the run log is empty, the execution duration is zero, and no data is processed.
Normal Run: The instance is scheduled and runs normally.

Upstream and downstream forced rerun scope

Parameter

Description

Upstream and downstream forced rerun scope

Specifies the scope of instances to rerun. The following options are available:

All instances: Forcibly reruns all upstream and downstream instance nodes of the start node. A specific list of nodes is not displayed.
Custom configuration: Lets you specify which upstream and downstream instances to rerun. If you select this option, you must then configure the Custom rerun scope.

Custom rerun scope

Specifies the rerun direction. You can select multiple options. You can configure this parameter only when you select Custom configuration for the Upstream and downstream forced rerun scope. For each selected direction, you must specify at least one instance in the corresponding node list. The following options are available:

Upstream instances: If you select this option, the Upstream node list is displayed, which lets you specify which upstream instances to rerun.
Downstream instances: If you select this option, the Downstream node list is displayed, which lets you specify which downstream instances to rerun.

In the node lists, if you filter by "Failed", the system selects all matching failed instances. The level defaults to 1 and shows only directly adjacent upstream instances. To expand the scope, adjust the level. You can also search for nodes by name or ID, or filter by running status, O&M engineer, project, and level.

After you click Submit, the Cascading required fields dialog box appears. Review the list and click OK.

To ensure data consistency, Dataphin automatically includes all fields that are in the same materialized node as your selected fields. It also includes any associated fields from other materialized nodes.

Fields and batch operations

The Fields and Batch Operations page for a logical table lists its fields. The information for each field includes the field name, running status, field type, start time, end time, duration, and supported operations.

Actions	Description
View production lineage	View the materialization lineage of fields and the field groups of materialized nodes within a modeling task instance. ① This area displays the fields of the current modeling task instance node and their running statuses. You can select a field to display its production lineage information on the right. ② This area displays the materialization lineage of the selected field. ③ This area displays all fields of the materialized node. ④ This area displays the materialized node for the current modeling node. If a logical table node has multiple materialized nodes, you can click this area to switch between them.
View run log	View the run log for the current instance.
View materialization code	View the materialization code for all materialized nodes in the modeling task instance. On the View Materialization Code page, you can switch between materialized nodes in area ① to view their corresponding materialization code. You can also hover over the icon in area ② to view the fields that belong to that materialized node.
View consumption lineage	View the field dependencies of a modeling task instance node to help you analyze the impact on these nodes when you manage the field. ① This area displays the fields of the current modeling task instance node. You can select a field to display its consumption lineage information on the right. ② This area displays the downstream nodes of the selected field, which helps you analyze the impact on them when you manage the field.
Rerun	Dataphin supports this operation only for logical table instances with a Succeeded or Failed running status, and for fields with a normalized status of Succeeded or Failed. After the rerun, Dataphin updates the data of the reran fields in the current instance and its downstream nodes. If your use case requires a rerun but the running status of the logical table instance or the normalized status of the field prevents it, you can perform a force rerun. For details, see Force rerun the current instance.
Rerun upstream and downstream	Dataphin supports this operation only for modeling task instances with a Succeeded or Failed running status, and for the upstream and downstream nodes of fields with a normalized status of Succeeded or Failed. If status limitations prevent this operation but your use case requires it, you can perform a force rerun. For details, see Force rerun the current instance. This operation is commonly used in the following scenarios: A data update in the current instance requires a corresponding update in downstream nodes. If an upstream or downstream node of an instance fails, you can rerun the failed node(s).
Stop	You cannot stop an instance if its Node Status is Succeeded, Failed, or Not Run. You can stop instances with any other Node Status. After you stop an instance, its node status changes to Failed, and its downstream nodes will not run. This operation is commonly used in the following scenarios: Stopping an instance due to a Dataphin anomaly or an operational error. A script appears to be running, but no data is being generated. You can manually stop the run and check the run log to troubleshoot the issue.
Force rerun	A force rerun does not check whether all upstream nodes have succeeded or whether the scheduled run time for the current instance has been reached. This risks run failures or data quality issues. Confirm that there will be no impact on downstream data before proceeding. For details, see Force rerun the current instance. A force rerun of upstream and downstream instances works as follows: A force rerun does not validate the status of upstream nodes, so you can forcibly rerun instances in any running status. When you perform this operation, Dataphin first sets all affected instances to the "Not Run" status. This prevents data corruption that could occur if a downstream node runs while an upstream node is rerunning. Only the starting point of the force rerun ignores upstream dependencies and scheduled run times. Downstream nodes must still adhere to their dependency relationships and start conditions. For details, see Force rerun upstream and downstream instances. For more information about instance start conditions, see Dependency relationships and run description.
Pause	This action pauses the field's current scheduling cycle. Other cycles are not affected.
Set as successful and continue scheduling	This action manually sets the status of a field from Failed or Not Run to "Succeeded", to allow scheduling to continue. For details, see Set as successful and continue scheduling.
Resume	Dataphin only supports resuming paused fields. After you resume a paused field, the system automatically checks whether the current node meets the following run conditions: All upstream nodes have run successfully. The scheduled time for the current field has been reached. If these conditions are met, the system runs the field automatically. If not, the system only removes the field's paused status, and the field will run later when its run conditions are met.

Set successful and continue scheduling

In the Set Status to Successful and Continue Scheduling-Logical Table dialog box, select the scope.
- Failed fields: Sets all fields whose normalized running status is "failed" as successful.
- Not-run fields: Sets all fields whose normalized running status is "unrun" as successful.
- Failed and not-run fields: Sets all fields whose normalized running status is "failed" or "unrun" as successful.
- Custom: Lets you select specific fields in the logical table instance with a "failed" or "unrun" normalized running status and set them as successful.
Click OK to complete the settings.