Rerun and Monitor Integration & Computing Instances - Dataphin - Alibaba Cloud - Dataphin

Periodic integration and compute tasks generate instances at their scheduled runtime. You can then manage these instances by viewing running logs, rerunning them, forcing a rerun of current or downstream instances, or viewing node code. This topic describes how to view and manage integration and compute task instances.

Prerequisites

You can view the Gantt chart only after you purchase the Intelligent O&M value-added service and enable the Intelligent O&M module for the current tenant.

Accessing integration and compute instances

In the top navigation bar of the Dataphin homepage, choose Development > Task O&M.
In the left-side navigation pane, choose Instance O&M > Periodic Instances.
In the top navigation bar, select the production or development environment.
On the Periodic Instances page, click the Integration and Compute tab.

Operations in the instance list

After a periodic integration and compute task generates an instance, it is displayed in the list on the Periodic Instances > Integration and Compute tab. The list displays information for each instance, including Instance object, Instance ID, Directory, Status, Scheduling cycle, Business date, Scheduled time, Start time, End time, Duration, Retries/Auto retries, Priority, Owner, Project, Associated baseline instances, HTTP path, Scheduling resource group, Tags, and available operations.

Instance object: An instance object is generated each time a periodic task runs. This column displays the name and ID of the instance object and indicates the task's scheduling method. You can click the icon next to the column name to sort the objects by name in ascending or descending order. For more information, see Periodic instance icon descriptions.
Directory: A snapshot of the task directory when the instance was generated. This information does not update if the task directory changes. Directory information is displayed only for instances generated after an upgrade to Dataphin V6.2. If you attempt to operate on an instance in a directory for which you do not have permissions, the system reports an error.
Status: The running status of the instance. Possible statuses include: Success, Failed, Running, Waiting for Scheduling Time, Throttling, Waiting for Scheduling Resources, and Not Running. For details about each status, see Periodic instance running status descriptions.
Start time: The time when the instance started running. Click the icon next to the column name to sort by start time in ascending or descending order.
Note
For a logical table node, the start time is the time when the earliest internal materialized node of the instance object starts running.
End time: The time when the instance finished. Click the icon next to the column name to sort by end time in ascending or descending order.
Note
For a logical table node, the end time is the time when the latest internal materialized node of the instance object finishes running.
Retries/auto retries: The number of manual and automatic reruns for the instance, where Retries = Number of runs - 1.
Duration: The elapsed time from start to finish. Click the icon next to the column name to sort by duration in ascending or descending order.
Note
For a logical table node, the duration is the time difference between the start time of the earliest internal materialized node and the end time of the latest one.
Priority: The priority level of the instance.
Note
If the baseline feature is enabled, the task's priority is set to the highest priority of its associated baselines, overriding its configured priority.
Project: The project to which the task belongs, displayed in the format ProjectEnglishName(ProjectChineseName).
Associated baseline instances: The baseline that guarantees the instance as an endpoint node, and related baselines that include the instance as an upstream node within their guarantee scope.
Note
This field is not displayed if the baseline feature is not enabled.
HTTP path: Displays the production or development environment HTTP path for a Databricks SQL instance, based on the selected environment.
Note
This field is displayed only for Databricks SQL instances. For other task types, it shows as -.
Scheduling resource group: The name of the scheduling resource group that the instance uses for execution.
If the custom resource group specified for the task is unavailable, the project's default resource group is used. If the project's default resource group is also unavailable, the tenant's default resource group is used. The priority is as follows: Custom resource group > Project default resource group > Tenant default resource group.
Note
After you change the project's default resource group, the updated information might be displayed with a delay. However, the task execution will use the newly specified resource group.
Tenant default resource group: This resource group does not belong to any project. Each tenant has only one default resource group. If a task does not specify a custom resource group and the project does not have a default one, the tenant's default resource group is used for scheduling. This applies only to dedicated-resource tasks, such as SQL and Virtual tasks.

The following table describes the operations available for integration and compute task instances.

Operation	Description
DAG	Click the icon to view the DAG for the periodic instance. For more information, see Periodic instance DAG.
View running log	Click the icon to view the running log for the periodic instance. The content of the running log varies depending on the instance type.
Rerun	Click the icon to rerun the periodic instance. You can only rerun integration and compute task instances that are in the Success or Failed state. When you rerun an instance, the system checks if all upstream nodes have a status of Success. If this check fails, the instance status is immediately set to Not Running. If your business scenario requires a rerun despite this check, you can perform a force rerun.
View Gantt chart	Click the icon and select View Gantt Chart to open the View Gantt Chart dialog box and configure the following parameters. Selected node as: You can set the current node as the Start node or End node. If you select Start node, the current node is automatically set as the start node and cannot be changed. The same applies if you select End node. Start node: Select the start node for the Gantt chart. You can search for nodes by keyword. The selectable nodes are listed in the format `{Node Name}({Node ID}, {Instance Scheduled Runtime})`. If you set the selected node as the end node, the start node defaults to the tenant's root node. Note Hourly and minute-level tasks have multiple nodes, one for each instance, which are distinguished by their scheduled runtime. End node: Select the end node for the Gantt chart. You can search for nodes by keyword. The selectable nodes are listed in the format `{Node Name}({Node ID}, {Instance Scheduled Runtime})`. Note When you select a logical table as the end node, you cannot select specific fields because the logical table is treated as a single unit. For more information about Gantt charts, see View critical path Gantt chart.
Download upstream and downstream nodes	Downloads a list of the upstream and downstream nodes for the current node. The list includes all columns, even those not currently displayed in the UI. Click Download Upstream and Downstream Nodes. In the Download Upstream and Downstream Nodes dialog box, select the levels for the upstream and downstream nodes. You can select from Level 1 to Level 10 or All levels. The default for both is Level 1. After you select the levels, click OK to start downloading an Excel file. The file is named `{task_name}_Upstream_and_Downstream_Nodes_{timestamp}.xlsx`.
View node code	Click the icon and select View Node Code to view the code for the current instance. For Flink_SQL integration and compute task instances, you can switch between Logical Code and Physical Code. Logical Code: The task code that you wrote. Physical Code: The code that is compiled to run on the Flink engine.
Periodic task	Click the icon and select Periodic Task to view the periodic task to which the current instance belongs.
Edit development node	Click the icon and select Edit Development Node to go to the instance's development page to make changes. Note This operation is available only for integration and compute task instances in projects that use the Dev-Prod mode.
View production node	Click the icon and select Edit Production Node to go to the production page for the instance. Note This operation is available only for integration and compute task instances in projects that use the Dev-Prod mode.
Edit node	Click the icon and select Edit Node to edit the integration and compute task that generated the instance. You can access this task from the instance list or the DAG in a Basic mode project. Note This operation is available only for integration and compute task instances in projects that use the Basic mode.
Rerun downstream	Click the icon and select Rerun Downstream to rerun the downstream nodes of the instance. You can only rerun downstream nodes of an integration and compute task instance that is in the Success or Failed state. Downstream instances with a status of Waiting (Waiting for scheduling time, Waiting for scheduling resources) or Running cannot be rerun. For more information, see Rerun downstream instances. To rerun the entire pipeline, we recommend using Force Rerun Downstream Instances. For more information, see Force rerun downstream instances. The Rerun Downstream operation is often used in the following scenarios: You need to update data in downstream nodes after the data in the current instance node has been updated. You need to rerun downstream nodes after they have failed.
Set to success and continue scheduling	Click the icon and select Set to Success and Continue Scheduling to manually set integration and computing task instance nodes that have Failed or are Not Run to successful and allow them to resume scheduling. Click Set to Success and Continue Scheduling, and in the Set to Success and Continue Scheduling prompt dialog box, click OK to complete the configuration.
Terminate	Click the icon and select Terminate to terminate the currently running instance. Its status then changes to Failed. After an instance is terminated, its downstream nodes will not run. Note You cannot terminate an instance that is already in a Success, Failed, or Not Running state. Instances in any other state can be terminated. The Terminate operation is often used in the following scenarios: You need to stop an integration and compute task instance due to a Dataphin error or an operational mistake. An integration and compute task instance is shown as running, but no data is being produced. You can manually terminate the run and check the logs to troubleshoot the issue.
Force rerun	Click the icon, select Force Rerun, and then choose either Force Rerun Current Instance or Force Rerun Downstream Instances. Force Rerun Current Instance: Forces a rerun of the instance without checking the status of its upstream nodes. If the upstream nodes have not all run successfully, the current instance may fail to rerun or produce incorrect data. Proceed with caution. Force-rerunning the current instance updates its data. Force Rerun Downstream Instances: When you perform this operation, all selected instances are first set to Not Running to prevent data conflicts that could arise if downstream nodes run while upstream nodes are rerunning. Only the start node bypasses checks for upstream dependencies and scheduled runtime. Downstream nodes still adhere to their dependencies and startup conditions. For more information, see Force rerun downstream instances. Important A force rerun does not check whether all upstream nodes have run successfully or whether the instance's scheduled runtime has been reached. This can lead to run failures or data quality issues. Confirm that there will be no impact on downstream data before proceeding.
Remove upstream dependencies	Click the icon and select Remove Upstream Dependencies to remove upstream dependencies from an integration and compute task instance, regardless of its running status. Important At least one upstream instance must be retained.
Pause	Click the icon and select Pause to suspend the currently running instance. Pausing an integration and compute task instance does not affect other instances of the same task. After an instance is paused, it and its downstream dependent instances will not be scheduled. Note You can only pause an instance that is in the Not Running, Waiting for scheduling time, or Throttling state. The Pause operation is often used to free up resources by pausing low-priority instances to ensure high-priority tasks can run first.
Resume	Click the icon and select Resume to resume a Paused integration and compute task instance. After being resumed, the instance and its downstream instances will be scheduled normally.
Modify HTTP path	Modify the production HTTP path for the corresponding task. You can select any HTTP path configured for the cluster associated with the production project. Note This operation is supported only when you select a Databricks SQL task in the production environment.
Modify scheduling resource group	Click the icon and select Modify Scheduling Resource Group to change the scheduling resource group used to run the instance. Note If you select tasks from multiple projects, the target scheduling resource group list will only show resource groups that are shared by and authorized for all selected projects. We recommend filtering by a single project before performing this bulk operation. This change affects only the current instance and has no impact on other instances of the same task.
Modify priority	Click the icon and select Modify Priority. In the Modify Priority dialog box, change the priority of the current task. You can select Highest, High, Medium, Low, or Lowest.

Operations for DAG nodes

The DAG provides a clear visualization of the upstream and downstream dependencies of an instance node. The system also allows you to manage upstream and downstream instance nodes. By default, the DAG displays the main node (the selected node) and its first-level upstream and downstream nodes. You can select an integration and compute task instance node to perform related O&M operations.

Dataphin supports O&M across different projects. To perform O&M on an integration and compute task node in a different project, you must have view and operation permissions for the project where the instance is located.

Operations in the DAG view

Operation	Description
Expand parent nodes	Expand different levels of dependent nodes for the main node in the DAG.
Expand child nodes
View task	Navigate to the DAG of the task that generated the current instance node. Here, you can view task node details, upstream and downstream information, and perform O&M. For more information, see Periodic Task.
View operation log	View the log of operations performed on this instance.

Operations for DAG nodes
Hover over a DAG node to view its name, type, scheduling cycle, owner, and description. The operations available for an integration and compute task instance DAG node are the same as those in the instance list. For more information, see Operations in the instance list.

Batch operations

The following table describes the batch operations supported for periodic integration and compute tasks.

Operation	Description
Rerun	This operation is only supported for instances in the Success or Failed state. If none of the selected instances are in the Success or Failed state, the batch rerun operation is not available. If your selection includes instances in Success or Failed states along with other states, the system only reruns the instances that are in the Success or Failed state.
Terminate	You cannot terminate instances that are in the Success, Failed, or Not Running state. Instances in any other state can be terminated. If all selected instances are in the Success, Failed, or Not Running state, the batch terminate operation is not available. If your selection includes instances in these states along with others, the system does not terminate the instances that are in the Success, Failed, or Not Running state.
Set to success and continue scheduling	After selecting multiple instances, you can manually set the status of integration and compute task instances from Failed or Not Running to Success, allowing them to proceed with scheduling.
Pause	This operation is only supported for instances in the Not Running, Waiting for scheduling time, or Throttling state. If none of the selected instances are in the Not Running, Waiting for scheduling time, or Throttling state, the batch pause operation is not available. If your selection includes instances in these states along with other states, the system only pauses the instances that are in the Not Running, Waiting for scheduling time, or Throttling state.
Resume	Resumes multiple periodic instances that are in the Paused state.
Modify HTTP path	Modify the production HTTP path for multiple Databricks SQL instances. If the selected instances belong to different Databricks clusters, you can specify different HTTP paths for each cluster. You can select any HTTP path configured for the corresponding cluster. Note This operation is supported only when you select Databricks SQL instances in the production environment.
Modify scheduling resource group	Modify the scheduling resource group used to run the instances. Note You can only modify the scheduling resource group for instances in a production environment project. This change affects only the current instance and has no impact on other instances of the same task.
Modify priority	Modify the priority of the selected instances. You can select Highest, High, Medium, Low, or Lowest.
Download all	Clicking this submits a download task, and the system generates the file asynchronously in the background. Once generated, the file is automatically downloaded to your local machine. The file is in .xlsx format with the name `Dataphin_periodic_instances_{timestamp}.xlsx`. The file includes the following information: Instance Object, Instance ID, Status, Scheduling Cycle, Business Date, Priority, Owner, Project (for summary logical tables with multiple projects, project names are separated by commas), Scheduled Time, Start Time, End Time, Duration, Retries/Auto Retries, Associated Baseline Instances (names of baselines associated with the instance, separated by commas), and Scheduling Resource Group (this field is empty for modeling task instances).

Rerun downstream instances

In the Rerun Downstream dialog box, configure the parameters.

Note

Downstream instances with a status of Waiting or Running cannot be rerun. To rerun the entire pipeline, we recommend using Force Rerun Downstream Instances. For more information, see Force rerun downstream instances.

Parameter

Description

Start node running mode

Defines how the start node runs. You can select either Dry Run or Normal Run.

Dry Run: The instance status is set to Success and it proceeds with normal scheduling, but the running log is empty, there is no duration, and it does not process any data.
Normal Run: The instance is scheduled and runs normally.

Downstream rerun scope

Select the scope of downstream nodes to rerun.

All Failed Instances: The system automatically selects all downstream instances with a Failed status to be rerun, without displaying a specific list.
Custom: If you need to specify which downstream instances to rerun, select this option. You can search by node name or ID, and filter by status, owner, or project.

Click OK.
After the downstream instances are rerun, their data is updated.

Force rerun downstream instances

In the Force Rerun Downstream dialog box, configure the parameters.

Parameter

Description

Start node running mode

Defines how the start node runs. You can select either Dry Run or Normal Run.

Dry Run: The instance status is set to Success and it proceeds with normal scheduling, but the running log is empty, there is no duration, and it does not process any data.
Normal Run: The instance is scheduled and runs normally.

Downstream force rerun scope

Select the scope of downstream nodes to force rerun.

All Instances: Selects all downstream instances of the start node.
Custom: If you need to specify which downstream instances to rerun, select this option. You can search by node name or ID, and filter by status, owner, or project.

Click OK to complete the setup.