Auto triggered nodes are nodes that are periodically scheduled based on their scheduling configurations after they are committed to the scheduling system. You can view auto triggered nodes in a specified workspace in the list and perform O&M operations on the nodes, such as automatically scheduling and manually running auto triggered nodes, viewing node running details, freezing nodes, and undeploying nodes. The following table describes the O&M operations that you can perform on auto triggered nodes.
Instructions
Auto triggered nodes can be automatically scheduled to generate instances only in Operation Center in the production environment. Auto triggered nodes cannot be automatically scheduled to generate instances in Operation Center in the development environment.
After you perform operations such as addition, modification, and undeployment on node code, scheduling configurations, resources, or functions in the production environment, you must commit and deploy the objects to make the configurations take effect.
After you modify an auto triggered node, you must deploy the node. After the node is successfully deployed, the modification takes effect in the production environment. During deployment, issues such as deployment failures, blocked deployment processes, or unexpected deployment versions may occur. Therefore, we recommend that you go to the page to check after you deploy a node. For more information, see Manage auto triggered nodes.
Overview
The following table describes the O&M operations that you can perform on auto triggered nodes.
O&M operation | Description |
You can view the details of an auto triggered node and perform operations on the node in the list of auto triggered nodes or in the directed acyclic graph (DAG) of the node on the Cycle Task page. | |
| |
| |
You can view the operation logs, versions, and operation history of an auto triggered node. |
View auto triggered nodes
In the list of auto triggered nodes, you can view the auto triggered nodes that are committed and deployed to the scheduling system in the production environment. You can check whether the code, scheduling parameter configurations, scheduling dependencies, and lineage of the auto triggered nodes meet your business requirements. You can view the details of an auto triggered node and perform operations on the node in the auto triggered node list or in the directed acyclic graph (DAG) of the node on the Cycle Task page. For more information, see Manage auto triggered nodes.
Only the auto triggered nodes that are deployed to the production environment are displayed in the list of auto triggered nodes on the Cycle Task page in Operation Center.
The following two types of nodes are not automatically scheduled: nodes that do not depend on other nodes and nodes that are used as the ancestor nodes of other nodes and depend on their descendant nodes.
Run auto triggered nodes
You can understand the modes in which nodes in DataWorks are run and perform O&M diagnostics based on the running situations of nodes in an efficient manner.
Running modes
DataWorks generates auto triggered node instances that are scheduled to run on the next day for an auto triggered node every night. You can click Backfill Data or Test in the Actions column of the auto triggered node to generate a data backfill instance or a test instance for the auto triggered node.
Instance type | Scenario | Relationship with auto triggered nodes (How instances are generated) | Trigger method (How instances are triggered to run) |
You want to perform periodic extract, transform, load (ETL) operations. | Every night, DataWorks automatically generates auto triggered node instances that are scheduled to run on the next day based on the snapshot information of an auto triggered node at the specified point in time. Note Auto triggered nodes cannot be automatically scheduled to generate instances in Operation Center in the development environment. | DataWorks triggers an auto triggered node instance to run. | |
You want to backfill data of a period of time in the past or in the future for the current auto triggered node and its descendant nodes. This means that you must perform ETL operations on the data of that period of time. | You must manually backfill data for the current auto triggered node to generate data backfill instances for the node. | After you backfill data, the data backfill instances are generated and triggered to run. | |
You want to test the current auto triggered node to check whether the node can be run as expected. Note The code of the auto triggered node is run during the test. | You must manually test the current auto triggered node to generate test instances for the node. | After you perform the test, the test instances are generated and triggered to run. |
Conditions for running an auto triggered node
To run a scheduled node, the following conditions must be met: All ancestor nodes of the node are successfully run, the scheduled time of the node arrives, sufficient scheduling resources are available, and the node is not frozen. For more information, see Conditions for running a scheduled node.
Diagnose node running issues
Issue: When the scheduled time of a node arrives but the node is not run, the possible causes are as follows: Not all ancestor nodes of the node are successfully run, the scheduled time of the node does not arrive, insufficient scheduling resources are available in the workspace, or the node is frozen.
Troubleshooting: We recommend that you use the Upstream Analysis feature on the DAG panel to quickly identify the key ancestor node that blocks the running of the current node. Then, use the Run Diagnosis feature to diagnose the cause of the issue or identify the issue with the key instance. If the node has complex dependencies, you can use this feature to quickly identify the issue and improve O&M efficiency. For more information, see Why is a node not run when its scheduled time arrives?, Waiting for Resources, and Freeze and unfreeze a node.
Emergency O&M operations
If an auto triggered node depends on multiple ancestor nodes and one of the ancestor nodes is not run, you can right-click the instance that is not run and select .
NoteYou must check whether this operation affects data output based on the code of the node for which the instance is generated and lineage of the instance.
If large-scale data quality issues occur, you can right-click the instance and select . For more information, see Appendix: Force Rerun Descendant Nodes.
In some extreme cases, such as unexpected server power outages or primary/secondary switchovers, DataWorks may not be able to completely terminate the related MaxCompute task processes. In this case, go to the corresponding MaxCompute compute project to terminate the job.
Manage auto triggered nodes
Deploy auto triggered nodes
Before you can view an auto triggered node in the list of auto triggered nodes on the Cycle Task page in Operation Center, you must deploy the node to the scheduling system in the production environment. For more information, see Deploy a node (Legacy DataStudio) and Deploy a node or workflow (New DataStudio).
Undeploy auto triggered nodes
If you no longer require an auto triggered node or a workflow, you can undeploy the node or all nodes in the workflow. After you undeploy an auto triggered node, you cannot find the node on the Cycle Task page. For more information, see Undeploy a node (Legacy DataStudio) and Undeploy a node (New DataStudio).
Suspend scheduling
Do not perform operations on the projectname_root node at will. This node is the root node of the workspace. All the instances of auto triggered nodes depend on this node. If this node is frozen, the instances of auto triggered nodes cannot be run.
Operation | Scenario | Description |
Freeze an auto triggered node | If an auto triggered node and its descendant nodes do not need to be run for a specific period of time, you can freeze the auto triggered node. |
|
Freeze an instance | If an instance generated for an auto triggered node does not need to be run, you can freeze the instance. | The freeze operation takes effect only on the current instance. Other instances that are generated on the same day as the current instance and the instances that are generated later than the current day are not affected. |
Set Recurrence to Dry Run for an auto triggered node | If an auto triggered node does not need to be run for a specific period of time but you do not want to block the running of its descendant nodes, you can set the Recurrence parameter to Dry Run for the node. | The auto triggered node in the dry-run state generates dry-run instances. The system does not run the dry-run instances to generate data, does not generate run logs for the dry-run instances, and does not display running duration for the dry-run instances. Note The operation of setting the Recurrence parameter to Skip Execution for an auto triggered node on the DataStudio page achieves the same effect as the operation of freezing the auto triggered node in Operation Center. The modifications that you made on an auto triggered node on the DataStudio page take effect only in the development environment. If you want the modifications to take effect in the production environment, you must deploy the auto triggered node. For more information, see Deploy a node (Legacy DataStudio) and Deploy a node or workflow (New DataStudio). |
For more information about the impacts of freezing and unfreezing auto triggered nodes and auto triggered node instances, see Freeze and unfreeze a node.
Manage node priorities
You can use the baseline management feature to adjust the priority of an auto triggered node. Scheduling resources are preferentially allocated to the auto triggered nodes with higher priorities. For more information, see Manage baselines.
Configure monitoring and alerting for an auto triggered node
You can find the auto triggered node that you want to monitor and configure a custom alert rule or monitoring rule for the node:
Configure custom alert rules to monitor the status of auto triggered node instances. For more information, see Create and manage custom alert rules.
Configure monitoring rules for table data that is generated when auto triggered node instances, data backfill instances, or test instances generated for auto triggered nodes are run. For more information, see Overview of Data Quality.
You can configure a custom alert rule to monitor a resource group used to run an auto triggered node based on the number of instances that are generated for the node and waiting for resources in the resource group or based on the resource usage of the resource group. For more information, see Create and manage custom alert rules.
Change the resource group used to run an auto triggered node
DataWorks allows you to change the resource group for scheduling or resource group for Data Integration used to run an auto triggered node. For more information, see General reference: Switch resource groups.
If you want to redefine scheduling properties of an auto triggered node, you can go to the DataStudio page, find the desired auto triggered node, and modify the configurations of the scheduling properties on the configuration tab of the auto triggered node. For more information, see Configure basic properties. If you want to modify multiple nodes at a time, you can go to the Batch Operation page.
Change the node owner
You must first enable Change Node Owner by RAM User. After Change Node Owner by RAM User is turned on, the workspace administrator can perform the following operations:
Change the owner of a node or the owners of multiple nodes at a time on the DataStudio page
Change the owner of a single node: Open the .
Change the owners of multiple nodes at a time: Go to the Batch Operation page. For more information, see Batch Operation.
After you change the owner of a node in the development environment, you must deploy the node to the production environment. This way, the change can take effect.
Change the owner of a node or the owners of multiple nodes at a time in the production environment:
Change the owner of a single node: Click in the Actions column of the node.
Change the owners of multiple nodes at a time: Select the nodes whose owners you want to change, and then click Change Owner at the bottom of the page.
NoteAfter you change the owner of a node in the production environment, the owner of the node in the development environment is also changed.
View operation records of auto triggered nodes
Entry | Description |
Operation Log of a node or an instance | Click a node or an instance and switch to the Operation Log tab to view the change records of the node or instance. |
Version of a node | If no details about an operation that is performed on and deployed for an auto triggered node are recorded, you can go to the configuration tab of the auto triggered node and compare an existing version of the node with the version of the node in the production environment to obtain details about version changes. For more information, see: Deploy a node (Legacy DataStudio) and Deploy a node or workflow (New DataStudio). |
Operation History | You can go to the Operation History page in Operation Center to view the operation records of an auto triggered node, an auto triggered node instance, or a baseline. For more information, see: View operation records in Operation Center. |
FAQ
For more information about frequently asked questions about auto triggered nodes, see FAQ about auto triggered nodes.