DataWorks provides auto triggered workflows for business scenarios that require periodic execution. This topic describes how to use auto triggered workflows.
Background information
A workflow is a tool for managing automated data processing flows. You can integrate various types of task nodes using a visual drag-and-drop interface. This simplifies building task dependencies, accelerates the creation of data processing flows, and improves development efficiency.
Scheduling dependencies supported by auto triggered workflows
The scheduling dependency configuration for an auto triggered workflow is similar to that of a regular node. Inner nodes of a workflow or the entire workflow can depend on other tasks or serve as dependencies for other nodes. The details are as follows:
The workflow as a whole can be a dependency for other tasks, such as independent tasks or other workflows.
The workflow as a whole can depend on other tasks, such as independent tasks or other workflows.
Runtime status transition of an auto triggered workflow
An auto triggered workflow supports setting a timed schedule for the entire workflow. The execution of inner nodes is affected by this timed schedule. If a descendant node depends on the entire workflow, the execution of the descendant node is also affected by the workflow's timed schedule. In a scheduling scenario, the success status of the entire workflow depends on the running status of its inner nodes.
Recurring instance status:
|
|
Not run
Waiting
Running
Successful
Failed
Paused/Frozen instance
Special scenarios:
If a workflow contains a frozen or paused instance, the entire workflow instance is set to the Failed status.
When you freeze a data backfill instance for a workflow node, the workflow instance is set to the Successful status.
In a data backfill scenario, if a task cannot be executed, the workflow is set to the Failed status.
There is a latency between the instance status update and the actual failure event.
If a workflow contains a merge node, upstream nodes of the merge node may fail. In this case, the success status of the workflow is determined by the status of the merge node.
Execution time and parameter substitution for inner nodes of an auto triggered workflow
Inner nodes of a workflow do not require a Scheduling Cycle configuration. You only need to set the Delayed Execution Time. This specifies the scheduling delay relative to the timed schedule defined for the workflow.
The actual execution time is calculated based on the Scheduling Time configured for the workflow and the Delayed Execution Time configured for the inner node.
The values of scheduling parameters for an inner node are determined by the overall timed schedule of the workflow, not by the delayed execution time.
The system automatically appends custom workflow parameters to the scheduling parameters of the inner nodes. You can retrieve these workflow parameters within the inner nodes.
Precautions
Auto triggered workflows are supported only in the new version of DataStudio. This feature is not available in the old version of Data Development.
When you create an auto triggered workflow, select Periodic Scheduling as the scheduling type.
A referenceable workflow cannot depend on other tasks. It is not automatically scheduled by default after being published to the production environment.
ImportantAfter you enable the Can Be Referenced feature for a workflow, the workflow and its inner nodes cannot depend on other tasks. This includes other workflows or nodes outside this workflow. The workflow also cannot serve as a dependency for other tasks, including the root node of the workspace. Otherwise, an error occurs when you publish the workflow, and the publishing fails.
A workflow with Can Be Referenced enabled is not automatically scheduled by default after being published to the production environment. It runs only when another workflow references it using a SUB_PROCESS node.
Go to the Create Workflow page
Go to the Workspaces page in the DataWorks console. In the top navigation bar, select a desired region. Find the desired workspace and choose in the Actions column.
In the navigation pane on the left, click
. Then, to the right of Workspace Directories, click the 
icon on the right side and select Create Workflow.NoteThe first time you perform operations in the Workspace Directories section of the DATA STUDIO pane, you can directly click Create Workflow to create a workflow.
Create an auto triggered workflow
On the Create Workflow page, select Periodic Scheduling as the scheduling type.
NoteTimed scheduling lets you specify the frequency for task scheduling. You can trigger task execution at a specified time.
For information about how to create an event-triggered workflow, see Event-triggered workflows.
Enter a workflow name and click OK to create it.
Design an auto triggered workflow
On the left side of the workflow canvas, select the nodes that correspond to your required task types. Drag the nodes to the canvas and connect them to configure dependencies.
DataWorks encapsulates DPI engine capabilities into nodes. This lets you perform visual data development using engine-type nodes instead of complex engine command lines. You can also use the general-purpose nodes that are provided by the platform to handle complex logic.
A single workflow supports a maximum of 400 inner nodes. For the best display and maintenance experience, we recommend that you keep the number of nodes under 100.
Development Cycle Workflow
Follow these steps to develop the workflow. For more information about how to reference workflows and pass parameters between a workflow and its inner nodes, see Appendix: Workflow reference.
Open the inner node editing page.
On the business flow canvas, move the pointer over the target node and click Open Node to open the editing page for the inner node.
Develop the inner node.
On the node editing page, edit the node code. Note the following when you develop the code:
The code syntax depends on the selected node type. Different task types have different configurations. For more information about the configurations, see Node development.
You can enable Copilot Intelligent Programming Assistant to obtain intelligent code completion suggestions and improve development efficiency.
For most node types, you can define variables using the
${variable_name}format. This lets you assign different values for quick debugging in the next step.When a scheduled task is executed, you can retrieve the value of a workflow parameter in an inner node using the
${workflow.Parameter_name}format.NoteWhen you use workflow parameters in a Shell or Python node, the system automatically appends these parameters to the node's parameter list in the
k=vformat. You can parse and retrieve the corresponding parameter values in this format within the node code.Only Serverless resource groups support retrieving workflow parameter values in inner nodes in this way.
Debug and run the inner node.
Configure debug parameters. After you finish editing the code, click Debugging Configurations on the right side of the node editing page to configure debug parameters.
In the Computing Resource section, specify the computing resource for the task during the debugging phase.
In the DataWorks Configurations section, specify the resource group for DataWorks task execution.
If you defined variables in the code using the
${variable_name}format, you must assign constant values to the corresponding variable names in Script Parameters.
Debug and run the task. After you complete the configuration, click the Run button in the toolbar. The node runs using the debug parameters that you configured in Debugging Configurations.
Debug an auto triggered workflow
After you develop and debug the inner nodes of the workflow, debug and run the workflow task.
Click the
icon in the toolbar above the business flow canvas. In the Enter Runtime Parameters dialog box, manually enter the Value Used in This Run. Click OK.On the business flow canvas, click an inner node of the workflow to view the task execution result.
Publish an auto triggered workflow
Follow the procedure below to define the scheduling configurations for the workflow and its inner nodes. Then, publish the workflow to the production environment for periodic execution.
Configure scheduling for inner nodes.
The scheduling configuration for the inner nodes of a workflow is similar to the configuration for regular nodes. For more information about the configuration, see Scheduling dependencies. Note the following during the configuration:
Inner nodes of a workflow do not require a separate timed schedule. You can set the Delayed Execution Time, which specifies the delay before the task runs after the workflow starts.
The values of variables used in the code of an inner node depend on the timed schedule of the workflow.
Configure scheduling for the workflow.
Configure the workflow's Scheduling Parameters, Scheduling Time, and Scheduling Dependencies.
Publish the workflow.
Click the
button in the toolbar to open the publishing panel. Click Start Deployment to Production Environment. The task is published based on the pre-publish check process. For more information, see Publish nodes/workflows.
What to do next: Workflow O&M
After an auto triggered workflow is published, it is scheduled to run periodically. You can go to Operation Center in the production environment to view the scheduling status of the auto triggered workflow and perform O&M operations. For more information, see Operation Center and Data backfill.
Other operations
Clone an auto triggered workflow
You can use the clone feature to quickly create a new auto triggered workflow from an existing one. The cloned content includes the inner nodes of the original workflow, including their Code, Debugging Configurations, and Scheduling. It also includes the dependencies between nodes and the workflow's own Scheduling, which contains the Scheduling Parameters and Scheduling Policy.
The cloned content does not include the Scheduling's Scheduling Time and Scheduling Dependencies information. You must configure these parameters.
In the Workspace Directories on the left, right-click the name of the auto triggered workflow you want to clone.
In the pop-up menu, select Clone to open the clone dialog box.
In the clone dialog box, modify the workflow Name and save Path. You can use the default values. Click Confirm to start cloning.
During the cloning process, you can view details such as the Current Progress, Duration, and Number of Completed Nodes in the dialog box.
After the cloning is complete, you can view the newly generated auto triggered workflow in the Workspace Directories.
To add a new node to the auto triggered workflow, you can create a node by cloning or create an inner node by dragging and dropping.
Version management
You can use the version management feature to revert a workflow to a historical version. This feature also provides version viewing and comparison capabilities to help you analyze differences and make adjustments.
In the Workspace Directories on the left, double-click the name of the target auto triggered workflow to open the workflow canvas.
On the right side of the workflow canvas, click Versions. On the Versions page, view and manage the Development Records and Publishing Records.
View versions:
On the Development Records or Publishing Records tab, find the version of the auto triggered workflow you want to view.
Click View in the Actions column. On the details page, you can view the auto triggered workflow's Code and Scheduling information.
NoteThe Scheduling information can be viewed in Code Editor and Visualization mode. You can switch the view mode in the upper-right corner of the Scheduling tab.
Compare versions:
You can compare different versions of an auto triggered workflow on the Development Records or Publishing Records tab. The following example describes how to perform a comparison using development records.
Compare versions in the development or publishing environment: On the Development Records tab, select two versions and click the Compare Selected button above the list. You can then compare the code and scheduling configuration information of the different workflow versions.
Compare versions between the development and publishing or build environments:
On the Development Records tab, locate a version of the auto triggered workflow.
Click the Compare button in the Actions column. In the Please Select Content to Compare window, select a version from the Publishing Records or Build Records tab to compare.
Revert a version:
You can revert an auto triggered workflow to a historical version only from the Development Records tab. On the Development Records tab, find the target version and click the Revert button in the Actions column. This reverts the auto triggered workflow to the target version.
NoteWhen you revert a workflow, the system restores the workflow based on the target version and generates a new version record.
Appendix: Workflow reference
Scenario 1: Reference a workflow
In a workflow, you can use a SUB_PROCESS node to reference another workflow. For the referenced workflow, you must enable the Can Be Referenced switch in the Properties configuration on the right side of the workflow editing page.
A workflow with Can Be Referenced enabled is not automatically scheduled after being published to the production environment. It runs only when another workflow references it using a SUB_PROCESS node.
The referenced workflow is scheduled and executed along with the referencing workflow.
Scenario 2: Retrieve workflow parameters
If you define a parameter in a workflow, the system automatically appends the custom workflow parameter to the scheduling parameters of the inner nodes when the task is scheduled. You can retrieve this parameter in an inner node of the workflow by performing the following steps:
Define workflow parameters.
In the Workspace Directories on the left, select the workflow you want to edit.
On the right side of the workflow canvas, in the Scheduling section, click Add Parameter. Enter a Parameter Name and Parameter Value.
Retrieve workflow parameters.
Go to the workflow that you edited. In the workflow, select the inner node that you want to edit.
Move the pointer over the inner node and select Open Node to open the inner node editing page.
On the right side of the inner node editing page, find the Parameter Name under Workflow Parameters in the Scheduling section.
Retrieve the workflow parameter.
You can retrieve the workflow parameter using the
${workflow.Parameter_name}format.
FAQ
Q: Why does my workflow run as expected during debugging but fail during a periodic scheduled run?
A: This issue may occur because the computing resources and resource groups that are selected in the scheduling configuration for the inner nodes of the workflow are different from those selected for the debug run. This can cause the debug run to succeed but the scheduled run to fail.