DataWorks:Periodic workflow

Step 1: Prepare compute engine and data

1. In your target workspace, bind MaxCompute as a compute engine.

2. In MaxCompute, create a table with the following structure to store the results.

   -- Create a simple result table.
   CREATE TABLE IF NOT EXISTS dw_order_count_test (
       order_date STRING,
       total_count BIGINT
   ) 
   PARTITIONED BY (ds STRING); -- Use ds as the partition key to store the daily statistics.

Step 2: Create a periodic workflow

1. Go to the Workspaces page in the DataWorks console. In the top navigation bar, select a desired region. Find the desired workspace and choose Shortcuts > Data Studio in the Actions column.

   The button labeled "Data Development" opens an older version of Data Studio. Do not click it.

2. Click in the left navigation bar, and to the right of Project Directory, click > Create Workflow to go to the Create Workflow page.

3. In the Create Workflow dialog box, set Scheduling Type to Periodic Scheduling, and enter a name such as minimal_daily_demo.

Step 3: Orchestrate the workflow

1. On the workflow canvas, drag a Zero-Load Node from the component panel on the left and name it start_node.

   A virtual node only defines the starting point of a business process and does not perform any actual execution.

2. Drag a MaxCompute SQL node onto the canvas and name it count_orders.

3. Click the dot at the bottom of the start_node node and drag a line to the dot at the top of the count_orders node. This connects the two nodes.

Step 4: Develop the node code

1. Double-click the count_orders node to open the code editor.

2. Write the business logic for the node. This example uses simulated data.

   -- bizdate is a scheduling variable that is defined in the scheduling configuration.
   INSERT OVERWRITE TABLE dw_order_count_test PARTITION (ds='${bizdate}')
   SELECT 
       '${bizdate}' as order_date,
       COUNT(*) as total_count
   FROM (SELECT 1 as id UNION ALL SELECT 2 as id) t; -- Simulated data

   For more information about node development, see MaxCompute SQL node.

3. Click the Save button at the top of the node editor.

Step 5: Configure scheduling and parameters

1. Return to the workflow canvas. In the panel on the right, click the Scheduling Settings > Scheduling time tab:

   1. Set Scheduling Frequency to Day.

   2. Set Scheduling time to 00:05. This schedules the workflow to run at 00:05 every day.

2. In the editor for the count_orders node, go to the Scheduling Settings > Scheduling Parameters section on the right. Add a parameter with the Parameter name bizdate and the Parameter Value $[yyyymmdd-1]. This expression represents the day before the current date.

Step 6: Debug the node and workflow

1. Debug the count_orders node:

   1. Configure debug parameters: On the right of the node editor, click Debug Configuration.

      1. In the Compute Resource section, select the MaxCompute compute engine that you prepared in Step 1.

      2. In the Script Parameters section, specify a value for Value Used in This Run. The default value is the day before the current date.

   2. Run the debug task: Click the Run button in the toolbar. This runs the node with the debug parameters that you specified in Debug Configuration.

   3. After you confirm that the node runs as expected, click Sync to Scheduling in the upper-right corner.

2. Debug the entire workflow:

   1. Return to the workflow canvas and click the icon in the top toolbar.

   2. In the dialog box that appears, enter a Value Used in This Run for the workflow variables. For example, if the current date is 20260120, DataWorks replaces the bizdate variable with 20260119.

Step 7: Deploy to production

1. On the workflow canvas, click the button in the top toolbar.

2. In the deployment panel, the system checks dependencies and configurations. After the checks pass, click Start Release Production and select Full Publishing.

3. After the deployment is successful, go to Operation and Maintenance Center and verify that the workflow appears in the list of periodic tasks.

Node and workflow orchestration

Configure scheduling dependencies

A scheduling dependency is a core mechanism that ensures the sequential execution, consistency, and accuracy of data processing. It connects isolated tasks into an orderly data pipeline by requiring that both the scheduled time is reached and all upstream tasks have completed successfully. When you orchestrate nodes within a workflow, scheduling dependencies are created between them by default. In addition to these default dependencies, you can configure more complex dependencies.

For more information, see Scheduling dependency.

Parameter design and flow

The new version of Data Studio in DataWorks supports a four-level parameter passing mechanism and allows for flexible, dynamic data transfer across nodes by using context parameters. The levels are ordered from the narrowest to the broadest scope:

Configure the scheduling cycle

The scheduling cycle can be set only at the workflow level. Nodes inside a workflow can only have a delayed execution time set, which is calculated as an offset from the workflow's defined scheduled time.

Single-node debugging

Single-node debugging validates a single node's code logic, such as an SQL statement, a Python script, or a data integration sync task. This mode runs only the selected node and does not trigger its upstream or downstream dependencies.

1. In the Debug Configuration panel on the right side of the node, configure the following parameters:

   Parameter	Description
   Compute engine	Select a bound compute engine. If none are available, select Create Compute Engine from the drop-down list. Important Ensure network connectivity between the compute engine and the resource group. For more information, see Network connectivity solutions.
   Resource group	Select a resource group that passed connectivity tests when the compute engine was bound. Some nodes support configuring a custom image on the resource group to extend the runtime environment.
   (Optional) Dataset	Some nodes, such as Shell and Python, support using datasets to access unstructured data stored in OSS or NAS.
   (Optional) Script parameters	When you configure the content of a node, define variables by using the ${parameter name} format. You must configure the Parameter name and Parameter Value in the Script Parameters section. When the task runs, the variable is dynamically replaced with its actual value. For more information, see Source and expressions of scheduling parameters.
   (Optional) Associated role	Some nodes, such as Shell and Python, support configuring an associated role to access resources from other cloud products.

2. In the toolbar above the node editor, click Save and then Run to run the node task.

3. View the run log and check the results at the bottom of the node editor.

Workflow debugging

Workflow debugging focuses on validating data dependencies, parameter passing, and execution order across multiple nodes in an end-to-end pipeline. After you finish single-node debugging, you can debug part of the pipeline or the entire workflow.

1. On the workflow's DAG canvas, click the Run button in the top toolbar. Alternatively, to validate a partial pipeline, right-click a node and select Run to this node or Run from this node.

2. In the dialog box, assign temporary values to all node variables in the workflow, such as ${bizdate}.

3. The system executes the nodes sequentially, following the dependencies defined in the DAG. You can monitor node statuses in real time on the canvas and click any node to view its run log.

4. Click the Return button on the left side of the canvas to return to the workflow development view.

   The run history on the right displays all debug run records for the workflow.

Manage workflow nodes

To enhance data development flexibility and efficiency, DataWorks lets you import existing standalone nodes into a workflow for orchestration. You can also move nodes out of a workflow or between different workflows. These features make node reuse, workflow refactoring, and modular management simpler and more effective.

Import an existing node into a workflow

The Import Node feature lets you add existing standalone nodes from your project to the current workflow canvas for rapid reuse.

1. Double-click the target workflow to open the canvas editor.

2. In the left component panel, switch to the Import Node tab.

3. This panel lists all available standalone nodes. You can filter and search by Node Type, Path, or Node Name to quickly find the node you need.

4. After you find the node, drag it onto the canvas to complete the import.

Move a node out of a workflow

You can move a node out of a workflow to make it a standalone node or move it directly to another workflow.

• Move out of a workflow to make it a standalone node

  Use this feature when you need to decouple a node from a workflow.

  1. In Workspace Directories or on the workflow canvas, right-click the target node.

  2. From the context menu, select Move out of Workflow.

  3. In the confirmation dialog box, select the destination path for the node and confirm.

• Move to another workflow

  Use this feature when you need to refactor your business process by moving a node to a different workflow.

  1. In Workspace Directories or on the workflow canvas, right-click the target node.

  2. From the context menu, select Move to another workflow.

  3. In the list that appears, select the target workflow and confirm.

     You can also move a node by dragging it directly to the target workflow.

Clone a workflow

Cloning a workflow creates a new, independent, and complete copy of an existing workflow, including all its internal nodes, code, and dependencies. In Project Directory, right-click the target workflow and select Cloning.

Cloning can introduce significant risks because it duplicates nearly all configurations. Before you commit and publish a cloned workflow, perform an "environment isolation" check by verifying the following configurations:

1. Output tables and target data sources: The cloned workflow's code will write to the exact same target tables as the original. You must modify the target table names in your code (for example, from ods_user_table to dev_ods_user_table) or update the node's output configuration. This prevents multiple tasks from writing to the same table, which can cause data conflicts.

2. Upstream dependencies: Check the workflow's upstream dependencies. By default, the cloned workflow still depends on the original upstream tasks. Ensure this is the intended behavior to prevent a disorganized data pipeline or unnecessary task runs.

3. Parameter configuration: Check the custom parameters of the workflow and its internal nodes, particularly those for date partitions (such as ${bizdate}) and input/output paths, to ensure they are correct in the new environment.

You can also clone individual nodes within a workflow, but copying nodes across different workflows is not supported.

Version management

Version management automatically records every change to a workflow, letting you view and compare historical versions and quickly revert to any previous state. This is a key feature for ensuring task stability and supporting team collaboration.

The Version panel is on the right side of the workflow canvas. This panel displays two types of records. Understanding their differences is crucial:

• Development Records: A new record is created each time you click Save on the canvas. It acts as a snapshot during the development process to prevent accidental code loss and does not affect any tasks running in production.

• Publishing Records: A new record is created after the workflow is successfully published to the production environment. This is the version that the scheduling system actually executes. When issues occur with production tasks, you typically use Publishing Records to perform a rollback.

Typical use cases

• Quick failure rollback: When a production task fails due to a code change, you can use the revert feature to instantly restore the workflow to the last stable Publishing Record, minimizing downtime.

• Change auditing and tracing: To determine when a piece of logic was modified and by whom, use the version list and compare features to trace every code change.

• Code recovery: If you accidentally delete some logic without saving, you can revert to the most recent Development Record to recover it.

Important notes

• Reverting overwrites the current development area: A revert operation overwrites all code and configurations on the current canvas with the content from the selected historical version. Before you revert, back up any uncommitted changes by copying them to a local text editor.

• Republish after reverting: A revert operation only restores the code to the development environment. The restored version is not automatically published to production. You must manually publish the workflow for the rollback to take effect in the production environment.

Publishing and operations

• Publish a node or workflow: After you finish developing your workflow, publish it to the production environment. The nodes in your development environment will generate corresponding periodic tasks in production. For details, see Publish a node or workflow.

• Node and workflow operations: In the production environment, a workflow runs periodically according to its scheduling configuration. Go to Operation and Maintenance Center to monitor the scheduling status of the periodic workflow and perform related operational tasks. For details, see: Basic operational tasks for periodic tasks, Operational tasks for Data Backfill instances.

Task execution mechanism

In a scheduling scenario, a workflow's overall success status is affected by the run status of its internal tasks.

A workflow succeeds or fails based on the final states of its internal nodes:

• Failed: If any critical node fails, the entire workflow instance is typically marked as Failed.

• Frozen/Suspended: If a node within the workflow is manually frozen or suspended and it is an upstream dependency for subsequent nodes, the pipeline is interrupted, and the entire workflow is also set to Failed.

• merge node: If a workflow uses a merge node, it can still return Succeeded even if a node in an upstream branch fails, as long as the merge node's logic evaluates to success. Design the upstream checks for the merge node carefully to prevent silent failures.

Special scenarios:

• If you freeze a Data Backfill instance of a workflow task, the workflow instance is set to Succeeded.

• In a Data Backfill scenario, if the system determines that a task cannot be executed, the workflow is set to Failed.