DataWorks:Workflow

Step 1: Prepare the compute engine and result table

This step sets up the infrastructure your workflow needs.

1. In your target workspace, bind MaxCompute as a compute engine.

2. In MaxCompute, create the result table:

   -- Create a result table to store daily order counts
   CREATE TABLE IF NOT EXISTS dw_order_count_test (
       order_date STRING,
       total_count BIGINT
   )
   PARTITIONED BY (ds STRING); -- Partitioned by date

Step 2: Create a periodic workflow

This step creates the workflow container that holds all your nodes and scheduling configuration.

1. Go to the Workspaces page in the DataWorks console. In the top navigation bar, select a desired region. Find the target workspace and choose Shortcuts > Data Studio in the Actions column. > If the button shows "Data Development", it opens the old version. Do not click it.

2. Click the icon in the left-side navigation pane. To the right of Workspace Directories, click  > Create Workflow.

3. In the New Workflow dialog box, set Scheduling Type to Periodic Scheduling, enter a name (for example, minimal_daily_demo), and complete the creation.

Step 3: Orchestrate the workflow

This step defines the pipeline structure using the DAG canvas.

1. On the workflow canvas, drag a virtual node from the left component panel and name it start_node. > A virtual node defines the starting point of a workflow and performs no computation.

2. Drag a MaxCompute SQL node onto the canvas and name it count_orders.

3. Click the dot at the bottom of start_node and drag a line to the top of count_orders to create a dependency.

Step 4: Write the node code

This step adds the business logic that runs each day.

1. Double-click count_orders to open the node code editor.

2. Write the SQL logic. This example uses simulated data:

   -- bizdate is a scheduling parameter injected at runtime.
   -- $[yyyymmdd-1] resolves to the previous day's date (e.g., 20260119 when today is 20260120).
   INSERT OVERWRITE TABLE dw_order_count_test PARTITION (ds='${bizdate}')
   SELECT
       '${bizdate}' AS order_date,
       COUNT(*) AS total_count
   FROM (SELECT 1 AS id UNION ALL SELECT 2 AS id) t; -- Simulated data

   > For details on node development, see MaxCompute SQL node.

3. Click Save.

Step 5: Configure the scheduling cycle and parameters

This step tells the workflow when to run and what date value to pass to the SQL.

1. Return to the workflow canvas. In the right-side panel, go to Scheduling > Scheduling Time:

   • Set Scheduling Cycle to Daily.

   • Set Schedule Time to 00:05.

2. In the count_orders node editor, go to Scheduling Configuration > Scheduling Parameters and add:

   • Parameter Name: bizdate

   • Parameter Value: $[yyyymmdd-1] (the previous day's date)

Step 6: Debug the node and workflow

Debugging validates your code logic before the workflow goes to production.

1. Debug the node:

   1. In the node editor, click Debugging Configurations on the right side.

      • Under Compute engine, select the MaxCompute engine from Step 1.

      • Under Script Parameters, set Value Used in This Run (defaults to yesterday's date).

   2. Click Run in the toolbar. The node runs with the parameters you specified.

   3. Once the results look correct, click Sync to Scheduling to apply the debug configuration to the scheduling configuration.

2. Debug the entire workflow:

   1. On the workflow canvas, click the icon in the top toolbar.

   2. In the dialog box, enter the Current Run Value for the workflow variables. For example, if today is 20260120, set bizdate to 20260119.

Step 7: Publish to production

Publishing creates a periodic task in the production environment that runs on your defined schedule.

1. Return to the workflow and click the button in the top toolbar.

2. In the publishing panel, the system runs dependency and configuration checks. After the checks pass, click Start Deployment to Production Environment and select Full Deployment.

3. Go to the Operation Center to confirm the workflow appears in the periodic task list.

The workflow now runs every day at 00:05.

Workflow orchestration

Sub-workflows for logic reuse

When multiple workflows share the same processing steps—such as data cleaning, summary statistics, and quality checks—encapsulate those steps as a reusable sub-workflow using a SUB_PROCESS node. This avoids duplicating logic across workflows: when the shared steps change, you update only the sub-workflow.

To create and use a sub-workflow:

1. In the child_workflow workflow, set Properties to Referable. The workflow becomes a sub-workflow.

2. In the main workflow, drag a SUB_PROCESS node and select child_workflow as the referenced workflow.

Sub-workflows have the following constraints:

• Internal-only dependencies: The sub-workflow and its internal nodes cannot depend on any external tasks.

• Isolation: The sub-workflow cannot be a direct dependency for any external task.

• Passive trigger: After publishing, no scheduling instances are generated automatically. The sub-workflow executes only when called by a SUB_PROCESS node. > For details, see SUB_PROCESS node.

Scheduling dependencies

Scheduling dependencies connect isolated nodes into an ordered data pipeline. A node runs only when its scheduled time arrives and all upstream tasks complete successfully.

For the full reference, see Scheduling dependency.

When you connect nodes on the canvas, DataWorks creates scheduling dependencies between them automatically. For more complex cross-workflow or cross-cycle dependencies, configure the following dependency types:

Parameter design

Workspace parameters are available only in DataWorks Professional Edition and later.

DataWorks supports a four-level parameter system, ordered from narrowest to broadest scope. Use the appropriate level to avoid duplicating configuration across nodes.

Scheduling cycles

The scheduling cycle is set at the workflow level. Nodes inside a workflow cannot have their own independent cycle—they inherit the workflow's cycle and can only specify a Delayed Execution Time as an offset from the workflow's scheduled time.

Single-node debugging

Single-node debugging validates the internal logic of one node—such as an SQL statement, Python script, or data integration task—without triggering any upstream or downstream dependencies.

1. In the right-side panel of the node, configure Debugging Configurations:

   Parameter	Description
   Compute engine	Select a bound compute engine. If none are available, select Create Compute Engine from the drop-down list. Ensure the compute engine and Resource Group are connected. See Network connectivity solutions.
   Resource Group	Select a Resource Group that passed connectivity tests when the compute engine was bound. Some nodes support a custom image on the Resource Group to extend the runtime environment.
   Dataset (optional)	Shell and Python nodes support using datasets to access unstructured data in OSS.
   Script parameters (optional)	Define variables in node code using ${parameter_name}. Set the Parameter Name and Parameter Value here. For details, see Scheduling parameter sources and their expressions.
   Associated role (optional)	Shell and Python nodes support configuring an associated role to access resources from other cloud products.

2. Click Save, then click Run.

3. View run logs and results at the bottom of the node editor.

Workflow debugging

Workflow debugging validates data dependencies, parameter passing, and execution order across the entire pipeline. After debugging individual nodes, debug the full workflow—or a partial pipeline.

1. On the workflow's DAG canvas, click Run in the top toolbar. To validate a subset, right-click a node and select Run to This Node or Run from This Node.

2. In the dialog box, assign a temporary value for all node variables (such as ${bizdate}) for this debugging session.

3. The system executes nodes sequentially from top to bottom, following the dependencies defined in the DAG. Monitor node statuses on the canvas in real time and click any node to view its run log.

4. Click Back on the left side of the canvas to return to the development state. The run history on the right shows all debugging run records.

Workflow node management

DataWorks lets you import standalone nodes into a workflow, move nodes out of a workflow, or transfer nodes between workflows—without rebuilding them from scratch.

Import an existing node into a workflow

1. Double-click the target workflow to open its canvas editor.

2. In the left component panel, switch to the Import Node tab.

3. Filter by Node Type, Path, or Node Name to find the target node.

4. Drag the node onto the canvas.

Move a node out of a workflow

• Convert to a standalone node: Right-click the node in the directory tree or canvas, select Move out of Workflow, choose the target path, and confirm.

• Move to another workflow: Right-click the node, select Move to another workflow, select the target workflow, and confirm. You can also drag the node directly to the target workflow.

Clone a workflow

Cloning a workflow creates a complete, independent copy including all internal nodes, code, and dependencies. In Workspace Directories, right-click the target workflow and select Clone.

Before committing and publishing a cloned workflow, review the following to prevent data conflicts and pipeline issues:

1. Output table and target data source: A cloned workflow writes to the same target table as the original. Update the table name in the code (for example, from ods_user_table to dev_ods_user_table) to prevent multiple tasks from writing to the same table.

2. Upstream dependencies: The cloned workflow still depends on the original upstream tasks by default. Confirm whether this is the intended behavior.

3. Parameter configuration: Check custom parameters—especially those involving date partitions (such as ${bizdate}) and input/output paths—to make sure they are correct in the new environment.

You can also clone individual nodes within a workflow, but you cannot copy nodes across different workflows.

Version management

Version management automatically records every change to a workflow. From the Versions panel on the right side of the workflow canvas, you can view, compare, and revert to any previous version.

The Versions panel shows two types of records:

• Development Records: Created each time you click Save. This is a work-in-progress snapshot and does not affect production.

• Publishing Records: Created when the workflow is successfully published to production. This is what the production scheduling system executes.

Typical use cases

Before reverting:

• Reverting overwrites the current development area: All code and configurations on the canvas are replaced with the historical version. If you have unsaved changes, back them up locally before reverting.

• Republish after reverting: Reverting only restores the development state. Manually publish the workflow for the rollback to take effect in production.

Publishing and operations

• Publish: After developing a workflow, publish it to the production environment. The nodes in the development environment generate corresponding periodic tasks in production. For details, see Publish a node or workflow.

• Operations: The workflow runs periodically based on its scheduling configuration. Go to the Operation Center to monitor scheduling status and manage tasks. For details, see Basic operational tasks for periodic tasks and Operational tasks for Data Backfill instances.

Workflow instance states

A workflow's overall status reflects the final state of its internal nodes. Understanding how the system determines workflow state helps you diagnose issues faster.

Special scenarios:

• Freezing a Data Backfill instance of a workflow task sets the workflow instance to Succeeded.

• In a Data Backfill scenario, if the system determines a task cannot be executed, the workflow is set to Failed.