Configure and use node context parameters - DataWorks - Alibaba Cloud Documentation Center

Node context parameters are the core mechanism in DataWorks for passing dynamic data between task nodes. They allow an upstream node (producer) to pass its output to one or more downstream nodes. Downstream nodes can then reference these values in their code to dynamically adjust their behavior based on the output of the upstream node. This process greatly enhances the flexibility and automation of workflows.

How it works

Node context parameters pass values between nodes. An upstream node (producer) defines an output parameter. A downstream node (consumer) then references that parameter to retrieve the value.

Upstream node (producer): Generates a value and sets it as an output parameter. You can provide a value in one of two ways:
- Pass a constant or variable: In the Node output parameters section of the upstream node, you can define a parameter and assign it a value. The value can be a constant, such as 'abc', or a system context variable, such as ${status}.
- Pass an assignment result: The system captures the result of the last query in the node's code, such as SELECT 'table_A';, and assigns it to a built-in output parameter named outputs. The value of this parameter is then passed to downstream nodes. The value depends on the result of the code at runtime. Assignment nodes and some SQL nodes support this method.
Downstream node (consumer): Retrieves and uses the value provided by the upstream node.
1. Configure input parameters: In the Node input parameters section of the downstream node, add an input parameter. Set the value source of the parameter to the output parameter of the upstream node.
2. Establish a scheduling dependency: After you configure the input parameter, the system automatically creates a same-cycle scheduling dependency from the downstream node to the upstream node.
3. Reference the parameter in code: In the code of the downstream node, reference the retrieved value using the format ${InputParameterName}. For example, if an upstream node passes the value table_A, the code SELECT * FROM ${input}; in the downstream node becomes SELECT * FROM table_A; at runtime.

Limits

Edition: Some nodes support the Add Assignment Parameter feature, which is used to pass query results. This feature requires DataWorks Standard Edition or higher.
Node types: The node types that support the Add Assignment Parameter feature are: EMR Hive, EMR Spark SQL, ODPS Script, Hologres SQL, AnalyticDB for PostgreSQL, ClickHouse SQL, and database node types.

Procedure

Step 1: Configure an upstream node to output parameters

Log on to the DataWorks console. After you switch to the destination region, choose Data Development & O&M > Data Development in the navigation pane on the left. From the drop-down list, select the desired workspace and click Go to Data Studio.
In the Data Studio pane, double-click the target upstream node to open its editor page.
On the right side of the canvas, click Scheduling. In the Input and Output Parameters section, configure the Node Output Parameters in one of the following ways.

Method 1: Pass a constant or variable

In the Node Output Parameters section, click Add Parameter.

Configure the parameter.

Parameter

Description

Parameter Name

A custom name for the output parameter, for example, my_param.

Parameter Value

The value of the parameter. The following types are supported:

Constant: such as hello.
System context variable: such as ${status}.
Scheduling parameter: such as $bizdate, or a custom scheduling parameter such as ${...} or $[...].

Method 2: Pass an assignment result

Use an assignment node
An assignment node (the upstream node) supports MaxCompute SQL, Python 2, and Shell. It automatically assigns the last query or output result to the node's output parameter (outputs). Downstream nodes can then reference this parameter to retrieve the output of the assignment node. For more information, see Assignment node.
Use an assignment parameter
In a node that supports assignment parameters, perform the following steps:
1. In the Node Output Parameters section, click Add Assignment Parameter.
2. The system automatically adds an output parameter named outputs. You do not need to configure this parameter. Its value is the result of the last query in the node's code.
3. Click Save.
  Note
  After you click Add Assignment Parameter, the assignment parameter passes the query result of the current node to downstream nodes that reference it. If the result is empty, the current node is not blocked, but the downstream nodes that reference the parameter might fail.
The usage is similar to how the MaxCompute language is used in an assignment node.

Important

Output parameters can be deleted. Before you delete an output parameter, ensure that it is not used by any downstream nodes. Otherwise, downstream tasks that depend on it will fail.

Step 2: Configure a downstream node to use parameters

Configure input parameters
1. Open the editor page for the downstream node and navigate to the Scheduling > Input and Output Parameters section. In the Node input parameters section, click Add Parameter.
2. Configure the input parameter. Select an output parameter from an upstream node as the Value Source and specify a Parameter Value for the current node.
3. In the toolbar, click Save to add the parameter.
Establish a dependency
After you bind the input parameter to the output parameter of an upstream node, the system automatically adds a same-cycle dependency to that upstream node. No manual configuration is required.
Reference the parameter
In the code of the downstream node, you can reference the parameter using the format ${InputParameterName}.
The following example shows how to reference the input parameter param in a Shell node:
```
echo "The value from upnode is ${param}"
```
If the upstream node passes an assignment result to the downstream node, the parameter value is typically a two-dimensional array or a comma-separated one-dimensional array. You can retrieve values from the array as follows:
- If the upstream node is an SQL node (two-dimensional array):
  - Row: ${param}.
  - Cell: ${param[j]}.
- If the upstream node is a Python/Shell node (one-dimensional array):
  Row: ${param}.
Indexes are 0-based.

Step 3: Debug and run

Context parameters are passed according to the scheduling order only in recurring instances that are triggered by a workflow. If you run a downstream node independently, it cannot retrieve upstream parameters, which will cause the task to fail. When you debug, you must start from the upstream node and run the nodes in the order of the business process.

Return to the workflow. In the toolbar above the workflow, click Run. Or, right-click the downstream node and select Run to This Node.
In the generated DAG instance, click a node to view its run log and verify that the result is as expected.

System context variables

System Variable	Description
${projectId}	Project ID.
${projectName}	MaxCompute project name.
${nodeId}	Node ID.
${gmtdate}	The start of the day (00:00:00) on which the instance is scheduled to run. The format is yyyy-MM-dd 00:00:00.
${taskId}	Task instance ID.
${seq}	The sequence number of the task instance. This number indicates the order of this instance among other instances of the same node that run on the same day.
${cyctime}	The scheduled time for the instance.
${status}	The status of the instance: SUCCESS or FAILURE.
${bizdate}	Data timestamp.
${finishTime}	The time when the instance finished running.
${taskType}	The run type of the instance: NORMAL, MANUAL, PAUSE, SKIP (dry-run), UNCHOOSE, or SKIP_CYCLE (cycle dry-run).
${nodeName}	Node name.