DataWorks:Configure node context

Limits

• Edition: DataWorks Standard Edition and later support Add assignment parameter for passing query results.

• Node types: Only some node types can pass query results through assignment parameters, including EMR Hive, EMR Spark SQL, ODPS Script, Hologres SQL, AnalyticDB for PostgreSQL, ClickHouse SQL, and MySQL nodes.

How it works

To pass a value, define an output parameter in an upstream node (provider) and reference it in a downstream node (consumer).

• Upstream node (provider): Generates a value and exposes it as an output parameter in one of two ways:

  • Configure a regular output parameter: In the upstream node's Current Node Output Parameters section, define a parameter and assign a constant (such as 'abc') or a system context variable (such as ${status}).

  • Configure an assignment output parameter: The system captures the query result from the last line of the node code (such as SELECT 'table_A';) and assigns it to the built-in output parameter outputs. The parameter value is passed to downstream nodes and depends on the code's result.Assignment nodes and some SQL nodes support this method.

• Downstream node (consumer): Receives and uses the value from the upstream node.

  1. Create a scheduling dependency: Configure the dependency on the upstream node.

  2. Configure an input parameter: In the downstream node's Current Node Input Parameters section, add an input parameter whose source is an upstream output parameter.

  3. Reference it in code: In downstream node code, reference the received value as ${input_parameter_name}. For example, if the upstream node passes table_A, the downstream code SELECT * FROM ${input}; becomes SELECT * FROM table_A; at runtime.

Open the parameter configuration area

1. Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.

2. In the Data Studio panel, double-click the node to open its editor.

3. Click Scheduling. In the Input and Output Parameters area, click on the right to open the node context parameter settings.

Configure output parameters

In the Input and Output ParametersConfigure the current node's output parameters for your business scenario in the

• Configure a regular output parameter: Pass a constant or variable configured in this node to downstream nodes.

• Configure an assignment output parameter: Pass this node's query result to downstream nodes as an output parameter.

After configuring Output Parameters, find Scheduling Dependency > Output Name of Current Node and locate the Add Mode whose value is Added Automatically and note its Output Name. Downstream nodes can use this Output Name to bind a dependency on this node.

Configure an input parameter

Configure the current node's input parameters in the node context settings to use output parameters from upstream nodes.

1. For the upstream node, Configure output parameters at runtime.

2. configure an upstream/downstream dependency.

   1. On the right side of the downstream node editor, click Scheduling at runtime.

   2. In the Scheduling Dependency > Parent Nodes and note its Node Output. In the dialog box, enter and select the Configure output parameters from the upstream node when you Output Name at runtime.

   3. Click Add column.

3. Configure the downstream node's input parameters.

   In the Input and Output ParametersUnder Add, configure the current node's input parameters using the table below:

   Parameter	Configuration
   No.	The system assigns and increments the number automatically.
   Parameter name	Define a name for the input parameter. You can reference the value passed from the upstream node by this name in the node editor.
   Value Source	The value comes from the upstream node's output parameter and equals its actual output result.
   Description	You can add a brief description.
   Parent Node ID	Automatically parsed from the upstream node.
   Source	Source of the current parameter. Options include Added Automatically, Analyze Code or Add manually at runtime.
   Operation	and Save, Modify or Delete.

4. After configuring the parameters, click Operation in the Save column.

Use input parameters in downstream nodes

Reference a defined input parameter like any other system variable, using ${input_parameter_name}. The following Shell node example shows how.

#!/bin/bash
#*************************************************************************#
##author: xxx est.aliyunid.com
##create xxx 10:06:02
#*************************************************************************#

echo 'input_from_up_const:' ${input_from_up_const}
echo 'input_from_up_var:' ${input_from_up_var}

Supported global variables

• System variables

  System variable	Description
  ${projectId}	Project ID.
  ${projectName}	MaxCompute project name.
  ${nodeId}	Node ID.
  ${gmtdate}	00:00:00 on the day of the instance's scheduled time, in the format yyyy-MM-dd 00:00:00 at runtime.
  ${taskId}	Task instance ID.
  ${seq}	Instance sequence number among the instances of the same node on that day.
  ${cyctime}	Instance scheduled time.
  ${status}	Instance status: SUCCESS or FAILURE.
  ${bizdate}	Business date.
  ${finishTime}	Instance finish time.
  ${taskType}	Instance run type: NORMAL, MANUAL, PAUSE, SKIP, UNCHOOSE, or SKIP_CYCLE.
  ${nodeName}	Node name.

• For other parameter settings, see Scheduling parameter formats at runtime.