Assign values to cross-node output parameters - Dataphin

The cross-node parameter feature allows nodes to pass messages to one another. Output nodes, such as SQL, Shell, and Python nodes, can output parameters. Input nodes can receive and use these parameters. This process eliminates the complexity and inefficiency of passing messages through public storage. This topic describes how to assign values to cross-node output parameters.

Background

In Dataphin, node dependencies form a directed acyclic graph (DAG). Previously, nodes in a DAG could only pass status information. This meant that the running status of an ancestor node would affect its descendant nodes. The system could not interpret the content of a node or pass this content to descendant nodes through the DAG. If a descendant node needed to use the output of an ancestor node, the ancestor node had to write the output to public storage. The descendant node then had to read the output from public storage. Data tables were the most common public storage location. However, passing messages through public storage has the following drawbacks:

The process of writing to and reading from public storage is complex, and some types of nodes cannot perform these operations. For example, a logical table node cannot independently read messages from public storage because its computing logic is determined entirely by its field computing logic.
Writing messages to and reading them from public storage is inefficient. If multiple descendant nodes need to use the same message, each node must read the message independently. Additionally, if the storage location or format of the message changes, all descendant nodes must adjust their reading methods.
This transfer method is cumbersome, especially because the messages passed are usually very small.

To resolve these issues, Dataphin introduced cross-node parameters for passing messages between nodes.

Output nodes: Nodes that output parameters to pass messages. Only Shell, Python, and SQL compute nodes support parameter output. Examples include MaxCompute SQL, Hive SQL, and AnalyticDB for PostgreSQL SQL.
Input nodes: Nodes that receive and use parameters. An output node can connect to multiple input nodes. Any direct descendant node of an output node can reference the cross-node parameters of that output node.

Instructions for assigning values to cross-node output parameters

After you declare cross-node output parameters, you can assign values to them in your code. For more information, see Best practices for cross-node parameters.

Important

Only Shell, Python, and SQL compute nodes can define cross-node output parameters.
Cross-node variable parameters can be used for the input parameters of integration nodes, compute nodes, and standard or hierarchical logical dimension and fact tables.

Task type	Assignment statement	Right-click menu shortcut
Shell Task	In the code, use the following command to assign a value to a cross-node parameter: `setv "{parameter_name}" {parameter_value}` Example: `setv "output_variable" 123` Important The setv "{variable_name}" {value} command assigns a value to the cross-node parameter {variable_name}. setv is a Shell function and must follow Shell syntax. setv is a system-reserved function. Avoid naming conflicts.	In the editor, right-click and select Set Cross-node Parameter.
Python Task	In the code, use the following command to assign a value to a cross-node parameter: `setv("{parameter_name}", {parameter_value})` Example: `setv("output_variable", 123)` Important The setv("{variable_name}", {value}) command assigns a value to the cross-node parameter {variable_name}. setv is a Python function and must follow Python syntax. setv is a system-reserved function. Avoid naming conflicts.	In the editor, right-click and select Set Cross-node Parameter.
SQL Job	`set dataphin.task.result.type = dp_context_param;` `select "value" as variable_name; or another query statement. The system uses the first row of the query result. The field name is the parameter name, and the field value is the parameter value.`	In the editor, right-click and select Set Cross-node Parameter. Note AnalyticDB for PostgreSQL SQL nodes do not support the right-click menu shortcut. Enter the commands manually.