DataWorks:Use workflow parameters

Supported node types

Currently, workflow parameters are supported only for the following node types: ODPS SQL, PyODPS 3, EMR Spark Shell, EMR Spark, EMR Shell, EMR Hive, EMR MR, EMR Presto, EMR Trino, EMR Spark SQL, EMR Spark Streaming, EMR Streaming SQL, and Hologres SQL. The UI shows the most current list of supported node types.

Usage notes

• After you save a workflow parameter, DataWorks loads the latest configuration whenever you open a supported node, placing the node into an editable state. For the new configuration to affect scheduled runs, you must then commit and deploy the node.

• If a workflow parameter and a node-level parameter have the same name, the workflow parameter's value overrides the node's.

• When you assign a value to a workflow parameter, the Value/Expression field does not support the ${key}:value format. For example, var:1 and ${var}:1 are invalid.

Workflow parameter settings

1. Go to the DataStudio page.

   Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.

2. Open the workflow parameter settings.

   In DataStudio, find your workflow in the tree on the Data Development pane and double-click it to open the workflow editor, where you can then access the Workflow Parameters settings. The left side of the tab shows a list of common nodes such as ODPS SQL, PyODPS 3, EMR Hive, EMR Spark SQL, and Hologres SQL. The toolbar at the top provides Run and Commit buttons. In the upper-right corner, you can find links to Deploy and Operation Center.

Configure workflow parameters

The method for configuring parameters is the same for all supported node types. This feature is recommended when you need to assign a value to the same parameter for multiple nodes in a workflow. This topic uses an example to demonstrate how to use a workflow parameter to change the value of the ReplaceMe parameter in an ODPS SQL node from 20230613 to 20230615.

1. Configure the node-level parameter.

   1. Open the node editor.

      Double-click the ODPS SQL node within your target workflow to open its editor.

   2. Configure the node parameter.

      Set the value of the ODPS SQL node parameter to 20230613. In the code editor, write the SELECT ${ReplaceMe}; statement. Click the Properties tab in the right-side pane. In the Parameters table, add a parameter with the name ReplaceMe and the value 20230613. Click Scheduling Parameter Preview. In the dialog box that appears, confirm that the parameter for each instance is correctly set to ReplaceMe=20230613. For more information about how to configure node parameters, see Configure scheduling parameters.

2. Use a workflow parameter to override the node-level parameter value.

   1. Double-click the target workflow, and in the right-side navigation pane of the workflow canvas, click Workflow Parameters.

   2. In the Workflow Parameters dialog box, configure the parameter.

      This example sets the value of the workflow parameter to 20230615. After you save the configuration, the parameter value in the ODPS SQL node is replaced with 20230615. You can also add multiple parameters as needed.

      Note

      • If a workflow parameter and a node-level parameter have the same name, the workflow parameter's value overrides the node's.

      • When you assign a value to a workflow parameter, the Value/Expression field does not support the ${key}:value format. For example, var:1 and ${var}:1 are invalid.

      In the Workflow Parameters dialog box, set the Parameter Name to ReplaceMe and its value to 20230615, and then click Save. For this example, where the ODPS SQL code is SELECT ${ReplaceMe};, the original parsed value for the ReplaceMe parameter is 20230613. If you click Scheduling Parameter Preview, you can see that the ReplaceMe parameter for the next three instances takes the workflow parameter value 20230615, which verifies that the workflow parameter has taken effect.

Verify the configuration

After you configure the workflow parameter, run the workflow to verify that its value correctly overrides the node-level value.

1. Run the workflow and assign a value to the variable.

   When you run a workflow that uses workflow parameters, you must manually assign a value to each parameter at runtime. For more information, see debugging procedure.

   1. In the toolbar of the workflow editor, click the Run icon.

   2. In the Enter parameters. dialog box, assign the value 20230615 to the ReplaceMe variable.

2. Check the run result.

   On the workflow canvas, right-click the ODPS SQL node and select View Log. The log appears at the bottom of the page and shows the parameter assignment. The node's parameter value has been changed from 20230613 to 20230615, confirming that the workflow parameter took effect and the run was successful. In the run log, the line SKYNET_PARAVALUE=ReplaceMe=20230615 indicates that the workflow parameter was applied successfully.

   2023-06-14 16:19:57 INFO Current task status:RUNNING
   2023-06-14 16:19:57 INFO Start execute shell on node cn-shanghai.475270149050624.1626085771455.
   2023-06-14 16:19:57 INFO Current working dir /home/admin/alisatasknode/taskinfo/20230614/phoenix/16/19/55/mm8wog2kwtj8czebilx1dbcp
   2023-06-14 16:19:57 INFO Full Command ..
   2023-06-14 16:19:57 INFO ——————————
   2023-06-14 16:19:57 INFO /opt/taobao/tbdpapp/odpswrapper/odpswrapper.py /home/admin/alisatasknode/taskinfo//20230614/phoenix/16/19/55/mm8wog2kwtj8czebilx1dbcp//275011_10728491_503348330 ReplaceMe=20230615
   2023-06-14 16:19:57 INFO ——————————
   2023-06-14 16:19:57 INFO List of passing environment ..
   2023-06-14 16:19:57 INFO ——————————
   2023-06-14 16:19:57 INFO SKYNET_BUSINESS_NAME=WorkFlow Parameters:
   2023-06-14 16:19:57 INFO SKYNET_ENDPOINT=http://service.cn.maxcompute.aliyun-inc.com/api:
   2023-06-14 16:19:57 INFO SKYNET_PTYPE=10:
   2023-06-14 16:19:57 INFO SKYNET_ACTIONID=1:
   2023-06-14 16:19:57 INFO TASK_WHITE_LIST=oapi.dingtalk.com:443,47.102.239.120:3306:
   2023-06-14 16:19:57 INFO SKYNET_RERUN_MODE=1:
   2023-06-14 16:19:57 INFO SKYNET_FLOW_PARAVALUE={"ReplaceMe":"20230615"}:
   2023-06-14 16:19:57 INFO SKYNET_ONDUTY=204288958124218159:
   2023-06-14 16:19:57 INFO SKYNET_SYSTEMID=dev:
   2023-06-14 16:19:57 INFO CALC_ENGINE_IDENTIFIER=saochen_mc_0508_dev:
   2023-06-14 16:19:57 INFO SKYNET_SOURCEID=700028402970:
   2023-06-14 16:19:57 INFO SKYNET_PARAVALUE=ReplaceMe=20230615:
   2023-06-14 16:19:57 INFO SKYNET_TASKID=708xxx:
   2023-06-14 16:19:57 INFO SKYNET_TENANT_ID=283789484710656:

Commit and deploy the workflow

After you verify that the workflow parameter configuration is correct, you must commit and deploy the workflow to apply the changes to scheduled runs.

1. Commit the workflow.

   When you commit, select the nodes that you want to include and provide a description of the changes.

   Note

   If this feature is enabled, a reviewer must approve the committed code before it can be deployed. For more information about this feature, see code review.

   Click the Commit button in the toolbar. In the dialog box that appears, select the nodes to commit, enter a Change Description, and configure Code Review and Force Modify as needed. You can also select the Ignore alerts for inconsistent I/O checkbox. Click OK.

2. Deploy the workflow.

   If your workspace is in standard mode, you must click the Deploy button in the upper-right corner after a successful commit. For more information, see Deploy nodes.