Configure scheduling parameters for different types of nodes - DataWorks

You can reference the method that is used to configure scheduling parameters for SQL nodes, such as MaxCompute SQL nodes, to define custom variables and assign scheduling parameters to the custom variables for other types of nodes, excluding common Shell nodes and PyODPS nodes. Specific configurations differ for common Shell nodes and PyODPS nodes. This topic provides examples on how to configure scheduling parameters for different types of nodes.

SQL nodes and batch synchronization nodes

Scheduling parameters are configured for SQL nodes and batch synchronization nodes using a method that is suitable for most types of nodes. You can reference the method to configure scheduling parameters for most types of nodes. In this example, a MaxCompute SQL node is used. This section describes how to assign custom parameters to custom variables, and how to reference the custom variables in code.

Note

Specific nodes do not support scheduling parameters. For more information about whether a type of node supports scheduling parameters, see the documentation of the type of node.

-- Obtain the data timestamp.
SELECT '${var1}' '${var2}';
-- Obtain the scheduling time.
SELECT '${var3}' '${var4}';
-- Obtain a constant.
SELECT '${var5}';

Assign values to the parameters in the parameter assignment area. Then, you can reference the built-in system variables `var1` and `var3`, the custom parameters `var2` and `var4`, and the constant `var5` in the code. The following examples show how to assign the values:

var1: Specify $bizdate as var1. This variable is used to obtain the data timestamp. $bizdate is a built-in system parameter.
var3: Specify $cyctime as var3. This variable is used to obtain the scheduling time. $cyctime is a built-in system parameter.
var2: Specify ${yyyymmdd} as var2. This variable is used to obtain the data timestamp.
var4: Specify $[yyyymmddhh24:mi:ss] as var4. This variable is used to obtain the scheduling time.
var5: Assign the constant abc to this variable.

For more information about how to configure and use scheduling parameters, see the Configure scheduling properties for a node section of the "Scheduling properties" topic. For more information about how to assign custom parameters to custom variables, see Supported formats of scheduling parameters.

PyODPS nodes

To avoid code changes, a PyODPS node does not allow you to replace defined variables with strings that are in the ${param_name} format in the code. However, you can add a dictionary object named args to the global variable before the code is run. This way, the PyODPS node can obtain the values of scheduling parameters from the dictionary object.

# Custom parameter: var1 gets a built-in parameter
a =args['var1']
print (format(a))
# Custom parameter: var2 gets a value in the ${...} format
b =args['var2']
print (format(b))
# Custom parameter: var3 gets a value in the $[...] format
c =args['var3']
print (format(c))

Assign values in the parameter assignment area. Then, you can reference the built-in parameter `var1` and the custom parameters `var2` and `var3` in the code. After you add the dictionary object, the parameters are referenced as args['var1'], args['var2'], and args['var3']. The following examples show how to assign the values:

var1: Specify $bizdate as var1. This variable is used to obtain the data timestamp. $bizdate is a built-in system parameter.
var2: Specify ${yyyymmdd} as var2. This variable is used to obtain the data timestamp.
var3: Specify $[yyyymmdd] as var3. This variable is used to obtain the data timestamp.

Shell Node Configuration Example

You are not allowed to specify custom variable names for common Shell nodes. The variables must be named based on their ordinal numbers, such as $1, $2, and $3. If the number of parameters reaches or exceeds 10, use ${Number} to declare the excess variables. For example, use ${10} to declare the tenth variable.

# Custom parameter: Get a built-in parameter
echo $1
# Custom parameter: ${...}
echo $2
# Custom parameter: $[...]
echo $3

Assign values in the parameter assignment area. Then, you can reference the parameters `$1`, `$2`, and `$3` in the code. The following examples show how the values are assigned:

Note

For common Shell nodes, you can assign custom parameters to custom variables only using expressions. Separate multiple custom parameters with spaces. Make sure that you assign the custom parameters to the custom variables in the same order in which the custom variables are defined. For example, the first parameter $bizdate that you enter in the Scheduling Parameters section is assigned to the first variable $1.

$1: Specify $bizdate as $1. This variable is used to obtain the data timestamp. $bizdate is a built-in system parameter.
$2: Specify ${yyyymmdd} as $2. This variable is used to obtain the data timestamp.
$3: Specify $[yyyymmdd] as $3. This variable is used to obtain the scheduling time.