Most node types — including SQL nodes and batch synchronization nodes — use the same method to define and reference scheduling parameters. Two node types work differently: PyODPS nodes retrieve parameters through a dictionary object instead of string replacement, and general Shell nodes use positional variables instead of named ones. This topic provides configuration examples for each node type.
SQL nodes and batch synchronization nodes
SQL nodes (such as ODPS SQL) and batch synchronization nodes use the same method to configure scheduling parameters. This method also applies to most other node types. This topic uses an ODPS SQL node as an example to show how to assign values to built-in variables and custom parameters and reference them in code.
Some nodes may not support scheduling parameters. For information about whether a specific node type supports scheduling parameters, see the documentation for that node type.
As shown in the preceding figure, you assign values to parameters in the parameter assignment area, and then reference built-in variables var1 and var3, custom parameters var2 and var4, and constant var5 in the code area. The following examples show how to assign values:
-
Assign the business date to built-in variable var1:
var1=$bizdate -
Assign the scheduled time to built-in variable var3:
var3=$cyctime -
Assign the business date to custom parameter var2:
var2=${yyyymmdd} -
Assign the scheduled time to custom parameter var4:
var4=$[yyyymmddhh24:mi:ss] -
Assign the constant abc to var5:
var5=abc
To obtain a relative time based on the scheduled time (for example, the previous hour or N minutes ago), you can apply offset calculations to the scheduled time parameter ($[...]): use N/24 for hour offsets and N/24/60 for minute offsets, where N is the number of hours or minutes to offset. Hour and minute offsets are supported only for the scheduled time. The business date (${...}) does not support these offsets. The following examples use "get the previous hour" as an example:
-
Get the previous hour of the scheduled time (full timestamp in yyyymmddhh24miss format):
$[yyyymmddhh24miss-1/24] -
Get only the previous hour (24-hour format):
$[hh24-1/24] -
Get the hour, minute, and second of the previous hour:
$[hh24miss-1/24] -
Get the date in yyyymmdd format for the previous hour:
$[yyyymmdd-1/24] -
Get the date for the previous day and previous hour:
$[yyyymmdd-1-1/24] -
Get the scheduled time minus 15 minutes:
$[yyyymmddhh24miss-15/24/60]
Hour and minute offset calculations may result in cross-day scenarios. For the complete list of supported assignment formats and information about cross-day handling, see Supported scheduling parameter formats.
PyODPS nodes
To prevent code injection, PyODPS nodes do not support defining variables by using the ${param_name} string replacement syntax directly in code. Before you run the code, you must retrieve scheduling parameters from the args global variable, which is of type dict (dictionary object). The parameter assignment method is the same as for other node types.
As shown in the preceding figure, you assign values to parameters in the parameter assignment area, and then reference built-in parameter var1 and custom parameters var2 and var3 in the code area. After the dictionary object is added, the parameters become args['var1'], args['var2'] , and args['var3']. The following examples show how to assign values:
-
Assign the business date to built-in parameter var1:
var1=$bizdate -
Assign the business date to custom parameter var2:
var2=${yyyymmdd} -
Assign the business date to custom parameter var3:
var3=$[yyyymmdd]
General Shell node configuration example
Variables in general Shell nodes cannot have custom names. They must be named as $1, $2, $3, and so on (numbered sequentially in ascending order). When the number of parameters exceeds 10, use the ${10} syntax to declare the variable.
As shown in the preceding figure, you assign values to parameters in the parameter assignment area, and then reference built-in parameter $1 and custom parameters $2 and $3 in the code area. The following examples show how to assign values:
General Shell nodes support only expression-based parameter assignment. Separate multiple parameter assignments with spaces. The values correspond to parameters in the order they are defined. For example, in the preceding figure, the first parameter defined in the Shell node is $1, so the first value in the parameter assignment area ($bizdate) is assigned to $1.
-
Assign the business date to built-in parameter $1:
$bizdate -
Assign the business date to custom parameter $2:
${yyyymmdd} -
Assign the scheduled time to custom parameter $3:
$[yyyymmdd]
For more supported assignment formats for scheduling parameters, see Supported scheduling parameter formats. For the complete workflow of configuring and using scheduling parameters, see Configure and use scheduling parameters.
Batch synchronization example
For typical scenarios of using scheduling parameters in Data Integration, see Use scheduling parameters in batch synchronization tasks.