In common data development scenarios, the code of different types of nodes may be subject to change from time to time. You must dynamically modify the values of some parameters, such as the date and time, based on the requirement changes and time changes. In this case, you can use the scheduling parameter configuration feature of DataWorks.

After relevant parameters are configured, auto triggered nodes can automatically parse the code to obtain required data. Configurable parameters in DataWorks are classified into system parameters and custom parameters. We recommend that you use custom parameters.

Parameter types

Configurable scheduling parameters in DataWorks are classified into system parameters and custom parameters.
  • You can directly reference system parameters in the code.
  • To use custom parameters, assign variables as values to the parameters and reference the variable names in the code.
Parameter type Applicable to Configuration method Parameter value Example
System parameters: including bdp.system.bizdate and bdp.system.cyctime All nodes Reference the parameters in code by using the following formats: ${bdp.system.bizdate} and ${bdp.system.cyctime}. None None
Non-system parameters: custom parameters (recommended) ODPS SQL nodes and sync nodes Reference the parameters in code by using the following formats: ${key1} and ${key2}. Assign values to the parameters by using the following format: key1=value1 key2=value2. Both constant parameters and variables are supported:
  • Constant parameters: key1="abc" key2=1234.
  • Variables: key1=${yyyymmdd}, the value of which is calculated based on the value of the bdp.system.bizdate parameter.

    key2=$[yyyy-mm-dd hh24:mi:ss], the value of which is calculated based on the value of the bdp.system.cyctime parameter.

PyODPS nodes Add a dictionary object named args to the global variable: args=['key1'] args=['key2']. key1 and key2 are variables.
Shell nodes Reference the parameters in code by using the following formats: $1, $2 .... Assign values to the parameters by using the following format: value1 value2. Both constant parameters and variables are supported:
  • Constant parameters: "abc" 1234
  • Variables: ${yyyymmdd}, the value of which is calculated based on the value of the bdp.system.bizdate parameter.

    $[yyyy-mm-dd hh24:mi:ss], the value of which is calculated based on the value of the bdp.system.cyctime parameter.

The following figure shows how to assign values to system parameters and custom parameters of an ODPS SQL node, and how to call the parameters in the code. The left section is the code compilation area, and the right section is the parameter configuration area.Assign a value

You can click the Scheduling configuration tab in the right-side navigation pane, and assign values to customer parameters in the Parameters field in the Basic properties section. When you configure the parameters, you may use the following formats:

time①=②$[yyyymmdd③hh24:mi:ss] and time1=$[yyyymmdd]④time2=$[hh24:mi:ss]. You must replace the symbol ④, not ③, with a space.
Notice
  • You do not need to add a space before or after the equal sign (=). Namely, you do not need to replace the symbols ① and ② with spaces.
  • The value of a parameter cannot contain spaces. Therefore, you cannot replace the symbol ③ with a space.
  • When you configure multiple parameters, separate the parameters with spaces. Therefore, you must replace the symbol ④ with a space.

System parameters

DataWorks provides the following system parameters:
  • ${bdp.system.cyctime}: the scheduled time to run an instance. Default format: yyyymmddhh24miss.

    This parameter can specify the hour and minutes of the scheduled time. The value of this parameter is the same as that of the cyctime parameter.

  • ${bdp.system.bizdate}: the timestamp of data to be analyzed by an instance. Default format: yyyymmdd. The default data timestamp is one day before the scheduled time. The value of this parameter is the same as that of the bizdate parameter.

The relationship between the two system parameters is: bdp.system.cyctime = bdp.system.bizdate + 1. The default data timestamp is one day before the scheduled time.

Custom parameters

DataWorks supports the following types of custom parameters: constant parameters, built-in system parameters bizdate and cyctime, ${...}, and $[...].
Note For more information about how to reference parameters for different types of nodes, see Parameter types.
The following subsections describe the built-in system parameters bizdate and cyctime and custom parameters in the ${…} and$[…] formats.
  • Built-in system parameters
    Parameter Description
    $jobid The ID of the workflow to which a node belongs. Example: jobid=$jobid.
    $nodeid The ID of a node. Example: nodeid=$nodeid.
    $taskid The instance ID of a node. Example: taskid=$taskid.
    $bizdate The timestamp of data to be analyzed by a node, in the format of yyyymmdd.

    This parameter is widely used. By default, the value of this parameter is one day before the scheduled time to run a node.

    $cyctime The scheduled time to run a node, in the format of yyyymmddhh24miss.

    If no scheduled time is configured for a node scheduled by day, $cyctime is set to 00:00 of the day. The time is accurate to seconds. This parameter is usually used for nodes scheduled by hour or minute. Example: cyctime=$cyctime.

    $gmtdate The current date, in the format of yyyymmdd.

    By default, the value of this parameter is the current date. During retroactive data generation, the input value is the data timestamp plus one day.

    $bizmonth The month of the data timestamp, in the format of yyyymm.
    • If the month of a data timestamp is the current month, the value of $bizmonth is the month of the data timestamp minus 1.
    • Otherwise, the value of $bizmonth is the month of the data timestamp.
  • ${...} custom parameters

    You can customize a time format based on the value of $bizdate, where yyyy indicates the four-digit year, yy indicates the two-digit year, mm indicates the month, and dd indicates the day. You can use any combination of these parameters, for example, ${yyyy}, ${yyyymm}, ${yyyymmdd}, and ${yyyy-mm-dd}.

    $bizdate is accurate to the day. Therefore, ${...} can specify only the year, month, or day.

    The following table describes how to specify other intervals based on $bizdate.
    Interval Expression
    N years later ${yyyy+N}
    N years before ${yyyy-N}
    N months later ${yyyymm+N}
    N months before ${yyyymm-N}
    N weeks later ${yyyymmdd+7*N}
    N weeks before ${yyyymmdd-7*N}
    N days later ${yyyymmdd+N}
    N days before ${yyyymmdd-N}
    N days after the specified date ${yyyymmdd+N}
    N days before the specified date ${yyyymmdd-N}
    N years after the specified year, in the format of yyyy ${yyyy+N}
    N years before the specified year, in the format of yyyy ${yyyy-N}
    N years after the specified year, in the format of yy ${yy+N}
    N years before the specified year, in the format of yy ${yy-N}
  • $[...] custom parameters

    You can customize a time format based on the value of $cyctime, where yyyy indicates the four-digit year, yy indicates the two-digit year, mm indicates the month, dd indicates the day, hh24 indicates the hour in the 24-hour format, hh indicates the hour in the 12-hour format, mi indicates the minutes, and ss indicates the seconds. You can use a combination of these parameters, for example, $[yyyymmdd], $[yyyy-mm-dd], $[hh24miss], $[hh24:mi:ss], and $[yyyymmddhh24miss].

    $cyctime is accurate to seconds. Therefore, $[...] can specify the hour, minutes, or seconds.

    The following table describes how to specify other intervals based on $cyctime.
    Interval Expression
    N years later $[add_months(yyyymmdd,12*N)]
    N years before $[add_months(yyyymmdd,-12*N)]
    N months later $[add_months(yyyymmdd,N)]
    N months before $[add_months(yyyymmdd,-N)]
    N weeks after the specified date $[yyyymmdd+7*N]
    N weeks before the specified date $[yyyymmdd-7*N]
    N days after the specified date $[yyyymmdd+N]
    N days before the specified date $[yyyymmdd-N]
    N hours later $[hh24miss+N/24]
    N hours before $[hh24miss-N/24]
    N minutes later $[hh24miss+N/24/60]
    N minutes before $[hh24miss-N/24/60]

    The following section uses an ODPS SQL node as an example and describes how to assign values to custom parameters for the node in the Scheduling configuration pane of the node configuration tab.

    Assume that the current date is November 1, 2019, and the node is scheduled to run at 00:00 every day. The following table lists various custom parameters and their values.
    Parameter Sample code Assigned value Replaced value
    ${yyyymmdd} pt=${datetime1} datetime1=${yyyymmdd} datetime1=20191031
    $[yyyymmddhh24miss] pt=${datetime2} datetime2=$[yyyymmddhh24miss] datetime2=201911010000
    $bizdate: the data timestamp. pt=${datetime3} datetime3=$bizdate datetime3=20191031
    $cyctime: the scheduled time, which is accurate to seconds. pt=${datetime4} datetime4=$cyctime datetime4=201911010000
    $gmtdate: the scheduled time, which is accurate to the day. pt=${datetime5} datetime5=$gmtdate datetime5=20191101
    $bizmonth: the month of the data timestamp. pt=${datetime6} datetime6=$bizmonth If the month of the data timestamp is the current month, the value of the $bizmonth parameter is the previous month. If the month of the data timestamp is not the current month, the value of the $bizmonth parameter is the month of the data timestamp. Examples:
    • If the current date is November 1, 2019, the value of datetime6 is 201910.
    • If the selected data timestamp is November 2, 2019, the value of datetime6 is 201910.
    • If the selected data timestamp is October 31, 2019, the value of datetime6 is 201910.

Differences between the time parameters configured by using $[] and ${}

  • $bizdate: the data timestamp. By default, the value of this parameter is one day before the scheduled time to run a node.
  • $cyctime: the scheduled time to run a node. If no scheduled time is configured for a node scheduled by day, $cyctime is set to 00:00 of the day. The time is accurate to seconds. This parameter is usually used for nodes scheduled by hour or minute.

    For example, the scheduled time yyyy-mm-dd 00:30:00 indicates 00:30 on the current day.

  • If a time parameter is configured by using ${}, $bizdate is used as the benchmark for running nodes. During retroactive data generation, the time parameter is replaced with the data timestamp selected.
  • If a time parameter is configured by using $[], $cyctime is used as the benchmark for running nodes. The time is calculated in the same way as the time in Oracle. During retroactive data generation, the time parameter is replaced with the data timestamp selected plus one day.

    For example, if the data timestamp is set to 20190720 for retroactive data generation, cyctime is replaced with 20190721.

Assume that the value of the $cyctime parameter is 20190720103000. The time parameters that are configured by using $[] have the following values:
  • $[yyyy] = 2019, $[yy] = 19, $[mm] = 07, $[dd] = 20, $[yyyy-mm-dd] = 2019-07-20, $[hh24:mi:ss] = 10:30:00, and $[yyyy-mm-dd] $[hh24:mi:ss] = 2019-07-20 10:30:00
  • $[hh24:mi:ss-1/24] = 09:30:00
  • $[yyyy-mm-dd] $[hh24:mi:ss -1/24/60] = 2019-07-20 10:29:00
  • $[yyyy-mm-dd] $[hh24:mi:ss -1/24] = 2019-07-20 09:30:00
  • $[add_months(yyyymmdd,-1)] = 2019-06-20
  • $[add_months(yyyymmdd,-12*1)] = 2019-07-20
  • $[hh24] = 10
  • $[mi] = 30
Examples of the time parameters configured by using ${}:
  • The code of an ODPS SQL node includes pt=${datetime}, and the parameter configured for the node is datetime=${yyyy-mm-dd}. If the node is run on July 20, 2019, the value of ${yyyy-mm-dd} is 2019-07-20.
  • The code of an ODPS SQL node includes pt=${datetime}, and the parameter configured for the node is datetime=${yyyymmdd-2}. If the node is run on July 20, 2019, the value of ${yyyymmdd-2} is 20190718.
  • The code of an ODPS SQL node includes pt=${datetime}, and the parameter configured for the node is datetime=${yyyymm-2}. If the node is run on July 20, 2019, the value of ${yyyymm-2} is 201905.
  • The code of an ODPS SQL node includes pt=${datetime}, and the parameter configured for the node is datetime=${yyyy-2}. If the node is run on July 20, 2019, the value of ${yyyy-2} is 2017.

If you assign values to multiple parameters when you configure an ODPS SQL node, separate the parameters with spaces. For example, enter startdatetime=$bizdate enddatetime=${yyyymmdd+1} starttime=${yyyy-mm-dd} endtime=${yyyy-mm-dd+1}.

Test scheduling parameters

On the node configuration tab, you can click the Run or Run with Arguments icon in the top navigation bar. The configured values are assigned to the variables in the code. You cannot check whether the parameter values configured on the Properties tab are as expected.

The parameter values are replaced only when the node is run in Operation Center. To test the parameter values on the node configuration tab, click the Run Smoke Test in Development Environment icon in the top navigation bar. In the dialog box that appears, you can enter a data timestamp to simulate automatic node scheduling and obtain the replaced values of scheduling parameters at the specified data timestamp.

Note
  • When you run a smoke test for a node in the development environment, if you modify the scheduling parameters of the node, you must save the node and commit the node in the development environment again. In this case, the code is updated.
  • When you run a smoke test in the development environment, a fee is charged for the generated test instance.

View the replaced parameter values in Operation Center

  1. On the DataStudio page, click the Icon icon in the upper-left corner and choose All Products > Operation Center.
  2. In the left-side navigation pane, choose Cycle Task Maintenance > Cycle Task.

The page that appears displays all the auto triggered node instances. You can view the dependencies and scheduling configurations of each auto triggered node instance in the production environment.

After you deploy a node to the production environment, you can check whether the configurations of the node meet your expectations. For example, you can view the values of the scheduling parameters.

For auto triggered nodes, you can check whether the replaced values of the scheduling parameters of the instances generated for each node every day meet your expectations. The values of the scheduling parameters of the instances generated for auto triggered nodes are replaced when the instances are generated. If an error occurs in the business logic, right-click the node in the directed acyclic graph (DAG) and select View Node Details. On the Node Information page, check the replaced values of the scheduling parameters of ancestor and descendant nodes.

Assume that you modify the scheduling parameters of an auto triggered node on the DataStudio page, commit the node, and then deploy the node to the production environment. In this case, you must check whether the parameter values of the node meet your expectations in the Execution parameters field on the Node Information page.
Notice
  • Do not rerun instances that have run or failed. After you modify the scheduling parameters of a node, instances are generated for the node again.
  • If the recurrence is modified for a node whose instances are generated immediately after the node is deployed, the modified scheduling parameters take effect for instances that are not run in the production environment. For more information about how to generate instances immediately after nodes are deployed, see Immediate instance generation.

Examples

  • The following figure shows how to configure the scheduling parameters of an ODPS SQL. This example also applies to configuring the scheduling parameters of a Data Integration node.Assign a value
    • If you use system parameters, you can directly reference them in the code, instead of configuring them in the Parameters field in the Scheduling configuration pane.
    • If you use custom parameters, you must assign values in the format of Variable name=Custom parameter in the Parameters field in the Scheduling configuration pane, and reference the variable names in the code.
  • The parameter configuration procedure of a Shell node is similar to that of an ODPS SQL node, except that the variable naming rules are different.

    Variable names for a Shell node cannot be customized, but must follow the $1,$2,$3... format. If the number of parameters in a Shell node reaches 10, use ${10} to declare the tenth variable.

  • To prevent code intrusion, the ${param_name} string of a PyODPS node is not replaced in the code. However, before the code is run, you can add a dictionary object named args to the global variable to obtain the values of the scheduling parameters.

    For example, in the Scheduling configuration pane of the node configuration tab, enter def=${yyyymmdd} in the Parameters field in the Basic properties section.