In common data development scenarios, you must dynamically assign values, such as the date and time, to variables in the code of different types of nodes based on the recurrence settings before the code is run. DataWorks provides the feature of configuring scheduling parameters to meet the requirements of the business scenarios.

Background information

Scheduling parameters are replaced only when a node is scheduled to run. When you click Run (①) or Run with Arguments (②) on the node configuration tab, scheduling parameters are not replaced with the parameter settings specified on the right side of the page. You must manually assign constants to the variables (③ and ④) in the code.

If you need to test the replacement of scheduling parameters, you must perform a smoke test in the development environment. For more information, see Test scheduling parameters.

If you have changed the variables in the code or need to re-assign values to the variables, you must click Run with Arguments.

After you configure parameters, auto triggered nodes can automatically parse the code to obtain required values. Parameters are classified into system parameters and custom parameters. We recommend that you use custom parameters.

Parameter types

Scheduling parameters are classified into built-in system variables and custom parameters.
  • You can directly use built-in system variables in your code, including ${bdp.system.bizdate} and ${bdp.system.cyctime}.
  • To use custom parameters, assign variables as values to the parameters and reference the variable names in the code.
Parameter type Applicable node Reference method Parameter value assignment Example in the parameter editing box
Built-in system variables: include bdp.system.bizdate that indicates the data timestamp and bdp.system.cyctime that indicates the scheduled time. All nodes Reference the ${bdp.system.bizdate} and ${bdp.system.cyctime} variables in the code. You do not need to assign values. N/A
Non-system parameters: custom parameters. We recommend that you use custom parameters. ODPS SQL nodes and sync nodes Reference ${key1} and ${key2} in the code. Assign values to scheduling parameters: key1=value1 key2=value2. Value assignment examples:
  • Assign constants to parameters: key1="abc" key2=1234.
  • Assign variables to parameters: key1=${yyyymmdd} -- data timestamp.

    key2=$[yyyy-mm-dd hh24:mi:ss] -- scheduled time.

PyODPS nodes Add a dictionary object named args to the global variable: args=['key1'] args=['key2']. key1 and key2 are variable names.
Shell nodes Reference variables in the code: $1, $2 ... Assign values to scheduling parameters: value1 value2. Value assignment examples:
  • Assign constants to parameters: "abc" 1234.
  • Assign variables to parameters: ${yyyymmdd} -- data timestamp.

    $[yyyy-mm-dd hh24:mi:ss] -- scheduled time.

The following figure shows how to assign values to system parameters and custom parameters of an ODPS SQL node, and how to reference the parameters in the code. The left section is the code editing area, and the right section is the parameter configuration area.Assign values

You can assign values to custom parameters in the Properties > General > Arguments field. Take note of the following items when you configure scheduling parameters:

Examples: time①=②$[yyyymmdd③hh24:mi:ss] and time1=$[yyyymmdd]④time2=$[hh24:mi:ss]. The symbols ①, ②, ③, and ④ indicate the locations where spaces may be added.
Notice
  • You do not need to add a space before or after the equal sign (=) for a scheduling parameter. Therefore, you cannot add spaces in the locations specified by the symbols ① and ②.
  • The value of a scheduling parameter cannot contain spaces. Therefore, you cannot add a space in the location specified by the symbol ③.
  • Separate two scheduling parameters with a space. Therefore, you must add a space in the location specified by the symbol ④.

System parameters

DataWorks provides the following system parameters:
  • ${bdp.system.cyctime}: the scheduled time to run an instance. Default format: yyyymmddhh24miss.

    You can specify the hour and minutes only for the scheduled time. The value is the same as that of the custom parameter cyctime.

  • ${bdp.system.bizdate}: the timestamp of data to be analyzed by an instance. By default, the data timestamp is one day before the scheduled time. Default format: yyyymmdd. The value is the same as that of the custom parameter bizdate.

The scheduled time and data timestamp meet the following formula: bdp.system.cyctime = bdp.system.bizdate + 1. By default, the data timestamp is one day before the scheduled time.

Custom parameters

DataWorks supports the following custom parameters: constant parameters, built-in system parameters bizdate and cyctime, ${...}, and $[...].
Note For more information about how to reference parameters for different types of nodes, see Parameter types.
Description of built-in system parameters bizdate and cyctime and custom parameters ${…} and $[…] :
  • Built-in system parameters
    Parameter Description
    $jobid The ID of the workflow to which a node belongs. Example: jobid=$jobid.
    $nodeid The ID of the node. Example: nodeid=$nodeid.
    $taskid The instance ID of the node. Example: taskid=$taskid.
    $bizdate The data timestamp, in the format of yyyymmdd.

    This parameter is widely used. By default, the data timestamp is one day before the scheduled time.

    $cyctime The scheduled time to run a node, in the format of yyyymmddhh24miss.

    If no scheduled time is configured for a node scheduled by day, the scheduled time is set to 00:00 of the day by default. The time is accurate to seconds. This parameter is usually used for nodes scheduled by hour or minute. Example: cyctime=$cyctime.

    $gmtdate The current date, in the format of yyyymmdd.

    By default, the value of this parameter is the current date. During retroactive data generation, the input value is the data timestamp plus one day.

    $bizmonth The month of the data timestamp, in the format of yyyymm.
    • If the month of a data timestamp is the current month, the value of $bizmonth is the month of the data timestamp minus 1.
    • If the month of a data timestamp is not the current month, the value of $bizmonth is the month of the data timestamp.
  • ${...}custom parameters

    You can customize a time format based on the $bizdate parameter, where yyyy indicates the four-digit year, yy indicates the two-digit year, mm indicates the month, and dd indicates the day. You can use any combination of these parameters, for example, ${yyyy}, ${yyyymm}, ${yyyymmdd}, and ${yyyy-mm-dd}.

    $bizdate is accurate to the day. Therefore, ${...} can specify only the year, month, or day.

    The following table describes how to specify time by adding or removing an interval.
    Interval that you want to add or remove Method
    N years later ${yyyy+N}
    N years before ${yyyy-N}
    N months later ${yyyymm+N}
    N months before ${yyyymm-N}
    N weeks later ${yyyymmdd+7*N}
    N weeks before ${yyyymmdd-7*N}
    N days later ${yyyymmdd+N}
    N days before ${yyyymmdd-N}
    N days after the specified date ${yyyymmdd+N}
    N days before the specified date ${yyyymmdd-N}
    N years after the specified year, in the format of yyyy ${yyyy+N}
    N years before the specified year, in the format of yyyy ${yyyy-N}
    N years after the specified year, in the format of yy ${yy+N}
    N years before the specified year, in the format of yy ${yy-N}
  • $[...]custom parameters

    You can customize a time format based on the $cyctime parameter, where yyyy indicates the four-digit year, yy indicates the two-digit year, mm indicates the month, dd indicates the day, hh24 indicates the hour in the 24-hour format, hh indicates the hour in the 12-hour format, mi indicates the minutes, and ss indicates the seconds. You can use a combination of these parameters, for example, $[yyyymmdd], $[yyyy-mm-dd], $[hh24miss], $[hh24:mi:ss], and $[yyyymmddhh24miss].

    $cyctime is accurate to seconds. Therefore, $[...] can specify the hour, minutes, or seconds.

    The following table describes how to specify time by adding or removing an interval.
    Interval that you want to add or remove Method
    N years later $[add_months(yyyymmdd,12*N)]
    N years before $[add_months(yyyymmdd,-12*N)]
    N months later $[add_months(yyyymmdd,N)]
    N months before $[add_months(yyyymmdd,-N)]
    N weeks later $[yyyymmdd+7*N]
    N weeks before $[yyyymmdd-7*N]
    N days later $[yyyymmdd+N]
    N days before $[yyyymmdd-N]
    N hours later $[hh24miss+N/24]
    N hours before $[hh24miss-N/24]
    N minutes later $[hh24miss+N/24/60]
    N minutes before $[hh24miss-N/24/60]

    The following section describes how to assign values to parameters of an ODPS SQL node on the Properties tab on the right side of the node configuration tab.

    Assume that the current date is November 1, 2019, and the node is scheduled to run at 00:00 every day. The following table compares the value assignment methods of different custom parameters.
    Parameter type Reference method in code Assigned value Replaced value
    ${yyyymmdd} pt=${datetime1} datetime1=${yyyymmdd} datetime1=20191031
    $[yyyymmddhh24miss] pt=${datetime2} datetime2=$[yyyymmddhh24miss] datetime2=201911010000
    $bizdate: the data timestamp. pt=${datetime3} datetime3=$bizdate datetime3=20191031
    $cyctime: the scheduled time, which is accurate to seconds. pt=${datetime4} datetime4=$cyctime datetime4=20191101000000
    $gmtdate: the scheduled time, which is accurate to the day. pt=${datetime5} datetime5=$gmtdate datetime5=20191101
    $bizmonth: the month of the data timestamp. pt=${datetime6} datetime6=$bizmonth If the month of the data timestamp is the current month, the value of the $bizmonth parameter is the previous month. If the month of the data timestamp is not the current month, the value of the $bizmonth parameter is the month of the data timestamp. Examples:
    • If the current date is November 1, 2019, the value of datetime6 is 201910.
    • If the selected data timestamp is November 2, 2019, which is in the current month, the value of datetime6 is 201910.
    • If the selected data timestamp is October 31, 2019, which is not in the current month, the value of datetime6 is 201910.

Differences between custom parameters $[] and ${}

  • $bizdate: the data timestamp. By default, the value of this parameter is one day before the current date.
  • $cyctime: the scheduled time to run a node. If no scheduled time is configured for a node scheduled by day, the scheduled time is set to 00:00 of the day by default. The time is accurate to seconds. This parameter is usually used for nodes scheduled by hour or minute.

    For example, the scheduled time yyyy-mm-dd 00:30:00 indicates 00:30 on the current day.

  • If a parameter is configured by using {}, $bizdate is used as the benchmark for running nodes. During retroactive data generation, the parameter is replaced with the data timestamp selected.
  • If a parameter is configured by using [], $cyctime is used as the benchmark for running nodes. The time is calculated in the same way as the time in Oracle. During retroactive data generation, the parameter is replaced with the data timestamp selected plus one day.

    For example, if 20190720 is selected as the data timestamp for retroactive data generation, cyctime is replaced with 20190721.

The following examples assume that the value of the $cyctime parameter is 20190720103000.
  • $[yyyy] = 2019, $[yy] = 19, $[mm] = 07, $[dd] = 20, $[yyyy-mm-dd] = 2019-07-20, $[hh24:mi:ss] = 10:30:00, and $[yyyy-mm-dd] $[hh24:mi:ss] = 2019-07-20 10:30:00
  • $[hh24:mi:ss-1/24] = 09:30:00
  • $[yyyy-mm-dd] $[hh24:mi:ss -1/24/60] = 2019-07-20 10:29:00
  • $[yyyy-mm-dd] $[hh24:mi:ss -1/24] = 2019-07-20 09:30:00
  • $[add_months(yyyymmdd,-1)] = 2019-06-20
  • $[add_months(yyyymmdd,-12*1)] = 2018-07-20
  • $[hh24] = 10
  • $[mi] = 30
Examples of custom parameters ${}:
  • The code of an ODPS SQL node includes pt=${datetime}, and the scheduling parameter configured for the node is datetime=${yyyy-mm-dd}. If the node is run on July 20, 2019, the ${yyyy-mm-dd} parameter is replaced with 2019-07-19 when the node is run.
  • The code of an ODPS SQL node includes pt=${datetime}, and the scheduling parameter configured for the node is datetime=${yyyymmdd-2}. If the node is run on July 20, 2019, the ${yyyymmdd-2} parameter is replaced with 20190717 when the node is run.
  • The code of an ODPS SQL node includes pt=${datetime}, and the scheduling parameter configured for the node is datetime=${yyyymm-2}. If the node is run on July 20, 2019, the ${yyyymm-2} parameter is replaced with 201905 when the node is run.
  • The code of an ODPS SQL node includes pt=${datetime}, and the scheduling parameter configured for the node is datetime=${yyyy-2}. If the node is run on July 20, 2019, the ${yyyy-2} parameter is replaced with 2017 when the node is run.

If you assign values to multiple parameters when you configure an ODPS SQL node, separate the parameters with spaces. Example: startdatetime=$bizdate enddatetime=${yyyymmdd+1} starttime=${yyyy-mm-dd} endtime=${yyyy-mm-dd+1}.

Test scheduling parameters

When you can click Run or Run with Arguments on the node configuration tab, the configured values are assigned to the variables in the code. You cannot check whether the parameters configured on the Properties tab are as expected.

Scheduling parameters are replaced only when a node is scheduled to run. To run a test on the node configuration tab, click the Run Smoke Test in Development Environment icon. You can enter a data timestamp to simulate automatic node scheduling and obtain the replaced values of scheduling parameters at the specified data timestamp.

Note
  • When you run a smoke test for a node in the development environment, if you modify the scheduling parameters, you must save the node and commit the node in the development environment again. In this case, the code is updated.
  • When you run a smoke test in the development environment, you are charged for the generated test instance.

View the replaced parameter values in Operation Center

  1. On the current page, click the Icon icon in the upper-left corner and choose All Products > Operation Center.
  2. In the left-side navigation pane, choose Cycle Task Maintenance > Cycle Task.

You can view the dependencies and scheduling configurations of nodes in the production environment on the Cycle Task page.

Each time a node is deployed to the production environment, you can check whether the configurations of the node meet your expectations on this page. For example, you can view the values of the scheduling parameters.

For auto triggered nodes, you can check whether the replaced values of the scheduling parameters of the instances generated for each node every day meet your expectations. The scheduling parameters of the instances generated for auto triggered nodes are replaced when the instances are generated. If an error occurs in the business logic, right-click the instance name and select View node details to check the replaced values of the scheduling parameters of ancestor and descendant nodes.

Assume that you modify the scheduling parameters of an auto triggered node on the DataStudio page, commit the node, and then deploy the node to the production environment. In this case, you must check whether the parameter values of the node meet your expectations in the Execution parameters field.
Notice
  • Do not rerun instances that have run or failed. After you modify the scheduling parameters of a node, instances are generated for the node again.
  • If the recurrence is modified for a node whose instances are generated immediately after the node is deployed, the modified scheduling parameters take effect for instances that are not run in the production environment. For more information, see Immediate instance generation.

Reference examples for different nodes

  • In the example shown in the following figure, an ODPS SQL node is used. This example also applies to configuring the scheduling parameters of a Data Integration node.Assignment
    • If you use system parameters, you can directly reference them in the code, instead of assigning values to variables on the Properties tab.
    • If you use custom parameters, you must assign values to variables in the format of Variable name=Custom parameter on the Properties tab, and reference the variable names in the code.
  • The parameter configuration procedure of a Shell node is similar to that of an ODPS SQL node, except that the rules are different.

    Variable names for a Shell node cannot be customized, but must follow the $1,$2,$3... format. If the number of parameters reaches 10, use ${10} to declare the variable.

  • To make its code unobtrusive, a PyODPS node does not replace strings in the ${param_name} format in the code. However, you can add a dictionary object named args to global variables before the code is run. This way, the PyODPS node can obtain the values of scheduling parameters from the dictionary object.

    For example, on the Properties tab of the node configuration tab, enter def=${yyyymmdd} in the Arguments field in the General section. The following figure shows how to obtain the value of the parameter in the code.