Scheduling parameters are used during the running of DataWorks nodes. The scheduling parameters are automatically replaced with specific values based on the business time of the nodes and the value formats of the scheduling parameters. This implements dynamic parameter settings during the running of nodes. You can configure scheduling parameters for a node in the Arguments field of the General section on the Properties tab. This topic describes the types of scheduling parameters and the precautions for using scheduling parameters. It also provides some typical configuration examples of scheduling parameters.

Precautions

  • You can configure scheduling parameters for an auto triggered node and deploy it to the production environment. This way, the scheduling system can automatically replace the scheduling parameters based on the business time of the node. You can click Run or Run with Arguments on the DataStudio page to perform a test for the node. However, the scheduling system does not automatically replace the parameters during the test. You must manually assign values to the scheduling parameters that you want to reference in the code of the node.

    For example, you configure the scheduling parameter bdp.system.bizdate for an auto triggered node. During the running of the node, the scheduling system automatically replaces this parameter with the specific business time. However, if you click Run on the DataStudio page to perform a test for the node, the parameter is not automatically replaced. In this case, you must manually assign a constant to the parameter in the code of the node. For more information, see Test scheduling parameters in the development environment.

  • If you want to check whether a scheduling parameter is replaced as expected, you must perform smoke testing in the development environment. For more information, see Test scheduling parameters in the development environment.
  • If you changed the variables in the code or you want to re-assign constants to the variables, you must click Run with Arguments.

    Each time you click Run with Arguments to run a node, the system prompts you to assign constants to the variables in the code. This ensures that the variable values in the code are correct each time the system runs the node. If you click Run to run a node for the first time, the system prompts you to assign constants to the variables. In subsequent running, you are no longer required to assign constants to the variables, and the constants assigned the first time are used. As a result, the constants may be invalid for the subsequent running.

  • Before you use ${...} and $[...] parameters, take note of the following items:
    • If you want to specify a time that is N hours or minutes ago, use a $[...] parameter.

      ${...} parameters are accurate only to the day. Therefore, you cannot use a ${...} parameter to specify a time that is accurate to the hour, minute, or second, such as ${yyyy-mm-dd-1/24}. If you want to specify a time that is accurate to the second, we recommend that you use a $[...] parameter, such as $[yyyy-mm-dd-1-1/24].

    • If you want to specify a time that is N years or months ago, use a ${...} parameter.

      You cannot use a $[...] parameter to specify a calculation-based time, such as $[yyyy-N] or $[mm-N]. If you want to specify a calculation-based time, we recommend that you use a ${...} parameter, such as ${yyyy-N} or ${mm-N}.

Types of scheduling parameters

Scheduling parameters are classified into built-in system variables and custom parameters based on whether value assignment is required. We recommend that you use custom parameters. Types of scheduling parameters

Built-in system variables

  • Built-in system variables
    Built-in system variable Description Remarks
    ${bdp.system.bizdate} The business date on which an instance computes data. By default, the business date is one day before the time when the instance is scheduled to run. Default format: yyyymmdd. The value can be accurate to the second.
    ${bdp.system.cyctime} The time when the instance is scheduled to run. Default format: yyyymmddhh24miss. The value can be accurate to the day.

    The following formula shows the relationship between the two built-in system variables: ${bdp.system.cyctime} = ${bdp.system.bizdate} + 1. This indicates that the business date of a node is one day before the time when the node is scheduled to run.

  • Assignment and reference requirements
    You can reference built-in system variables in the code of your node without the need to assign values to them. Built-in system variables

Custom parameters: built-in system parameters

  • Parameters
    Table 1. Built-in system parameters
    Built-in system parameter Description
    $bizdate The business date of a node, in the format of yyyymmdd.

    This parameter is widely used. By default, the business date of a node is one day before the time when the node is scheduled to run.

    $cyctime The time when the node is scheduled to run, in the format of yyyymmddhh24miss.
    $gmtdate The current date, in the format of yyyymmdd.

    By default, the value of this parameter is the current date. During retroactive data generation, the input value is one day after the business date.

    $bizmonth The month of the business date, in the format of yyyymm.
    • If the month of a business date is the current month, the value of this parameter is one month before the month of the business date.
    • If the month of a business date is not the current month, the value of this parameter is the month of the business date.
    $jobid The ID of the workflow to which the node belongs.
    $nodeid The ID of the node.
    $taskid The ID of the node instance.
  • Assignment and reference requirements for MaxCompute SQL nodes, batch synchronization nodes, and PyODPS nodesAssignment requirements
    • You must assign built-in system parameters to variables in the Arguments field of the General section on the Properties tab. Separate multiple assignment equations with spaces.
    • Assign built-in system parameters in the key1=value1 key2=value2 format. key1 and key2 are custom variables. value1 and value2 are built-in system parameters. For more information about the built-in system parameters provided by DataWorks, see Table 1.
    • Reference built-in system parameters in the following ways:
      • MaxCompute SQL nodes and batch synchronization nodes

        Directly reference ${key1} or ${key2} in code. key1 and key2 are custom variables.

        Example: dt=${key1}.

      • PyODPS nodes

        Add a dictionary object named args to the global variable: args=['key1'] args=['key2']. key1 and key2 are custom variables.

    • Assignment and reference requirements for Shell nodes
      • Assign built-in system parameters in the value1 value2 format. value1 and value2 are built-in system parameters. For more information about the built-in system parameters provided by DataWorks, see #d6e262.
      • Reference built-in system parameters in the following way:

        Reference variables $1, $2 ... in code.

Custom parameters: ${...} parameters

  • Parameter definition
    ${...} parameters are custom time parameters that are based on the built-in system parameter $bizdate. You can use the following items to customize such a time parameter:
    • yyyy: indicates the four-digit year, which is the year in the value of the $bizdate parameter.
    • yy: indicates the two-digit year, which is the year in the value of the $bizdate parameter.
    • mm: indicates the month, which is the month in the value of the $bizdate parameter.
    • dd: indicates the day, which is the day in the value of the $bizdate parameter.
    You can specify one or more of the preceding items based on your requirements, such as ${yyyy}, ${yyyymm}, ${yyyymmdd}, and ${yyyy-mm-dd}.
    Note The $bizdate parameter is accurate to the day. Therefore, ${...} parameters are also accurate only to the day. Therefore, you cannot use a ${...} parameter to specify a time that is accurate to the hour, minute, or second, such as ${yyyy-mm-dd-1/24}. If you want to specify a time that is accurate to the second, we recommend that you use a $[...] parameter, such as $[yyyy-mm-dd-1-1/24].
  • Assignment and reference requirements for MaxCompute SQL nodes, batch synchronization nodes, and PyODPS nodesAssignment requirements
    • You must assign custom parameters to variables in the Arguments field of the General section on the Properties tab. After you assign custom parameters to variables, you can reference the variables in the code of your node.
    • Assign custom parameters in the key1=value1 key2=value2 format. key1 and key2 are custom variables. value1 and value2 are custom parameters. The following table lists the examples of the custom parameters.
      Interval that you want to add or remove Custom parameter
      N years later ${yyyy+N}
      N years ago ${yyyy-N}
      N months later ${yyyymm+N}
      N months ago ${yyyymm-N}
      N weeks later ${yyyymmdd+7*N}
      N weeks ago ${yyyymmdd-7*N}
      N days later ${yyyymmdd+N}
      N days ago ${yyyymmdd-N}
      N days after the specified date ${yyyymmdd+N}
      N days before the specified date ${yyyymmdd-N}
    • Reference built-in system parameters in the following ways:
      • MaxCompute SQL nodes and batch synchronization nodes

        Directly reference ${key1} or ${key2} in code. key1 and key2 are custom variables.

        Example: dt=${key1}.

      • PyODPS nodes

        Add a dictionary object named args to the global variable: args=['key1'] args=['key2']. key1 and key2 are custom variables.

  • Assignment and reference requirements for Shell nodes
    • Assign custom parameters in the value1 value2 format. value1 and value2 are custom parameters.
    • Reference built-in system parameters in the following way:

      Reference variables $1, $2 ... in code.

Custom parameters: $[...] parameters

  • Parameter definition
    $[...] parameters are custom time parameters that are based on the built-in system parameter $cyctime. You can use the following items to customize such a time parameter:
    • yyyy: indicates the four-digit year, which is the year in the value of the $cyctime parameter.
    • yy: indicates the two-digit year, which is the year in the value of the $cyctime parameter.
    • mm: indicates the month, which is the month in the value of the $cyctime parameter.
    • dd: indicates the day, which is the day in the value of the $cyctime parameter.
    • hh24 (24-hour format) and hh (12-hour format): indicates the hour, which is the hour in the value of the $cyctime parameter.
    • mi: indicates the minute, which is the minute in the value of the $cyctime parameter.
    • ss: indicates the second, which is the second in the value of the $cyctime parameter.
    You can specify one or more of the preceding items, such as $[yyyymmdd], $[yyyy-mm-dd], $[hh24miss], $[hh24:mi:ss], and $[yyyymmddhh24miss].
    Note The $cyctime parameter is accurate to the second. Therefore, $[...] parameters are also accurate to the second.
  • Assignment and reference requirements for MaxCompute SQL nodes, batch synchronization nodes, and PyODPS nodesAssignment requirements
    • You must assign custom parameters to variables in the Arguments field of the General section on the Properties tab. After you assign custom parameters to variables, you can reference the variables in the code of your node.
    • Assign custom parameters in the key1=value1 key2=value2 format. key1 and key2 are custom variables. value1 and value2 are custom parameters. The following table lists the examples of the custom parameters.
      Interval that you want to add or remove Custom parameter
      N years later $[add_months(yyyymmdd,12*N)]
      N years ago $[add_months(yyyymmdd,-12*N)]
      N months later $[add_months(yyyymmdd,N)]
      N months ago $[add_months(yyyymmdd,-N)]
      N weeks later $[yyyymmdd+7*N]
      N weeks ago $[yyyymmdd-7*N]
      N days later $[yyyymmdd+N]
      N days ago $[yyyymmdd-N]
      N hours later $[hh24miss+N/24]
      N hours ago $[hh24miss-N/24]
      N minutes later $[hh24miss+N/24/60]
      N minutes ago $[hh24miss-N/24/60]
    • Reference built-in system parameters in the following ways:
      • MaxCompute SQL nodes and batch synchronization nodes

        Directly reference ${key1} or ${key2} in code. key1 and key2 are custom variables.

        Example: dt=${key1}.

      • PyODPS nodes

        Add a dictionary object named args to the global variable: args=['key1'] args=['key2']. key1 and key2 are custom variables.

  • Assignment and reference requirements for Shell nodes
    • Assign built-in system parameters in the value1 value2 format. value1 and value2 are built-in system parameters. For more information about the built-in system parameters provided by DataWorks, see #d6e262.
    • Reference built-in system parameters in the following way:

      Reference variables $1, $2 ... in code.

Custom parameters: constant parameters

  • Constant parameters are classified into numeric constant parameters, such as 123, and character constant parameters, such as "abc".
    Note If you want to assign a constant parameter that contains a space, you must customize two variables. When you reference the two variables, use a space to combine them.
  • Assignment and reference requirements for MaxCompute SQL nodes, batch synchronization nodes, and PyODPS nodesAssignment requirements
    • You must assign custom parameters to variables in the Arguments field of the General section on the Properties tab. After you assign custom parameters to variables, you can reference the variables in the code of your node.
    • Assign constant parameters in the key1=value1 key2=value2 format. key1 and key2 are custom variables. value1 and value2 are custom constant parameters, such as key1="abc" key2=1234.
    • Reference built-in system parameters in the following ways:
      • MaxCompute SQL nodes and batch synchronization nodes

        Directly reference ${key1} or ${key2} in code. key1 and key2 are custom variables.

        Example: dt=${key1}.

      • PyODPS nodes

        Add a dictionary object named args to the global variable: args=['key1'] args=['key2']. key1 and key2 are custom variables.

  • Assignment and reference requirements for Shell nodes
    • Assign constant parameters in the value1 value2 format. value1 and value2 are custom constant parameters, such as "abc" 1234.
    • Reference built-in system parameters in the following way:

      Reference variables $1, $2 ... in code.

Comparisons of custom parameters

For example, the current date is November 1, 2019, and a node is scheduled to run at 00:00 every day. The following table compares the assignment methods of different custom parameters.
Parameter Reference method in code Assignment Replacement
${yyyymmdd} pt=${datetime1} datetime1=${yyyymmdd} datetime1=20191031
$[yyyymmddhh24miss] pt=${datetime2} datetime2=$[yyyymmddhh24miss] datetime2=201911010000
$bizdate (business date) pt=${datetime3} datetime3=$bizdate datetime3=20191031
$cyctime (time when the node is scheduled to run, which is accurate to the second) pt=${datetime4} datetime4=$cyctime datetime4=20191101000000
$gmtdate (current date, which is accurate to the day) pt=${datetime5} datetime5=$gmtdate datetime5=20191101
$bizmonth (month of the business date) pt=${datetime6} datetime6=$bizmonth If the month of the business date is the current month, the value of the $bizmonth parameter is the previous month. If the month of the business date is not the current month, the value of the $bizmonth parameter is the month of the business date.

Examples:

  • If the current date is November 1, 2019, the value of datetime6 is 201910.
  • If the selected business date is November 2, 2019, which is in the current month, the value of datetime6 is 201910.
  • If the selected business date is October 31, 2019, which is not in the current month, the value of datetime6 is 201910.

Differences between ${...} and $[...] parameters

Item ${...} parameter $[...] parameter
Benchmark The value of the $bizdate parameter is used as a benchmark for running nodes. The $bizdate parameter indicates the business date. By default, the value of this parameter is one day before the current date. The value of the $cyctime parameter is used as a benchmark for running nodes. The $cyctime parameter indicates the time when a node is scheduled to run.

For example, the value yyyy-mm-dd 00:30:00 indicates 00:30 on the current day.

Retroactive data generation The parameter is replaced with the selected business date. The time is calculated in the same way as in Oracle. During retroactive data generation, the parameter is replaced with the date that is one day after the selected business date.

For example, 20190720 is selected as the business date for retroactive data generation. In this case, cyctime is replaced with 20190721.

Time granularity Accurate to the day. Accurate to the second.

${yyyy-mm-dd-1/24} is not supported. We recommend that you use $[yyyy-mm-dd-1-1/24].

For example, the current time is 10:30:00 on July 20, 2019. The following table lists the example values of ${...} and $[...] parameters configured for a MaxCompute SQL node.
Note If you assign multiple parameters when you configure a MaxCompute SQL node, separate the parameters with spaces, such as startdatetime=$bizdate enddatetime=${yyyymmdd+1} starttime=${yyyy-mm-dd} endtime=${yyyy-mm-dd+1}.
Time ${...} parameter $[...] parameter
Year: 2019
  • Assignment: datetime=${yyyy}
  • Reference in code: pt=${datetime}
  • Replacement result: pt=${yyyy}=2019
  • Assignment: datetime=$[yyyy]
  • Reference in code: pt=$[datetime]
  • Replacement result: pt=$[yyyy]=2019
Year: 19
  • Assignment: datetime=${yy}
  • Reference in code: pt=${datetime}
  • Replacement result: pt=${yy}=19
  • Assignment: datetime=$[yy]
  • Reference in code: pt=$[datetime]
  • Replacement result: pt=$[yy]=19
Year: 2018
  • Assignment: datetime=${yyyy-1}
  • Reference in code: pt=${datetime}
  • Replacement result: pt=${yyyy-1}=2018
Not supported
Month
  • Assignment: datetime=${mm}
  • Reference in code: pt=${datetime}
  • Replacement result: pt=${mm}=07
  • Assignment: datetime=$[mm]
  • Reference in code: pt=$[datetime]
  • Replacement result: pt=$[mm]=07
Date
  • Assignment: datetime=${dd}
  • Reference in code: pt=${datetime}
  • Replacement result: pt=${dd}=19
  • Assignment: datetime=$[dd]
  • Reference in code: pt=$[datetime]
  • Replacement result: pt=$[dd]=20
Date: June 20, 2019
  • Assignment: datetime=${yyyy-mm-dd-29}
  • Reference in code: pt=${datetime}
  • Replacement result: pt=${yyyy-mm-dd-29}=2019-06-20
  • Assignment: datetime=$[add_months(yyyymmdd,-1)]
  • Reference in code: pt=$[datetime]
  • Replacement result: pt=$[dadd_months(yyyymmdd,-1)]=2019-06-20
Date: July 19, 2019
  • Assignment: datetime=${yyyy-mm-dd}
  • Reference in code: pt=${datetime}
  • Replacement result: pt=${yyyy-mm-dd}=2019-07-19
  • Assignment: datetime=$[yyyy-mm-dd-1]
  • Reference in code: pt=$[datetime]
  • Replacement result: pt=$[yyyy-mm-dd-1]=2019-07-19
Date: July 20, 2018
  • Assignment: datetime=${yyyy-mm-dd-364}
  • Reference in code: pt=${datetime}
  • Replacement result: pt=${yyyy-mm-dd}=2019-07-20
  • Assignment: datetime=$[add_months(yyyymmdd,-12*1)]
  • Reference in code: pt=$[datetime]
  • Replacement result: pt=$[dadd_months(yyyymmdd,-12*1)]=2018-07-20
Time: 10:30:00 Not supported
  • Assignment: datetime=$[hh24:mi:ss]
  • Reference in code: pt=$[datetime]
  • Replacement result: pt=$[hh24:mi:ss]=10:30:00
Time: 2019-07-20 10:30:00 Not supported
  • Assignment:

    You must customize two variables datetime1 and datetime2, and separate two assignment equations with a space.

    datetime1=$[yyyy-mm-dd] datetime2=$[hh24:mi:ss]
  • Reference in code: pt=$[datetime1] $[datetime2]
  • Replacement result:
    • datetime1=$[yyyy-mm-dd]=2019-07-20
    • datatime2=$[hh24:mi:ss]=10:30:00
    Therefore, the result is pt=2019-07-20 10:30:00.
Time: 2019-07-20 10:29:00 Not supported
  • Assignment:

    You must customize two variables datetime1 and datetime2, and separate two assignment equations with a space.

    datetime1=$[yyyy-mm-dd] datetime2=$[hh24:mi:ss-1/24/60]
  • Reference in code: pt=$[datetime1] $[datetime2]
  • Replacement result:
    • datetime1=$[yyyy-mm-dd]=2019-07-20
    • datatime2=$[hh24:mi:ss-1/24/60]=10:29:00
    Therefore, the result is pt=2019-07-20 10:29:00.
Time: 2019-07-20 08:30:00 Not supported
  • Assignment:

    You must customize two variables datetime1 and datetime2, and separate two assignment equations with a space.

    datetime1=$[yyyy-mm-dd] datetime2=$[hh24:mi:ss-1/24]
  • Reference in code: pt=$[datetime1] $[datetime2]
  • Replacement result:
    • datetime1=$[yyyy-mm-dd]=2019-07-20
    • datatime2=$[hh24:mi:ss-1/24]=08:30:00
    Therefore, the result is pt=2019-07-20 08:30:00.

Examples of scheduling parameters used for different types of nodes

  • In the following example, a MaxCompute SQL node or batch synchronization node is used. Assignment
    • If you use built-in system variables, you can reference them in the code without the need to assign values to them on the Properties tab.
    • If you use custom parameters, you must assign them to variables in the format of Custom variable=Custom parameter on the Properties tab. Then, the variables are referenced in the code.
  • In the following example, a PyODPS node is used.

    To make code unobtrusive, a PyODPS node does not replace strings that are in the ${param_name} format in the code. However, you can add a dictionary object named args to the global variable before the code is run. This way, the PyODPS node can obtain the values of scheduling parameters from the dictionary object.

    For example, on the Properties tab of the node configuration tab, enter def=${yyyymmdd} in the Arguments field of the General section. The following figure shows how to obtain the value of the parameter in the code. PyODPS
  • In the following example, a Shell node is used. Shell

    You are not allowed to customize variable names for Shell nodes. The variable names must follow the $1,$2,$3... format. If the number of parameters reaches 10, use ${10} to declare the tenth variable.

Test scheduling parameters in the development environment

If you use scheduling parameters during data development, you can perform smoke testing on the DataStudio page to check whether the values of the parameters meet your expectations. During the smoke testing, you must manually replace the parameters with constants.
  • Perform smoke testing
    You can click Run Smoke Test in Development Environment in the toolbar of the node configuration tab. Then, enter the business date to simulate the scenario in which the node is scheduled to run. Obtain the values of the scheduling parameters after the parameters are replaced, and check whether the values meet your expectations based on the smoke testing result. Smoke testing in the development environment
    Note
    • If you modify the scheduling parameters when you perform smoke testing for a node in the development environment, save and commit the node in the development environment again. Otherwise, the code cannot be updated.
    • When you perform smoke testing in the development environment, you are charged for the test instances that is generated.
  • Manually replace scheduling parameters
    After you click Run (1) or Run with Arguments (2) on the node configuration tab, scheduling parameters are not automatically replaced with the variables specified on the Properties tab. You must manually assign constants to the variables (3 and 4) in the code. Manually assign parameters

View the parameter replacement results in Operation Center

After an auto triggered node is developed, committed, and deployed, the scheduling parameters of node instances are replaced when the instances are generated. You can check whether the scheduling parameters are replaced as expected on the Cycle Task page. To go to this page, choose Cycle Task Management > Cycle Task in the left-side navigation pane of the Operation Center page.

For example, you modify the scheduling parameters of an auto triggered node on the DataStudio page, commit the node, and then deploy it in the production environment. In this case, you must check whether the parameter values in the Execution parameters field meet your expectations.

FAQ