DataWorks:Configure and use scheduling parameters

Precautions

Before you click the or icon in the toolbar to run a node, you must manually assign constants to the variables that you want to reference in the code of the node. In this case, you cannot check whether the configured scheduling parameters work as expected.

Process of configuring scheduling parameters

Define variables

You can perform the steps in the following figure to define variables in the code of a node and configure the scheduling parameters that you want to assign to the variables as values in the Parameters section of the Properties tab based on your business requirements.

Note Some nodes use the built-in variable ${bizdate}. The built-in variable is automatically assigned $bizdate as a value. You can replace bizdate with a custom variable name based on your business requirements.

Assign scheduling parameters as values to variables in the code of a node

DataWorks allows you to add scheduling parameters in a visualized manner or by using expressions.

Method to define scheduling parameters	Feature	Description	Screenshot
Visualization mode	Add Parameter	You can click Add Parameter to configure multiple scheduling parameters for a node. For more information about scheduling parameters, see Supported formats of scheduling parameters.
	Load Parameters in Code	After you click Load Parameters in Code, DataWorks identifies the variable names defined in the code of the current node and adds the identified variable names to the Parameters section. Note In most cases, custom variables are defined in the format of ${Custom variable name} in the code. For PyODPS nodes or common Shell nodes, variable names are defined by using a different method from that for other types of nodes. For more information about how to define custom variables for different types of nodes, see Configure scheduling parameters for different types of nodes.
Expression mode	Define by expression	By default, the system provides an interface on which you can configure scheduling parameters for a node. You can also click Define by expression to configure scheduling parameters by using an expression. Note When you use an expression to configure multiple scheduling parameters, separate multiple assignment equations with spaces. If you add, delete, or modify scheduling parameters by using expressions, DataWorks checks the syntax of the expressions. If the check fails, the scheduling parameters fail to be configured. For example, if a space exists on either side of the equal sign (=), DataWorks reports an error.

Test whether scheduling parameters are replaced as expected

After you assign values to scheduling parameters, you can use the scheduling parameter preview feature to test whether the scheduling parameters are replaced as expected.

DataWorks replaces the scheduling parameters based on the scheduling configurations of the node when the node is scheduled to run in a test environment. You can check whether the scheduling parameters are replaced with valid values. If the values do not meet your business requirements, you can adjust the scheduling configurations of the node at the earliest opportunity to prevent the invalid configurations from affecting the running of the node.

Test whether the code of the node is executed and the scheduling parameters are replaced as expected

After you assign values to the scheduling parameters, we recommend that you use the smoke testing feature to specify a data timestamp for the node and test whether the code of the node is executed and the scheduling parameters are replaced as expected when the node is scheduled to run. If the node is not run as expected, you can adjust the scheduling configurations of the node at the earliest opportunity to prevent the invalid configurations from affecting the running of the node.

After you modify the code of the node, you must click the icon and then the icon to save the modifications and commit the node to the development environment. Otherwise, you cannot perform smoke testing on the node in the development environment.

1. Perform smoke testing
2. View smoke testing logs

View the configurations of scheduling parameters in Operation Center in the production environment

To prevent unexpected configurations of scheduling parameters from affecting the running of an auto triggered node, we recommend that you check the configurations of the scheduling parameters for the auto triggered node on the Cycle Task page in Operation Center in the production environment after the auto triggered node is deployed. For more information about auto triggered nodes, see View and manage auto triggered nodes.

Note If the scheduling parameters of an auto triggered node are not configured as expected or you cannot find the auto triggered node in Operation Center, check whether the node is successfully deployed. For more information about how to deploy nodes, see Deploy nodes.

Example on how to configure scheduling parameters

1. Edit the code of the node and configure the scheduling parameters.
   The following figure shows the code and scheduling parameter configurations of the ODPS SQL node.
   1. Define variables in the code.
      Define variables '${var1}' and '${var2}' to reference built-in parameters and define variables '${var3}' and '${var4}' to reference custom parameters in the code of the ODPS SQL node, as shown in area 1 in the preceding figure.
   2. Assign values to the variables.
      Go to the Parameters section of the Properties tab and assign values to the variables, as shown in area 2 in the preceding figure.

      • var1=$bizdate: the data timestamp in the yyyymmdd format.
      • var2=$cyctime: the time at which the node is scheduled to run in the yyyymmddhh24miss format.
      • var3=${yyyymmdd}: the data timestamp in the yyyymmdd format.
      • var4=$[yyyymmddhh24miss]: the time at which the node is scheduled to run in the yyyymmddhh24miss format.
   3. Optional:Configure a scheduling cycle.
      Configure the ODPS SQL node to run at an interval of a specified number of hours, as shown in area 3 in the preceding figure.

      Note You can determine whether to configure a scheduling cycle for a node based on your business requirements. In this example, a scheduling cycle is configured.

      • Start from: 16:00
      • End at: 23:59
      • Interval: 1 hour
2. In the top toolbar of the configuration tab of the node, click the icon and then the icon to save the configurations of the ODPS SQL node and commit the ODPS SQL node.
3. Perform smoke testing in the development environment.
   1. Click the icon. In the Perform Smoke Testing in Development Environment dialog box, configure a data timestamp to simulate the scheduling cycle of the node.
      In this example, configure a data timestamp in the following way:

      • Data timestamp: 2022-03-09
      • Start from: 16:00
      • End at: 17:00

      The ODPS SQL node is scheduled by hour. Two instances are expected to be generated for the node at 16:00 and 17:00 on March 10, 2022.

      Note The data timestamp is one day earlier than the time at which the node is scheduled to run. Therefore, the node is scheduled to run on March 10, 2022.

      The following results are expected for the instance that is generated at 16:00:

      • var1=20220309
      • var2=20220310160000
      • var3=20220309
      • var4=20220310160000

      The following results are expected for the instance that is generated at 17:00:

      • var1=20220309
      • var2=20220310170000
      • var3=20220309
      • var4=20220310170000
   2. Click OK. The node is run at its scheduling time.
   3. After the smoke testing is complete, click the icon to view the smoke testing logs.
      The two instances generated by the node are successfully run and the test result meets expectations.
4. On the configuration tab of the ODPS SQL node, click Deploy on the right side of the top navigation bar to deploy the current node.
   For more information about how to deploy nodes, see Deploy nodes.
5. Go to Operation Center and check the configurations of the scheduling parameters of the node.
   
   1. On the DataStudio page, click Operation Center on the right side of the top navigation bar to go to the Operation Center page.
   2. In the left-side navigation pane, choose Cycle Task Maintenance > Cycle Task. On the Cycle Task page, find the node that you want to manage.
      Note You can find a node on the Cycle Task page only after the node is deployed.
   3. Choose More > View Node Details in the Actions column of the node. On the Node Information page, view the value of Execution Parameters.
      In this example, the scheduling parameters of the node are configured as expected in the following way: var1=$bizdate var2=$cyctime var3=${yyyymmdd} var4=$[yyyymmddhh24miss].