This topic provides answers to some frequently asked questions about scheduling parameters.

I run an instance of a node at 00:00 on the current day to analyze the data in the partition that corresponds to 23:00 on the previous day. However, the data in the partition that corresponds to 23:00 on the current day is analyzed. What do I do?

  • Problem description: The table partition format is day=yyyymmdd,hour=hh24. The $[yyyymmdd] $[hh24-1/24] variable is used to specify the date and time of a partition. If you run an instance at 00:00, the custom variable datetime=$[yyyymmdd] specifies the current day instead of the previous day. As a result, the data in the partition that corresponds to 23:00 on the current day is analyzed.
  • Solution: Change the value of datetime to $[yyyymmdd-1/24] and retain the value $[hh24-1/24] for hour.
    How to configure:
    • In the code: day=datetime, hour={hour},
    • Scheduling parameters that are configured for nodes: datetime=[yyyymmdd-1/24],hour=[hh24-1/24]
    Scenario:
    • For an instance that is scheduled to run at 2021-07-21 00:00:00, the value of $[yyyymmdd-1/24] is 20210720, and the value of $[hh24-1/24] is 23. This happens because 1 hour before 2021-07-21 00:00:00 is a point in time on the previous day.
    • For an instance that is scheduled to run at 2021-07-21 01:00:00, the value of $[yyyymmdd-1/24] is 20210721, and the value of $[hh24-1/24] is 00. This happens because 1 hour before 2021-07-21 01:00:00 is still a point in time on the current day.

How do I specify a table partition in a format that contains a space, such as pt=yyyy-mm-dd hh24:mi:ss?

Notice Spaces are not allowed in scheduling parameters.
Solution: Use the custom variable datetime=$[yyyy-mm-dd] to obtain the date and the custom variable hour=$[hh24:mi:ss] to obtain the time. Then, join the variables with a space to form pt=${datetime} ${hour}. Properties

A node is scheduled to run at the time specified by the $cyctime or $[yyyymmddhh24miss] variable. The node is scheduled to run at 20:00 every day, but the ancestor node of the node fails to run as scheduled. As a result, the node is delayed and run at 00:00 on the next day. In this case, is the value of the $cyctime or $[yyyymmddhh24miss] variable 20:00 or 00:00?

If the resources are insufficient, the time at which an instance actually runs may be different from the time at which the instance is scheduled to run. The scheduled time of an instance is fixed at the time when the instance is generated and does not change even if the time at which the instance runs changes. Therefore, scheduling parameters are configured based on the fixed scheduled time and do not change even if the time at which the instance runs changes.

调度参数定时时间小时参数取值

How do I configure the time properties of an ODPS Spark node?

After you create an ODPS Spark node, you must configure variables in the Parameters field on the node configuration tab. DataWorks

After you configure the variables, click the Properties tab in the right-side navigation pane of the node configuration tab. In the Properties panel, assign values for the variables. You can assign values based on the description of scheduling parameters in this topic. spark

How do I test the configurations of the scheduling parameters on the DataStudio page?

The values of the scheduling parameters are automatically replaced in the scheduling system only after you deploy the scheduling parameters in the production environment. If you want to test whether the values of the scheduling parameters are valid on the DataStudio page, click the Perform Smoke Testing in Development Environment icon in the top toolbar of the node configuration tab.

Note For a data integration node, you cannot test whether the values of the scheduling parameters are valid in the development environment. If you want to perform such a test, you must create an SQL node and then test the configurations of the scheduling parameters by clicking the Perform Smoke Testing in Development Environment icon. If the scheduling parameters pass the test, you can use the configurations of these parameters in the data integration node.

What can I do if the FAILED: ODPS-0130161:[1,84] Parse exception - invalid token '$' error is reported?

Cause: The scheduling parameters are not specified or the values of the scheduling parameters are invalid.

Solution:
  1. Check whether the scheduling parameters are specified.
  2. Check whether the values of the scheduling parameters are valid. For more information, see Configure scheduling parameters.
Notice After you modify the scheduling parameters of a node, you must commit and deploy them. After the scheduling parameters are deployed, go to the Cycle Task page in Operation Center and check whether the values of the scheduling parameters are updated on the General tab of the node.
检查调度参数

What do I do if the params format error, please check your params(key=values) error is reported?

  1. Check whether values are assigned to variables.
  2. Check whether spaces are used in the scheduling parameters.
  3. Check whether a node name contains periods (.) and Chinese characters at the same time.

time1=2$[yyyymmdd3hh24:mi:ss] and time1=$[yyyymmdd]4time2=$[hh24:mi:ss]. Symbols 1, 2, 3, and 4 represent the positions where spaces may be added.

Do not add a space before or after the equal sign (=) in a scheduling parameter. In this example, do not add spaces in the positions specified by Symbols 1 and 2.

Do not include a space in the value of a scheduling parameter. In this example, do not add a space in the position specified by Symbol 3.

Separate two scheduling parameters with a space. In this case, add a space in the position specified by Symbol 4.

What are the differences in the value assignment logic of scheduling parameters among the Run, Run with Parameters, and Perform Smoke Testing in Development Environment modes?

  • Run: The first time you click the Run icon, you must manually assign constants to variables in the node code. The constants are recorded by DataWorks. If you modify the code, the variables still use the constants that you assigned.
  • Run with Parameters: If you use the Run with Parameters mode, you must manually assign constants to variables in the code. If you modify the variables in the code, you must use the Run with Parameters mode to reassign constants to the variables.
  • Perform Smoke Testing in Development Environment: You can enter a data timestamp to simulate automatic node scheduling and obtain the new values of the scheduling parameters at the specified data timestamp.
高级运行
Note If you want to change the resource group that is used by a node, click the Run with Parameters icon.

How do I check the validity of the values of scheduling parameters in the production environment?

After you modify the scheduling parameters of a node on the DataStudio page and commit and deploy the node, you can check whether the scheduling parameters are specified based on your requirements on the General tab of the Cycle Task page in Operation Center. If the configurations do not meet your requirements, check whether the deployment package of the node is generated as expected on the deployment page. Cycle Task

Check whether the values of the scheduling parameters of a single instance are valid on the General tab of the Cycle Instance page.

Notice When you modify the scheduling parameters of an auto triggered node for which a single instance is generated, the configurations of the scheduling parameters of the single instance are updated in real time. The real-time update is performed no matter whether the instance is run.
Scenario:
  • For example, you assign $bizdate to the time1 scheduling parameter of Instance A1 of Node A. If Instance A1 is run successfully on the current day, the time1 scheduling parameter is set to the data timestamp specified by bizdate in the code.
  • If you change the value of the time1 scheduling parameter from $bizdate to $cyctime at a point in time on the current day, Instance A1 is run at the scheduled time that is specified by cyctime on the current day.
  • If you rerun Instance A1, the latest configuration of time1 is used. In this example, time1=$cyctime.
  • If you want to view the values of the scheduling parameters of the instance before the value changes, check logs. For more information about how to check whether the values of the scheduling parameters of an instance are valid by viewing logs, see How do I check whether the values of the scheduling parameters of an instance are valid by viewing logs?.
调度参数替换

How do I check whether the values of the scheduling parameters of an instance are valid by viewing logs?

Find SKYNET_PARAVALUE in the code.

Check whether the values of scheduling parameters of an instance are valid by viewing logs

How are node instances generated on the day when daylight saving time begins and ends?

DataWorks supports the immediate instance generation and daylight saving time-based parameter computing features. This way, nodes can be run as expected when daylight saving time begins or ends. For example, the time zone is UTC-8.
  • When daylight saving time begins, 10 minutes before 03:00 is 01:50, and 23 instances are generated on that day. The system does not run the instance that is scheduled to run at 02:00 on that day.
  • When daylight saving time ends, 10 minutes before 03:00 is 02:50, and 24 instances are generated on that day.

If a node scheduled by day, week, or month is scheduled to run within the period that is skipped when daylight saving time begins, a node instance is generated and run at 00:00 on that day.