This topic provides answers to some frequently asked questions about scheduling parameters.

I run an instance of a node at 00:00 on the current day to analyze the data in the partition that corresponds to 23:00 on the previous day. However, the data in the partition that corresponds to 23:00 on the current day is analyzed. What do I do?

  • Problem description: The table partition format is day=yyyymmdd,hour=hh24. The $[yyyymmdd] $[hh24-1/24] variable is used to specify the date and time of a partition. If I run an instance at 00:00, the custom variable datetime=$[yyyymmdd] specifies the current day instead of the previous day. As a result, the data in the partition that corresponds to 23:00 on the current day is analyzed.
  • Solution: Change the value of datetime to $[yyyymmdd-1/24] and retain the value $[hh24-1/24] for hour.
    How to configure:
    • In the code: day=datetime, hour={hour},
    • Scheduling parameters that are configured for nodes: datetime=[yyyymmdd-1/24],hour=[hh24-1/24]
    Scenarios:
    • For an instance that is scheduled to run at 2021-07-21 00:00:00, the value of $[yyyymmdd-1/24] is 20210720, and the value of $[hh24-1/24] is 23. This happens because 1 hour before 2021-07-21 00:00:00 is a point in time on the previous day.
    • For an instance that is scheduled to run at 2021-07-21 01:00:00, the value of $[yyyymmdd-1/24] is 20210721, and the value of $[hh24-1/24] is 00. This happens because 1 hour before 2021-07-21 01:00:00 is still a point in time on the current day.

How do I specify a table partition in a format that contains a space, such as pt=yyyy-mm-dd hh24:mi:ss?

Notice Spaces are not allowed in scheduling parameters.
Solution: Use the custom variable datetime=$[yyyy-mm-dd] to obtain the date and the custom variable hour=$[hh24:mi:ss] to obtain the time. Then, join the variables with a space to form pt=${datetime} ${hour}. Properties panel

A node is scheduled to run at the time specified by the $cyctime or $[yyyymmddhh24miss] variable. The node is scheduled to run at 20:00 every day, but the ancestor node of the node fails to run as scheduled. As a result, the node is delayed and runs at 00:00 on the next day. In this case, is the value of the $cyctime or $[yyyymmddhh24miss] variable 20:00 or 00:00?

If the resources are insufficient, the time at which an instance actually runs may be different from the time at which the instance is scheduled to run. The scheduled time of an instance is fixed at the time when the instance is generated and does not change even if the time at which the instance runs changes. Therefore, scheduling parameters are configured based on the fixed scheduled time and do not change even if the time at which the instance runs changes.

How do I configure the time properties of an ODPS Spark node?

After you create an ODPS Spark node, you must configure variables in the Parameters field on the node configuration tab.

After you configure the variables, click the Properties tab in the right-side navigation pane of the node configuration tab. In the Properties panel, assign values for the variables. You can assign values based on the description of scheduling parameters in this topic. Configure the ODPS Spark node

How can I reprocess the return values of the scheduling parameters for a node if the node cannot process the return values?

After you configure scheduling parameters for some nodes, such as batch synchronization nodes, the nodes cannot process the return values of the scheduling parameters unless you reprocess the return values. You can configure assignment nodes as the ancestor nodes of these nodes. This way, you can use the assignment nodes to reference the scheduling parameters and reprocess the return values of scheduling parameters. Then, you can use context-based parameters to pass the reprocessed values to the required descendant nodes. For more information about how to configure assignment nodes, see Configure an assignment node. For more information about how to configure context-based parameters, see Configure context-based parameters.

What are the differences between the return values of a MaxCompute date function and a scheduling parameter?

  • If you use a MaxCompute date function, the return value is the system time when the instance runs. If the instance runs at different points in time, the return values are different.
  • If you use a scheduling parameter, the return result is a calculated result of the scheduled time. If the instance runs at different points in time, the return values remain the same.

How do I test the configurations of the scheduling parameters on the DataStudio page?

The values of the scheduling parameters are automatically replaced in the scheduling system only after you deploy the scheduling parameters in the production environment. If you want to check whether the values of the scheduling parameters are valid on the DataStudio page, click the Perform Smoke Testing in Development Environment icon in the top toolbar of the node configuration tab.

Note For a data integration node, you cannot check whether the values of the scheduling parameters are valid in the development environment. If you want to perform such a test, you must create an SQL node and then test the configurations of the scheduling parameters by clicking the Perform Smoke Testing in Development Environment icon. If the scheduling parameters pass the test, you can use the configurations of these parameters in the data integration node.

FAILED: ODPS-0130161:[1,84] Parse exception - invalid token '$'

Cause: The scheduling parameters are not specified or the values of the scheduling parameters are invalid.

Solution:
  1. Check whether the scheduling parameters are specified.
  2. Check whether the values of the scheduling parameters are valid. For more information, see Configure scheduling parameters.
Notice After you modify the scheduling parameters of a node, you must commit and deploy them. After the scheduling parameters are deployed, go to the Cycle Task page in Operation Center and check whether the values of the scheduling parameters are updated on the General tab of the node.

What do I do if the params format error, please check your params(key=values) error is reported?

  1. Check whether values are assigned to variables.
  2. Check whether spaces are used in the scheduling parameters.
  3. Check whether a node name contains periods (.) and Chinese characters at the same time.

time1=2$[yyyymmdd3hh24:mi:ss] and time1=$[yyyymmdd]4time2=$[hh24:mi:ss]. Symbols 1, 2, 3, and 4 represent the positions where spaces may be added.

Do not add a space before or after the equal sign (=) in a scheduling parameter. In this example, do not add spaces in the positions specified by Symbols 1 and 2.

Do not include a space in the value of a scheduling parameter. In this example, do not add a space in the position specified by Symbol 3.

Separate two scheduling parameters with a space. In this case, add a space in the position specified by Symbol 4.

What are the differences in the value assignment logic of scheduling parameters among the Run, Run with Parameters, and Perform Smoke Testing in Development Environment modes?

  • Run: The first time you click the Run icon, you must manually assign constants to variables in the node code. The constants are recorded by DataWorks. If you modify the code, the variables still use the constants that you assigned.
  • Run with Parameters: If you use the Run with Parameters mode, you must manually assign constants to variables in the code. If you modify the variables in the code, you must use the Run with Parameters mode to reassign constants to the variables.
  • Perform Smoke Testing in Development Environment: You can enter a data timestamp to simulate automatic node scheduling and obtain the new values of the scheduling parameters at the specified data timestamp.
Note If you want to change the resource group that is used by a node, click the Run with Parameters icon.

How do I check the validity of the values of scheduling parameters in the production environment?

After you modify the scheduling parameters of a node on the DataStudio page and commit and deploy the node, you can check whether the scheduling parameters are specified based on your requirements on the General tab of the Cycle Task page in Operation Center. If the configurations do not meet your requirements, check whether the deployment package of the node is generated as expected on the deployment page. Cycle Task

Check whether the values of the scheduling parameters of a single instance are valid on the General tab of the Cycle Instance page.

Notice When you modify the scheduling parameters of an auto triggered node for which a single instance is generated, the configurations of the scheduling parameters of the single instance are updated in real time. The real-time update is performed no matter whether the instance is run.
Scenarios:
  • For example, you assign $bizdate to the time1 scheduling parameter of Instance A1 of Node A. If Instance A1 is run successfully on the current day, the time1 scheduling parameter is set to the data timestamp specified by bizdate in the code.
  • If you change the value of the time1 scheduling parameter from $bizdate to $cyctime at a point in time on the current day, Instance A1 is run at the scheduled time that is specified by cyctime on the current day.
  • If you rerun Instance A1, the latest configuration of time1 is used. In this example, time1=$cyctime.
  • If you want to view the values of the scheduling parameters of the instance before the value changes, check logs. For more information about how to check whether the values of the scheduling parameters of an instance are valid by viewing logs, see How do I check whether the values of the scheduling parameters of an instance are valid by viewing logs?.

How do I check whether the values of the scheduling parameters of an instance are valid by viewing logs?

Find SKYNET_PARAVALUE in the code.

Check whether the values of scheduling parameters of an instance are valid by viewing logs

How are node instances generated on the day when daylight saving time begins and ends?

DataWorks supports the immediate instance generation and daylight saving time-based parameter computing features. This way, nodes can be run as expected when daylight saving time begins or ends. For example, the time zone is UTC-8.
  • When daylight saving time begins, 10 minutes before 03:00 is 01:50, and 23 instances are generated on that day. The system does not run the instance that is scheduled to run at 02:00 on that day.
  • When daylight saving time ends, 10 minutes before 03:00 is 02:50, and 24 instances are generated on that day.

If a node scheduled by day, week, or month is scheduled to run within the period that is skipped when daylight saving time begins, a node instance is generated and run at 00:00 on that day.

I configure a scheduling parameter for a node and commit and deploy the node, but the return value of the scheduling parameter remains unchanged. What do I do?

Check whether the scheduling parameter is overwritten by a workflow parameter with the same name. For more information, see Use workflow parameters. If a workflow parameter with the same name exists, you can delete the workflow parameter based on your business requirements. If you need to retain the workflow parameter, you must change the name of the scheduling parameter for the node.