This topic provides answers to some frequently asked questions about scheduling parameters.

How do I specify a table partition in a format that contains a space?

  • Problem description: The table partition format is pt=yyyy-mm-dd hh24:mi:ss, but spaces are not allowed in scheduling parameters.
  • Solution: Use the custom variable datetime=$[yyyy-mm-dd] to obtain the date and the custom variable hour=$[hh24:mi:ss] to obtain the time. Then, join the variables to form pt=${datetime} ${hour} in the code.

    Separate the two variables with a space.

I have run an instance of a node at 00:00 on the current day to analyze the data in the partition that corresponds to 23:00 on the previous day. However, the data in the partition that corresponds to 23:00 on the current day is analyzed. What do I do?

  • Problem description: pt=${datetime} ${hour} is used in the code to specify a partition. The custom variables datetime=$[yyyymmdd] and hour=$[hh24-1/24] are used to specify the date and time of the partition. However, if you run an instance at 00:00, the custom variable datetime=$[yyyymmdd] specifies the current day instead of the previous day. As a result, the data in the partition that corresponds to 23:00 on the current day is analyzed.
  • Solution:
    Change the value of datetime to $[yyyymmdd-1/24] and retain the value $[hh24-1/24] for hour. The node is run in the following way:
    • For an instance that is scheduled to run at 2015-10-27 00:00:00, the value of $[yyyymmdd-1/24] is 20151026 and the value of $[hh24-1/24] is 23. This is because the scheduled time minus 1 hour is a point in time on the previous day.
    • For an instance that is scheduled to run at 2015-10-27 01:00:00, the value of $[yyyymmdd-1/24] is 20151027 and that of $[hh24-1/24] is 00. This is because the scheduled time minus 1 hour is a point in time on the current day.

What are the modes that I can use to run nodes in DataWorks?

DataWorks offers the following modes to run nodes:
  • You can run a node on the DataStudio page in one of the following modes:
    • Run: The first time you click the Run icon, you must manually assign values to the variables in the node code. DataWorks records these values. If you modify the code, the variables still use the values assigned when the node is run for the first time.
    • Run with Parameters: If you want to change the values assigned to the variables in the code or change the resource group on which the node is run, click this icon.
    • Perform Smoke Testing in Development Environment: You can enter a data timestamp to simulate automatic node scheduling and obtain the new values of scheduling parameters at the specified data timestamp.
    Note If you select the Run or Run with Parameters mode, the system replaces the variables in the code with the values you specify without verifying the values. If you want the system to verify the values, select the Perform Smoke Testing in Development Environment mode.
  • Run a node in the production environment: The scheduling system automatically replaces the values of scheduling parameters based on the time at which the current instance is scheduled to run. The scheduling parameters include system parameters and custom parameters.
  • Test a node or backfill data: You must specify a data timestamp. You can use the related formula to calculate the time at which each instance is scheduled to run.

What do I do if the system cannot parse the custom parameters specified for a node and the node fails to be run?

  • Problem description: The system cannot parse the custom parameter $[yyyymmdd] that is specified in INSERT into table log_in PARTITION (dat) SELECT id,name,age,sex,${var} from test1;. As a result, the related node fails to be run.
  • Solution: Perform smoke testing for the system to parse the custom parameter.

How are node instances generated on the day when daylight saving time begins or ends?

DataWorks supports the immediate instance generation and daylight saving time-based parameter computing features. This way, nodes can be run as expected when daylight saving time begins or ends. For example, the time zone is UTC-8.
  • When daylight saving time begins, 10 minutes before 03:00 is 01:50, and 23 instances are generated on that day. The system does not actually run the instance that is scheduled to run at 02:00 on that day.
  • When daylight saving time ends, 10 minutes before 03:00 is 02:50, and 24 instances are generated on that day.

If a node scheduled by day, week, or month is scheduled to run within the skipped period of the day when daylight saving time begins, a node instance is generated and run at 00:00 on that day.