DataWorks:Scheduling parameters

Use two custom variables:

• Year, month, and day: Parameter datetime=$[yyyy-mm-dd]

• Hour, minute, and second: Parameter hour=$[hh24:mi:ss]

After obtaining the date and time separately, join them with a space in the code as pt=${datetime} ${hour}. For example, use this SQL: select * from xc_demo_partition where dt='${key1} ${key2}';. Under Schedule > Parameters, configure key1=$[yyyymmdd] key2=$[hh24miss].

Scheduling parameters use the instance scheduled time. Their resolved values are set when the instance is generated and do not change with its actual run time. In Operation Center > Periodic Instances, the Properties panel shows Scheduled Time as the expected run time, such as 2021-07-13 00:00:00. Compare it with Start Time, for example, 00:00:27. In Data Development, configure Schedule > Parameters with var1=$bizdate var2=$cyctime var3=$[yyyymmdd] var4=$[yyyymmddhh24miss], and keep the calculation formula for Set the scheduling cycle to Day and and the scheduled run time to 00:00. In the code, use select '${var1}' '${var2}'; to check the business date and scheduled time values separately.

After defining variables in the code, click Schedule on the right side of the editor and assign values in the parameter panel. For details about scheduling parameters, see Scheduling parameter formats. For example, enter Parameters in the ODPS Spark node field: $(var). Add a row to the parameter table with the parameter name var and value $bizdate. Click Scheduling Parameter Preview to verify the resolved values.

Some nodes, such as batch synchronization nodes, cannot directly use a scheduling parameter result and need further conversion. Configure an assignment node upstream. Have it reference the scheduling parameter and convert the result, then pass the final value to the downstream node through node context.

• For configuring an assignment node, see Assignment node.

• For configuring node context, see Configure node context.

Scheduling parameters are automatically replaced only by the scheduling system. To test them in Data Development as they would be replaced by the scheduler, use a development environment smoke test.

SolutionSolution:

1. Check whether the scheduling parameter has a value.

2. Check whether the configured parameter meets the requirements. For details, see Scheduling parameter formats.

Solution:

1. Check whether the variable has a value.

2. Check for extra spaces in the scheduling parameter.

3. Do not combine a period (.) and Chinese characters in a node name.

Example:time①=②$[yyyymmdd③hh24:mi:ss]time1=$[yyyymmdd]④time2=$[hh24:mi:ss] (①②③④ mark the locations of spaces).

• Do not put spaces around the equals sign (=) in a scheduling parameter; see ① and ②.

• Spaces are not supported within a scheduling parameter assignment; see ③.

• Separate two scheduling parameters with one space; see ④.

After changing a scheduling parameter in Data Development and committing and publishing the task, check its value in the periodic task Properties panel in the production Operation Center. If it is not as expected, check the task publishing status and publish package to confirm that publishing succeeded. In the periodic instance Properties panel, view the replacement values for an individual instance.

Real-time scenario:

• On the current day, the scheduling parameter time1 of periodic instance A1 for node A has the value $bizdate. The instance ran successfully that day, and the value used in the code was the bizdate business date (specific value).

• Later that day, the parameter time1 changes from $bizdate to $cyctime. The execution parameter shown for that day’s instance becomes the cyctime scheduled time (specific value).

• If you rerun that instance, it uses the latest configuration: time1=$cyctime

• To see parameter replacement values used before the change, check the logs. For details, see the FAQ about viewing replacement values in logs.How do I view the scheduling parameter values used by an instance in its logs?.

In the Operation Center > Periodic Instances page, select an instance. In the Properties panel’s Execution Parameters field to see the actual resolved values, such as var1=20210708233000 var2=20210709000000, and confirm that the values are updated as expected.

Find SKYNET_PARAVALUE in the log.

07-09 00:00:19  INFO SKYNET_RERUN_MODE=1:
07-09 00:00:19  INFO SKYNET_FLOW_PARAVALUE=:
07-09 00:00:19  INFO SKYNET_ONDUTY=xxx:
07-09 00:00:19  INFO SKYNET_SYSTEMID=:
07-09 00:00:19  INFO CALC_ENGINE_IDENTIFIER=xxx:
07-09 00:00:19  INFO SKYNET_SOURCEID=xxx:
07-09 00:00:19  INFO SKYNET_PARAVALUE=var1=20210708233000 var2=20210709000000:
07-09 00:00:19  INFO SKYNET_TASKID=xxx:
07-09 00:00:19  INFO SKYNET_TENANT_ID=283789484710656:
07-09 00:00:19  INFO SKYNET_ID=700003341088:
07-09 00:00:19  INFO SKYNET_JOBID=700200619122:
07-09 00:00:19  INFO SKYNET_NODENAME=xc_partition_copy:
07-09 00:00:19  INFO SKYNET_CYCTYPE=0:
07-09 00:00:19  INFO SKYNET_DSC_JOB_VERSION=1:
07-09 00:00:19  INFO SKYNET_TASK_INPUT={}:
07-09 00:00:19  INFO SKYNET_FLOWNAME=ATCLOUD_FLOW:
07-09 00:00:19  INFO FILE_ID=700003341088:

Daylight saving time is supported for instance generation and parameter partition calculations, so tasks can run correctly on transition days. For example, in US Pacific time:

• On the day daylight saving time begins, 23 instances are generated. Subtracting 10 minutes from 03:00 gives 01:50. A task scheduled for 02:00 on that day runs as an empty task.

• On the day daylight saving time ends, 24 instances are generated. Subtracting 10 minutes from 03:00 gives 02:50.