Learn how to configure the scheduling properties and scheduling dependencies for a DataWorks task. This topic uses a weekly schedule for the write_result task as an example.
Prerequisites
-
A zero load node named start and an ODPS SQL node named insert_data have been created. For more information, see Create a workflow.
-
A synchronization task has been created from the synchronization node named write_result. For more information, see Create a synchronization task.
Background information
DataWorks provides powerful scheduling capabilities that support time-based and dependency-based triggers. DataWorks ensures that tens of millions of tasks run accurately and on time every day based on their DAG relationships. You can configure scheduling cycles by minute, hour, day, week, or month. For more information, see Configure time properties.
Configure time properties
-
Go to the DataStudio page.
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.
-
In the relevant workflow, double-click the offline synchronization node write_result to open its configuration tab.
-
In the right-side navigation pane, click Scheduling.
NoteThe scheduler does not run nodes in a manually triggered workflow; you must trigger them manually.
-
In the Schedule section, configure the scheduling properties for the task.
Parameter
Description
Instance Generation Mode
Valid values: Next Day and Immediately After Deployment. For more information, see Instance generation modes.
Recurrence
-
Normal: The task runs and processes data according to the configured schedule.
-
Suspend scheduling: At the scheduled time, the scheduler sets the node's status to suspended and skips data processing.
-
Dry Run: The scheduler triggers the task at the scheduled time, but the node performs a dry run and does not process data.
Scheduling period
The node's run cycle, such as year, month, week, day, hour, or minute. This example sets the schedule to run at
00:00every Monday and Tuesday.cron expression
This field displays the cron expression based on your schedule settings and cannot be edited.
Timeout Definition
If a task's runtime exceeds the specified timeout, the system automatically terminates it.
-
The timeout period applies to periodically scheduled instances, data backfill instances, and test instances.
-
The default timeout period is 3 to 7 days. The system dynamically adjusts the default timeout within this range based on the actual workload.
Note-
The maximum value for Timeout Definition can be set to 168 hours.
-
Charges still apply for traffic and computing resources consumed by a task that times out.
Rerun attribute
Valid values: You can run again after successful or failed operation., Do not re-run after successful operation, and re-run after failure., and Disallow Regardless of Running Status.
Auto Rerun upon Failure
This property appears if Rerun attribute is set to You can run again after successful or failed operation. or Do not re-run after successful operation, and re-run after failure.. This property is hidden if Disallow Regardless of Running Status is set to Disallow Regardless of Running Status, meaning the task will not automatically rerun if it fails.
Rerun Times
If you select the Error automatic re-run checkbox, you must configure the Rerun Times.
Rerun Interval
If you select the Error automatic re-run checkbox, you must configure the Rerun Interval. The interval can be set from 1 to 30 minutes, and the default is 30 minutes.
Validity Period
The period during which the node is active. Configure this based on your requirements.
For more information about time properties, see Configure time properties.
-
Configure scheduling dependencies
You can configure scheduling dependencies by setting ancestor nodes. This means that even if a node's scheduled time is reached, its instance triggers only after its ancestor node instances run successfully.
For example, the instance for the current node is triggered only after the instance for the insert_data ancestor node completes.
Scheduling dependencies ensure a downstream node waits for an upstream node to generate data before processing begins. For example, a downstream node might perform further data cleansing on the table generated by an upstream node, or it might write the results to another database. Both operations require the upstream node to run successfully and produce output first. This dependency model guarantees that upstream data is ready before the downstream node runs. For more information about the logic of scheduling dependencies, see Scheduling dependency configuration guide.
The scheduling system creates a default root node for each workspace. The root node is named in the WorkspaceName_root format. If a node does not have an ancestor node, it can depend directly on the root node.
Commit and deploy the task
-
On the configuration tab of the write_result node, click the
icon in the toolbar to save the node. -
Commit the task.
NoteYou must configure the Rerun attribute and specify the Parent Nodes before you can commit the task.
-
Click the
icon in the toolbar. -
In the Submit dialog box, enter a Change Description.
-
Click OK.
If you use a workspace in Standard mode, submitting the task commits it to the development environment. To deploy the task to the Production Environment for periodic scheduling, click Deploy in the upper-left corner. You can then deploy the synchronization task to the production environment on the Deploy page. For more information, see Deploy a task.
You must commit a synchronization task to the scheduling system. The system then generates instances and runs them at the scheduled times, starting the next day.
NoteIf you commit a synchronization task after 23:30, the scheduling system begins generating and running instances two days later.
-
Next steps
After configuring the task, learn how to perform routine O&M, view logs, and troubleshoot errors in the next tutorial: Run a task and troubleshoot errors.