Create table and delete table

Last Updated: Nov 12, 2017

In the previous operations, you have set a synchronization task to be run at 02:00 every Tuesday. After the task is submitted, you can view the automatic operation results in the scheduling system from the second day. Now, how can we check whether the instance schedule and dependency are as expected? DataWorks provides three triggering methods: test run, data population, and periodic running. Details about the three methods are as follows:

  • Test run: The task is triggered manually. If you need to check the timing and operation of a single task, test run is recommended.

  • Data population: The task is triggered manually. This method applies if you need to check the timing and dependencies of multiple tasks or re-execute data analysis and computing from a root task.

  • Periodic running: The task is triggered automatically. After a task is submitted successfully, the scheduling system automatically generates task instances at different time points starting from 00:00 of the second day. It checks whether upstream instances of each instance have run successfully at the scheduled time. If all the upstream instances have run successfully at the scheduled time, the current instance runs automatically without manual intervention.

NOTE:

The scheduling system periodically generates instances based on the same rules that apply in both manual and automatic triggering modes.

  • The period can be set to monthly, weekly, daily, hourly, or even by minute. The scheduling system always generates an instance for the task on the specified day or at the specified time.

  • The scheduling system only regularly runs the instance on the specified date and generates operation logs.

  • Instances rather than on the specified date are not run, and their statuses are directly changed to “Successful” when the running conditions are met. Therefore, no running logs are generated.

The following instructions show how to configure these three triggering methods.

Test run

Manually trigger the test run

  1. Click the Test Run button on the flow page.

    29

  2. As promoted on the page, click Confirm and Run.

    30

    20111

  3. Click Go to O&M Center to view the task operation status.

    32

View the information and operation logs of the test instance

Click the task name to view the instance DAG. In the instance DAG view, right-click an instance to view its dependencies and detailed information. Also, you can terminate or re-run the instance. In the instance DAG view, double-click an instance and a dialog box appears, showing the task attributes, running logs, operation logs, and code.

Note:

  • In test run mode, the task is triggered manually. The task runs immediately as long as the set time is reached, regardless of the instance’s upstream dependencies.

  • According to the previously mentioned instance generation rules, set up the task write_result to run at 02:00 each Tuesday. If the business date of test run is Monday (business date = running date -1), the instance runs at 02:00. If not, the instance status is changed to “Successful” at 2:00 and no logs are generated.

Data population

Manually trigger data population

If you need to check the timing and dependency of multiple tasks or re-execute data analysis and computing from a root task, go to the O&M Center > Task List > Task Scheduling page and click Data Population Task to run multiple tasks of a specific period of time.

Instruction

  1. Log on to the O&M Center > Task Scheduling page and enter the task name.

  2. Select the task query results and click the Data Population button.

  3. Set the business date of the data population as May 11, 2017 to May 12, 2017, select the insert_data and write_result node tasks, and click OK.

  4. Click View Data Population Results.

View the information and operation logs of the data population instance

On the Data Population Instance page, find the task instance: Click the task name to view the instance DAG. In the instance DAG view, right-click an instance to view its dependencies and detailed information. Also, you can terminate or re-run the instance. In the instance DAG view, double-click an instance and a dialog box appears, showing the task attributes, running logs, operation logs, and code.

Note:

  • Data population task instances are dependent on instances from the previous day. For example, for a data population task within the period from September 15, 2017 to September 18, 2017, if the instance on the 15th is failed to run, the instance on the 16th is not run.

  • According to the previously mentioned instance generation rules, set up the task write_result to run at 02:00 each Tuesday. If the business date selected during data population is Monday (service date = running date -1), the instance runs at 02:00. If not, the instance status is changed to “Successful” at 02:00 and no logs are generated.

Periodic automatic run

In periodic automatic run mode, the scheduling system automatically triggers tasks according to all task scheduling configurations. Therefore, no operation portal is provided on the page. You can view the instance information and operation logs in either of the following methods:

  • Go to the O&M Center > Task Scheduling page, select parameters such as service date or running date, search instances corresponding to the task write_result, and then right-click on an instance to view its information and operation logs.

  • Click the task name to view the instance DAG. In the instance DAG view, right-click an instance to view its dependencies and detailed information. Also, you can terminate or re-run the instance. In the instance DAG view, double-click an instance and a dialog box appears, showing the task attributes, running logs, operation logs, and code.

Note:

  • If the initial status of a task instance is “Not Run”, when the scheduled time is reached, the scheduling system checks whether all the upstream instances are successful.

  • The instance is triggered only when all of its upstream instances are successful and its scheduled time is reached.

  • For an instance in Not Run status, check that all its upstream instances are successful and its scheduled time has been reached.

Thank you! We've received your feedback.