edit-icon download-icon

Perform periodic O&M and view log troubleshooting results

Last Updated: Apr 03, 2018

In the previous operations, you have set a synchronization task to run at 02:00 every Tuesday. After the task is submitted, you can view the automatic operation results in the scheduling system from the next day.

Now, how can we check whether the instance schedule and dependency are as expected? To work this out, DataWorks provides three triggering methods: test run, data population, and periodic running, which are described as follows:

  • Test run: The task is triggered manually. If you must check the timing and operation of a single task, test run is recommended.

  • Data population: The task is triggered manually. This method applies if you must check the timing and dependencies of multiple tasks or re-execute data analysis and computing from a root task.

  • Periodic running: The task is triggered automatically. After successful submission, the scheduling system automatically generates task instances at different time points starting from 00:00 of the next day. It checks whether upstream instances of each instance have run successfully according to the scheduled time. If all the upstream instances have run successfully at the scheduled time, the current instance runs automatically without manual intervention.

Note:

The scheduling system periodically generates instances based on the same rules that apply to both manual and automatic triggering modes.

  • The period can be set to monthly, weekly, daily, hourly, or even by minute. The scheduling system always generates an instance for the task on a specified day or at a specified time.

  • The scheduling system regularly runs the instance on a specified date and generates operation logs.

  • Instances rather than on a specified date does not run, and their statuses are directly changed to “Successful” if the running conditions are met. Therefore, no running logs are generated.

Procedure

The following procedures show how to configure these three triggering methods.

Test run

Manually trigger the test run

  1. Click the Test Run button on the flow page.

    29

  2. As prompted on the page, click OK and Run.

    30

    20111

  3. Click Go to O&M Center to view the task operation status.

    32

View the information and operation logs of the test instance

Click the task name to view the instance DAG. In the instance DAG view, right-click an instance to view its dependencies and more information. Also, you can terminate or re-run the instance. In the instance DAG view, double-click an instance and a dialog box appears, showing the task attributes, running logs, operation logs, and code.

Note:

  • In test run mode, the task is triggered manually. The task runs immediately if the set time is reached, regardless of the instance’s upstream dependencies.

  • According to the previously mentioned instance generation rules, set up the task write_result to run at 02:00 every Tuesday. If the business date of test run is Monday (business date = running date -1), the instance runs at 02:00. If not, the instance status is changed to “Successful” at 02:00 and no logs are generated.

Data population

Manually trigger data population

If you must check the timing and dependency of multiple tasks or re-execute data analysis and computing from a root task, go to the O&M Center > Task List > Cycle Task page and click PatchData to run multiple tasks of a specific period of time.

  1. Log on to the O&M Center > Cycle Task and enter the task name.

  2. Select the task query results and click PatchData. See the following figure.

    PatchData

  3. Set the business date of the data population as May 11, 2017 to May 12, 2017, select the insert_data and write_result node tasks, and click OK.

  4. Click PatchData Instance. See the following figure.

    PatchDataInstance

View information and operation logs of patchdata instance

On the PatchData Instance page, find the task instance: Click the task name to view the instance DAG. In the instance DAG view, right-click an instance to view its dependencies and more information. Also, you can terminate or re-run the instance. In the instance DAG view, double-click an instance and a dialog box appears, showing the task attributes, running logs, operation logs, and code.

Note:

  • Data population task instances depends on the previous day instances. For example, for a patchdata task within the period from September 15, 2017 to September 18, 2017, if the instance on the 15th fails to run, the instance on the 16th is not run.

  • According to the previously mentioned instance generation rules, set up the task write_result to run at 02:00 each Tuesday. If the business date selected during patchdata is Monday (service date = running date -1), the instance runs at 02:00. If not, the instance status is changed to “Successful” at 02:00 and no logs are generated.

Periodic automatic run

In periodic automatic run mode, the scheduling system automatically triggers tasks according to all task scheduling configurations. Therefore, no operation portal is provided. You can view the instance information and operation logs by using either of the following methods.

  • Go to the O&M Center > Cycle Task page, select parameters such as service date or running date, search instances corresponding to the task write_result, and then right-click an instance to view its information and operation logs.

  • Click the task name to view the instance DAG. In the instance DAG view, right-click an instance to view its dependencies and more information. Also, you can terminate or re-run the instance. In the instance DAG view, double-click an instance and a dialog box appears, showing the task attributes, running logs, operation logs, and code.

Note:

  • If the initial status of a task instance is “Not Run”, when the scheduled time is reached, the scheduling system checks whether all the upstream instances are successful or not.

  • The instance is triggered only when all of its upstream instances are successful and its scheduled time is reached.

  • For an instance in a “Not Run” status, check that all its upstream instances are successful and its scheduled time is reached.

Thank you! We've received your feedback.