Batch sync nodes based on MaxCompute require that the correct data be generated in a timely manner. This topic describes how to use the Monitor service of DataWorks to monitor data timeliness.

Prerequisites

DataWorks Standard Edition or a more advanced edition is purchased. Complete features of the Monitor service are only available in DataWorks Standard Edition or more advanced editions. For more information, see Feature comparison among DataWorks editions. For more information about features provided by the Monitor service, see Overview.

Background information

Before you create rules to monitor data timeliness, you must determine the priorities of nodes. A node with a higher data class has a higher priority. You can configure stricter timeliness monitoring and alerting rules for the node.

Procedure

  1. Go to the Rule Management page.
    1. Log on to the DataWorks console. Go to the DataStudio page, click the DataWorks icon in the upper-left corner, and then click Operation Center.
    2. On the page that appears, choose Alarm > Rule Management in the left-side navigation pane. The Rule Management page appears. For more information about the operations that you can perform on the Rule Management page, see Manage alert triggers.
  2. Create a custom rule.
    Click Create Alert Trigger in the upper-right corner. In the Create Alert Trigger dialog box that appears, specify the parameters and click OK. In this example, create a rule to monitor the Workshop workflow and trigger an alert if the running duration of the workflow exceeds 30 minutes. Configure the system to send SMS and email notifications when the rule triggers the same alert for the Workshop workflow for three consecutive times.
    Parameter Description
    Trigger Name The name of the alert trigger.
    Object Type The granularity of monitored objects. Valid values: Node, Workflow, Baseline, and Workspace.
    Object The monitored object. Enter the name or ID of a node or workflow and click the icon on the right to add the object.
    Trigger Condition The conditions for triggering alerts. Valid values: Completed, Uncompleted, Error, Cycle Uncompleted, and Overtime.
    Maximum Alerts The maximum number of alert reporting times. If the number of alerting times exceeds the preset threshold, no alerts are generated.
    Minimum Alert Interval The minimum time interval at which DataWorks reports alerts.
    Quiet Hours The specified period during which no alerts are reported.
    Notification Method The method of reporting alerts. Valid values: Email and SMS.
    Recipient The person who receives alerts. You can set this parameter to the node owner or another recipient.
    DingTalk Chatbot The DingTalk chatbot for receiving alerts. For more information about how to add a DingTalk chatbot and obtain a webhook URL, see the following section.
    Note If you want to add multiple DingTalk chatbots, add multiple webhook URLs.

    You can also create a rule with other trigger conditions to monitor a specific node.

  3. Improve the data timeliness.
    The following table lists the common causes of poor timeliness of data and provides improvement measures.
    Cause Improvement measure
    Insufficient computing resources
    • The total resource capacity is insufficient. For example, the nodes you commit require more resources than those available.
    • Resources are improperly allocated. Important nodes are not allocated resources preferentially.
    Scale out resources or allocate exclusive resources to core computing nodes.
    Low execution efficiency of code
    • Redundant code exists. For example, the code that scans all partitions exists.
    • Nodes are improperly configured. For example, long-tail computing occurs.
    Adjust the execution time of nodes. Postpone nodes with lower priorities during peak hours.
    No emergency plans available for O&M staff Fully test each node before you deploy the node.