To ensure that important nodes are complete as expected, you can use the baseline management feature to add the nodes to a baseline and specify a committed completion time for the baseline. DataWorks calculates the estimated completion time of the nodes in the baseline based on the status of the nodes. If DataWorks estimates that a node in the baseline cannot be complete before the committed completion time, DataWorks sends you an alert notification. This topic describes how to create and manage baselines.

Background information

An intelligent baseline enables DataWorks to identify an exception that prevents a node in the intelligent baseline from being complete as expected and send you an alert notification about the exception at the earliest opportunity. This ensures that the data you want to obtain is generated as expected in scenarios where dependencies between nodes in a baseline are complex. For more information, see Overview.

Limits

  • Editions:

    Only DataWorks Standard Edition and more advanced editions support the baseline management feature. If you use DataWorks Basic Edition, you must upgrade it to DataWorks Standard Edition or a more advanced edition before you can use the baseline management feature. For more information, see Differences among DataWorks editions.

  • Permissions:
    • Only workspace administrators, Alibaba Cloud accounts, and tenant administrators can be used to create baselines. For more information about how to assign roles, see Add workspace members and assign roles to them.
    • Only tenant administrators and baseline owners can be used to delete, modify, enable, or disable baselines.
  • Alert notification methods: Event alerts and baseline alerts can be sent by using the Email, SMS, DingTalk Chatbot, and WebHook notification methods. Take note of the following limits on the supported alert notification methods:
    • SMS: This method is supported in the Singapore (Singapore), Malaysia (Kuala Lumpur), and Germany (Frankfurt) regions. If you want to use this method in other regions, submit a ticket to contact Alibaba Cloud DataWorks technical support.
    • WebHook:
      • Only DataWorks Enterprise Edition and DataWorks Ultimate Edition support this method.
      • This method is supported in the following regions: China (Shanghai), China (Chengdu), China (Zhangjiakou), China (Beijing), China (Hangzhou), China (Shenzhen), China (Hong Kong), Germany (Frankfurt), and Singapore (Singapore).
      • Alert notifications can be sent only to WeCom or Lark by using this notification method.
    Note
    • DataWorks supports the webhook URLs of DingTalk, WeCom, and Lark. If you want to use the webhook URLs of other applications, submit a ticket to contact Alibaba Cloud DataWorks technical support.
    • If you want to receive alert notifications by text message or phone call as a RAM user, you must add the RAM user as an alert contact on the Alert Contacts page. If an error is reported when a node is running, DataWorks sends an alert notification to the specified contacts. For more information, see Configure and view alert contacts.

Create a baseline

  1. Go to the Operation Center page. For more information, see Go to the Overview page.
  2. In the left-side navigation pane of the Operation Center page, choose Alarm > Smart Baseline.
  3. On the Baselines tab, click Create Baseline. Create Baseline
  4. In the Create Baseline dialog box, configure the basic information about the baseline. Create Baseline
    Parameter Description
    Baseline Name The name of the baseline.
    Workspace The workspace of the nodes that you want to add to the baseline.
    Owner The name or ID of the baseline owner.
    Baseline Type The type of the baseline. Valid values:
    • Day-level Baseline: DataWorks monitors the nodes in the baseline by day. Select this value for nodes that are scheduled by day.
    • Hour-level Baseline: DataWorks monitors the nodes in the baseline by hour. Select this value for nodes that are scheduled by hour.
    Nodes The nodes that you want to add to the baseline.
    • Node: Enter the name or ID of a node and click Add on the right side. You can add multiple nodes to the baseline.
    • Workflow: Enter the name or ID of a workflow and click Add on the right side. By default, if you add a workflow to the baseline, all nodes in the workflow are added to the baseline.
      Note We recommend that you add only the most descendant node in a workflow instead of all nodes in the workflow to a baseline. This way, all ancestor nodes that affect the data production of the most descendant node are monitored.
    Priority The priority of the baseline. The greater the value, the higher the priority of the baseline, and the higher the priorities of nodes in the baseline. A baseline with a higher priority is scheduled preferentially in case of insufficient scheduling resources.
    Note This priority is mapped onto the priority of compute nodes in MaxCompute.
    Estimated Completion Time The estimated completion time of the baseline that DataWorks calculates based on the time that is required for the nodes in the baseline to complete during historical scheduling. If the historical data is insufficient, the The completion time cannot be estimated due to the lack of historical data message appears.
    Committed Completion Time The deadline by which the nodes in the baseline are committed to being complete. DataWorks calculates the alert time for the baseline based on the value of this parameter. Configure this parameter based on the estimated completion time. Make sure that the alert time is later than the estimated completion time of the baseline.
    Note
    • The alert time for a baseline is calculated by subtracting the alert margin threshold from the committed completion time. If a node in the baseline cannot be complete before the alert time, an alert notification is sent. For example, the committed completion time for a baseline is set to 03:30, and the alert margin threshold is set to 10 minutes. If DataWorks estimates that a node in the baseline cannot be complete before 03:20, an alert notification is sent.
    • For an hour-level baseline, you must specify hour-level instances and set the committed completion time for the instances to ensure data production.
    Alert Margin Threshold The time interval between the alert time and committed completion time for the baseline. We recommend that you configure the alert margin threshold based on the time required for the nodes in the baseline to complete.
  5. Configure the alert details of the baseline.

    You can specify whether to enable the alert notification feature for the baseline by turning on or off Enable Alerting. If you turn on Enable Alerting, you can specify the notification methods that are used to send a baseline alert if the data of a node in the baseline cannot be generated as expected. You can also specify the methods that are used to send an event alert notification if the nodes that affect the data production of the baseline or their ancestor nodes slow down or fail to run.

    Alert Details
    Parameter Description
    Enable Alerting You can turn on or off Enable Alerting to enable or disable the Event Alert and Baseline Alert features.
    • Turn on Enable Alerting:
      • If DataWorks estimates that a node in a baseline cannot be complete before the committed completion time, DataWorks sends a baseline alert notification by using the specified notification methods. For more information, see Core logic: baseline alert.
      • If a node in a baseline or an ancestor node of the node fails to run, or a node on the key path slows down, DataWorks sends an event alert notification by using the specified notification methods. You can view existing events on the Events tab. For more information, see Manage events.
      Note If Enable Alerting is turned on, you can configure the baseline alert or event alert feature based on your business requirements. We recommend that you configure both the baseline alert and event alert features for important nodes.
    • Turn off Enable Alerting: No alert notifications are sent for the baseline. However, if the baseline is enabled, baseline instances are normally generated, and the baseline priority still takes effect.
    Baseline Alert
    • Alert Notification Method: You can select Email and SMS to send alert notifications to the baseline owner, the recipients on duty on the current day in the specified shift schedule, or other specified recipients. For more information about shift schedules, see Create and manage a shift schedule. You can also select DingTalk Chatbot to set the webhook URL of a DingTalk chatbot to receive alert notifications, or select WebHook to set the webhook URL of another application to receive alert notifications. DataWorks supports the webhook URLs of DingTalk, WeCom, and Lark.
      • If you select DingTalk Chatbot or WebHook for the Alert Notification Method parameter, you can click Send Test Message in the Actions column in the table that appears to check whether an alert notification can be sent. For more information about how to configure a DingTalk chatbot, see Scenario practices: Send alert notifications to a DingTalk group.
      • If you select SMS or Email for the Alert Notification Method parameter, you can click Check Contact Information to verify whether the mobile phone number or email address of a recipient is valid.
      Note
      • You can select Phone for the Alert Notification Method parameter only in DataWorks Professional Edition or more advanced editions.
      • If you select the Phone alert notification method, DataWorks filters alert calls to prevent recipients from frequently receiving alert calls. A recipient can receive at most one alert call within 20 minutes. Other alert calls are downgraded and converted to text messages.
    • Alerting frequency control
      • Maximum Alerts: the maximum number of alert notifications can be sent. If the maximum number is exceeded, no alert notifications are sent.
      • Minimum Alert Interval: the minimum interval at which alert notifications are sent.
      • Alerting Do-Not Disturb Period: DataWorks does not send alert notifications during the period of time that is specified by this parameter.

        For example, if you set this parameter to 00:00 to 08:00 for the baseline and an exception occurs on a node in the baseline, DataWorks does not send baseline or event alert notifications. If the exception still exists after 08:00, DataWorks sends an alert notification.

    Event Alert
    • Event Type: You can select Error or Slow.
      • Error: indicates that a node in the baseline fails to run.
      • Slow: indicates that the time required by a node to complete is significantly longer than the average time required for the node to complete in the previous periods.
    • Alert Notification Method: You can select Email and SMS to send alert notifications to the baseline owner, the recipients on duty on the current day in the specified shift schedule, or other specified recipients. For more information about shift schedules, see Create and manage a shift schedule. You can also select DingTalk Chatbot to set the webhook URL of a DingTalk chatbot to receive alert notifications, or select WebHook to set the webhook URL of another application to receive alert notifications. DataWorks supports the webhook URLs of DingTalk, WeCom, and Lark.
      • If you select DingTalk Chatbot or WebHook for the Alert Notification Method parameter, you can click Send Test Message in the Actions column in the table that appears to check whether an alert notification can be sent. For more information about how to configure a DingTalk chatbot, see Scenario practices: Send alert notifications to a DingTalk group.
      • If you select SMS or Email for the Alert Notification Method parameter, you can click Check Contact Information to verify whether the mobile phone number or email address of a recipient is valid.
      Note
      • You can select Phone for the Alert Notification Method parameter only in DataWorks Professional Edition or more advanced editions.
      • If you select the Phone alert notification method, DataWorks filters alert calls to prevent recipients from frequently receiving alert calls. A recipient can receive at most one alert call within 20 minutes. Other alert calls are downgraded and converted to text messages.
    • Alerting frequency control
      • Maximum Alerts: the maximum number of alert notifications can be sent. If the maximum number is exceeded, no alert notifications are sent.
      • Minimum Alert Interval: the minimum interval at which alert notifications are sent.
      • Alerting Do-Not Disturb Period: DataWorks does not send alert notifications during the period of time that is specified by this parameter.

        For example, if you set this parameter to 00:00 to 08:00 for the baseline and an exception occurs on a node in the baseline, DataWorks does not send baseline or event alert notifications. If the exception still exists after 08:00, DataWorks sends an alert notification.

  6. Click OK. The baseline is created.

Manage baselines

Manage baselines
Operation Description
Filter baselines In the area marked with 1 in the preceding figure, you can specify filter conditions to search for baselines.

You can filter baselines by configuring the Owner, Workspace, Baseline Name, Recurrence, Priority, and Show Enabled Baselines Only parameters.

Operations related to baselines In the area marked with 2 in the preceding figure, you can click View Details, Edit, Enable or Disable, View Change Records, and Delete in the Actions column that corresponds to a baseline to perform related operations.
  • View Details: Click View Details to view the basic information about the baseline.
  • Edit: Click Edit to modify the baseline.
  • View Change Records: Click View Change Records to view the historical change records of the baseline.
  • Enable or Disable: Click Enable or Disable to enable or disable the baseline. Instances can be generated for a baseline only after you enable the baseline. A baseline instance is generated for a baseline every day if the baseline is enabled. You can view the details of the daily baseline instances on the Baseline Instances tab. For more information, see Manage baseline instances.
  • Delete: Click Delete to delete the baseline.

Add nodes to a baseline

A node can be added to only one baseline. For example, if Node A has been added to Baseline A, when you create Baseline B and add Node A to Baseline B, Node A is removed from Baseline A and added to Baseline B.
Note If a baseline that is enabled contains no nodes, the baseline becomes an empty baseline, and a baseline instance in the Empty Baseline state is generated. For more information about empty baselines, see Why is a baseline in the Empty Baseline state?.
You use one of the following methods to add nodes to a baseline:
  • On the Baselines tab, click Create Baseline to create a baseline, and add nodes to the baseline.
  • On the Cycle Task page, add one or more nodes to a baseline.
    • Find the node that you want to add to a baseline and choose More > Add to Baseline to Baseline in the Actions column.Add to Baseline
    • Select multiple nodes that you want to add to a baseline and choose More > Add to Baseline in the lower part of the page.Add to Baseline

What to do next

  • A baseline instance is generated for a baseline every day if the baseline is enabled. You can view the details of the daily baseline instances on the Baseline Instances tab. For more information, see Manage baseline instances.
  • You can go to the Operating Record page to view the operation records of baselines. For more information, see View operation records.