Create Node Alerts to Catch Instance Exceptions Early - DataWorks - Alibaba Cloud - DataWorks

Custom alert rules allow you to monitor task run status and resource usage based on your business requirements, so you can promptly detect and handle exceptions. This topic describes how to create custom alert rules on the Rule Management page and how to obtain a webhook URL by adding a DingTalk chatbot.

Limitations

Custom alert rules apply only to auto-triggered Scheduled Instance. They do not monitor the results of test runs or data backfill tasks.
Custom alert rules support notifications by Email, SMS, phone call, DingTalk Chatbot, and Webhook. The following limitations apply:
- Telephone: Phone call notifications are supported only for mobile phone numbers in the Chinese mainland.
- Webhook:
  - DataWorks Basic Edition supports sending alerts to DingTalk, Lark, and WeCom groups by using group webhooks.
  - DataWorks Enterprise Edition also supports user-defined webhooks for receiving alerts.
Note
If you want to send messages by using a custom webhook, adapt it by following the instructions in Custom webhooks for intelligent monitoring. After you complete the adaptation, contact us for further assistance.Submit a ticket to contact us.
- Only DataWorks Professional Edition and higher support trigger conditions such as number of instances with errors, proportion of instances with errors, and node logs contain keywords. For more information, see Features of DataWorks editions. To subscribe to DataWorks, see Subscription.
- Only the owner of the monitoring rule, a tenant administrator, or the Alibaba Cloud account can modify the monitoring rule.

Notes

When you monitor scheduled instances by using a custom alert rule, the monitoring time range varies based on the trigger condition.

Monitoring scope	Trigger condition	Description
Business date (yesterday, T)	Task-to-instance generation complete Instance count fluctuation Instance run complete Number of instances with errors Proportion of instances with errors Node logs contain keywords	DataWorks monitors scheduled instances whose business date is yesterday and whose scheduled time is today. An alert is triggered when the trigger condition is met.
Business date (yesterday T) and the day before yesterday (T-1)	Instance run incomplete Instance incomplete in cycle Instance run timeout	DataWorks monitors scheduled instances whose business date is yesterday or the day before yesterday and whose scheduled time is today or yesterday. An alert is triggered when the trigger condition is met.
Business date (yesterday T), the day before yesterday (T-1), and two days before yesterday (T-2)	Instance run error Error persisting after automatic rerun	DataWorks monitors scheduled instances whose business date is yesterday, the day before yesterday, or two days before yesterday and whose scheduled time is today, yesterday, or the day before yesterday. An alert is triggered when the trigger condition is met.

Note

Scheduled instances outside the monitoring scope do not trigger alerts even if the trigger condition is met. For details about the monitoring rules for different trigger conditions, see Configure trigger methods.

Go to Rule Management

Go to the Operation Center page.

Log on to the DataWorks console, switch to the target region, and then click Data Development and O&M > Operation Center in the left-side navigation pane. Select the target workspace from the drop-down list and click Operation Center.

In the left-side navigation pane, click Monitoring and Alerting > Rule Management.
Note
You can also go to the Scheduled Tasks page to select nodes in batches and create monitoring rules. For more information, see Create monitoring rules for scheduled tasks.

Create a custom rule

On the Rule Management page, you can create custom monitoring rules as needed.

Configure basic information

Parameter	Description
Rule Name	Enter a name for the custom rule.
Object Type	Specify the type of the monitoring object to define the monitoring granularity. Valid values: Node, Baseline, Workspace, Workflow, Exclusive Resource Group for Scheduling, and Exclusive Resource Group for Data Integration. Note When the object type is baseline, only the run status of tasks on the baseline is monitored. Upstream tasks of baseline tasks are not monitored. If you also want to monitor upstream tasks of baseline tasks, see Intelligent baseline.
Rule Object	Specify the monitoring objects. Enter the name or ID of a monitoring object, select the desired object from the list, and then click Add. The supported monitoring object types and their maximum quantities are as follows: Node: 50 Baseline: 5 Workflow: 5 Workspace: 1
Task Blacklist	Specify which tasks among the monitoring objects are excluded from the current rule. When Object Type is set to Baseline, Workspace, or Workflow, you can enter a node name or ID and click Add to add the task to the blocklist. Note Tasks in the blocklist are not monitored. You can add up to 50 tasks to the blocklist.
Resource Group Name	Specify the name of the exclusive resource group to monitor. When Object Type is set to Exclusive Resource Group for Scheduling or Exclusive Resource Group for Data Integration, select the Resource Group Name to monitor from the drop-down list.

Configure trigger methods

Note

In the custom alert logic, a frozen task is considered complete.

Object type	Trigger condition	Description
Node, Baseline, Workspace, Workflow	Complete	The system monitors a task from the time it starts running. An alert is sent when the task runs successfully. When the object type is Baseline or Workflow, the alert is triggered only when all tasks on the baseline or in the workflow run successfully. When the object type is Node and multiple nodes are added, the alert is triggered only when all nodes finish running. When the object type is Workspace, the trigger condition cannot be set to Complete. Note For hourly tasks, the task is considered complete only after all cycles run successfully.
	Incomplete	The system monitors a task from the time it starts running. If the task has not finished running by the specified target time, an alert is sent. Note This alert differs from the intelligent baseline alert policy. An intelligent baseline can detect exceptions that may prevent tasks on the baseline from being completed on time and send early warnings. For more information, see Intelligent baseline. Examples: Scenario 1: A task is scheduled to run at 01:00, and the incomplete time is set to 02:00. The task starts running at 01:00. If the task has not finished running by 02:00, an alert is sent. Scenario 2: A task is scheduled to run every hour from 00:00 to 23:59 every day, and the incomplete time is set to 12:00. The rule triggers an alert every day. Scenario 3: A baseline completion time is set to 10:00. If any task on the baseline has not been completed by 10:00, an alert is sent. Note For hourly or minute-level tasks, the system checks whether all cycles have not been completed by the specified time on that day.
	Error	The system monitors a task from the time it starts running. If the task encounters an error, an alert is sent. Note An instance run error means that the target instance shows the status in the basic information column under Operation Center > Auto Triggered Task O&M > Scheduled Instance. When the object type is Baseline, Workspace, or Workflow, an alert is triggered as long as any task on the baseline, in the workspace, or in the workflow encounters an error. An alert is triggered each time a task encounters an error. If a task is configured to trigger an alert twice upon errors and the task fails twice during reruns, four alerts are triggered in total. If you want to trigger an alert only when a task still encounters an error after automatic reruns, set the trigger condition to Error Persisting After Automatic Rerun of Node. When you select Error as the trigger condition, you can select Alert upon Automatic Rerun. When enabled, an alert is also triggered when a failed instance triggers an automatic rerun for a task that has automatic rerun configured.
	Number of instances with errors	Indicates the number of instances in the error state, including instances that fail Data Quality checks and instances that fail due to code logic errors. When the object type is Workspace and the Trigger Condition is set to Instances with Errors, you must configure a threshold. An alert is triggered when the number of instances with errors reaches the threshold. Note This feature is available only in DataWorks Professional Edition and higher. `Today` refers specifically to scheduled instances whose business date is yesterday and whose scheduled time is today.
	Proportion of Instances with Errors	When the object type is Workspace and the Trigger Condition is set to Proportion of Instances with Errors, you must configure a threshold. An alert is triggered when the proportion of instances with errors to the total number of instances today reaches the threshold. Note This feature is available only in DataWorks Professional Edition and higher. `Today` refers specifically to scheduled instances whose business date is yesterday and whose scheduled time is today.
	Node Logs Contain Keywords	When the object type is Workspace and the Trigger Condition is set to Node Logs Contain Keywords, you must configure keywords. An alert is triggered when the runtime logs of tasks today contain the specified keywords. Note This feature is available only in DataWorks Professional Edition and higher. `Today` refers specifically to scheduled instances whose business date is yesterday and whose scheduled time is today. For exclusive resource groups created before August 24, 2025, if you want to use the node logs contain keywords alert feature, submit a ticket to contact us for an upgrade. Otherwise, you cannot receive the related alerts.
	Incomplete in Cycle	The system monitors a task within a specified cycle. If the task has not finished running within the cycle, an alert is sent. This condition is typically used for instances scheduled on an hourly basis. When a workflow is configured with the Incomplete in Cycle monitoring condition, the system monitors the Nth cycle of tasks in the workflow (such as daily, hourly, and minute-level tasks) based on the cycle N you specify. If the number of instances is less than N, the alert for that task is ignored. For example, if the cycle is set to 3, the workflow contains the following two tasks, and the alert monitoring works as follows: Hourly task A: Scheduled every 2 hours with a runtime of 25 minutes. The start time is 00:00 every day, so the task has 12 cycles per day (24 hours). 00:00 is the first cycle, and the 3rd cycle is at 04:00. Under normal conditions, the 3rd cycle finishes at 04:25. If the trigger condition is set to Incomplete in Cycle and the 3rd cycle is specified to be complete by 04:30, an alert is sent if the task has not finished by 04:30. Minute-level task B: Scheduled every 10 minutes with a runtime of 2 minutes. The start time is 00:00 every day, so the task has 6 cycles per hour. 00:00 is the first cycle, and the 3rd cycle is at 00:20. Under normal conditions, the 3rd cycle finishes at 00:22. If the trigger condition is set to Incomplete in Cycle and the 3rd cycle is specified to be complete by 00:23, an alert is sent if the task has not finished by 00:23.
	Timeout	The system monitors a task from the time it starts running. If the task has not finished running after the specified duration, an alert is sent. This condition is typically used to monitor task duration.
	Error Persisting After Automatic Rerun of Node	The system monitors a task from the time it starts running. If the task encounters an error and still fails after automatic reruns, an alert is sent. Note If you want to trigger an alert each time a task encounters an error, set the trigger condition to Error.
	Instance Generated	This trigger condition is available only when the object type is Workspace.
	Fluctuation of Instance Count	This trigger condition is available only when the object type is Workspace. DataWorks generates scheduled instances for the next day before 24:00 every day. You receive this alert when the number of instances in the workspace fluctuates significantly compared with the historical average.
Exclusive Resource Group for Scheduling, Exclusive Resource Group for Data Integration	Resource Group Usage	An alert is triggered when the Usage exceeds a specified value for a specified duration. Example: An alert is triggered when Usage exceeds 50% for 15 consecutive minutes.
	Number of instances waiting for resources in resource group	An alert is triggered when Number of Instances Waiting for Resources in Resource Group exceeds a specified value for a specified duration. Example: An alert is triggered when Number of Instances Waiting for Resources in Resource Group exceeds 10 for 15 consecutive minutes.

Configure alert actions

Alert notification method	Alert recipient	Description
Email, SMS, phone call	Alerts can be sent to Node Owner, Shift Schedule, and Others.	You can click Verify Contact Information to verify whether the phone number or email address is correctly configured. You must subscribe to DataWorks Professional Edition or higher to use the phone alert feature. If you set Alert Notification Method to Telephone, you must acknowledge the following: To prevent a large number of alert phone calls in a short period, DataWorks filters alert phone calls. A user can receive at most one alert phone call within 20 minutes. Additional alert phone calls are downgraded to SMS messages. You must configure a duty roster first, and then select the duty roster as the alert recipient. For more information about how to configure a duty roster, see Configure a duty roster. Note After you configure a duty roster, the first two alerts are sent only to the primary on-duty person. Starting from the third alert, the system sends alert notifications to both the primary and backup on-duty persons. Phone alerts are supported only for mobile phone numbers in the Chinese mainland. If you want to send alerts to overseas phone numbers, you must set up a custom webhook service. DataWorks pushes messages to your webhook service, which then forwards the messages to the phone.
DingTalk ChatbotWebhook	Group members	You can configure the @ policy to @ @all or @ Specify Recipients. Click Send Test Message in the Actions column to test whether the alert can be sent normally. If you do not receive the alert, troubleshoot the issue by referring to Troubleshoot alert issues. The security settings of the DingTalk chatbot support only keyword configuration. The keywords must include DataWorks.

Configure alert fatigue control

Parameter	Description
Maximum Alerts	The maximum number of alerts. No more alerts are sent after this number is reached.
Minimum Alert Interval	The minimum time interval between two consecutive alerts.
Alerting Do-Not-Disturb Period	After you set a do-not-disturb period, no alerts are sent during that period. For example, if the trigger conditions are set to Timeout, Error, and Incomplete, and the do-not-disturb period is set to `00:00` to `08:00`, no alert is sent during this period. If the task is still in an abnormal state at 08:00, an alert is sent.

After you complete the configuration, click OK to generate the rule. On the Rule Management page, you can click View Details, Enable/Disable, or Delete next to a rule to perform the corresponding operations.

View Details: Click View Details to view the basic information of the rule.
Enable or Disable: Controls the status of the rule. The rule monitors task execution only after it is enabled. You can view alert details in the Alert details panel.
Delete: Click Delete to directly delete the rule.

Modify a monitoring rule

A created rule can be modified only by the owner of the monitoring rule, a tenant administrator, or the Alibaba Cloud account.

Best practice: Send alert messages to a DingTalk group

Open the target DingTalk group to which you want to send alerts, and click the Group Settings icon in the upper-right corner.
Click Smart Group Assistant.
On the Smart Group Assistant page, click Add Robot.
On the Group Robots page, click the icon.
On the Select a Robot to Add page, click Custom.
In the Custom dialog, click Add.

In the Add Robot dialog, configure the parameters.

Parameter	Description
Robot Name	The custom name of the robot.
Add to Group	The group to which the robot is added. This value cannot be modified.
Custom Keywords	After you set custom keywords, only messages that contain the keywords can be sent. You must add DataWorks as a keyword. Note that keywords are case-sensitive. Note You can set up to 10 keywords. A message must contain at least one of the keywords to be sent successfully.

Select I have read and agree to the Custom Robot Service and Disclaimer, and click Complete.
After you complete the security settings, copy the Webhook URL of the robot, and click Complete.
Important
Keep the webhook URL secure. Exposure of the URL may pose security risks.
Go to the Rule Management page, click Create Custom Rule, set the alert notification method to DingTalk Chatbot, and paste the Webhook URL into the DingTalk Chatbot Webhook URL field.

FAQ

Q: How do I avoid frequent alert notifications?

A: You can configure alert fatigue control to set the maximum number of alerts, alert intervals, and a do-not-disturb period. For details, see Alert fatigue control.