DataWorks:Intelligent Monitoring

Recurring nodes

Baselines

This applies to AI analysis after alerts are enabled for an existing key business baseline.

1. In Operation Center, go to Intelligent Baseline > Baseline Management and find the target baseline.

2. Make sure that the baseline is enabled, and configure the guaranteed tasks, the committed time, and the warning settings as required by your business.

3. In the Actions column of the baseline, click Intelligent Monitoring.

4. On the Monitoring Configuration tab, turn on Enable Intelligent Monitoring and configure a webhook for diagnostic results as needed.

5. Click OK and return to the list to verify the Monitoring indicator.

After monitoring is enabled, when the baseline triggers a warning or a baseline break alert, the system generates a work item and assigns an agent for analysis. AI monitoring does not replace the committed time, warning configuration, or original alert notifications of the baseline itself.

Custom rules

This applies to enabling AI analysis for tasks that require a custom monitoring scope or alert threshold.

1. In Operation Center, go to Monitoring and Alerting > Rule Management.

2. Create or edit a custom rule, and set the rule name, owner, object type, monitored object, trigger conditions, and alert actions. Make sure that the rule itself is enabled.

3. For a supported trigger condition, in the Alert Action section, select Enable Intelligent Monitoring, configure diagnostic result notifications as needed, and then save the rule.

4. You can also click Intelligent Monitoring or Monitoring in the Actions column of an existing rule, and then enable and save it in Monitoring Configuration.

The following custom rule trigger conditions support Intelligent Monitoring. The conditions that you can actually select are also limited by the monitored object and the availability scope of the feature.

Configure baselines or custom rules in batches

Unlike batch configuration for nodes, batch configuration for baselines and rules supports item-by-item checks and result display.

1. In the corresponding list or in Configure Monitoring of the Workbench, select multiple objects of the same type.

2. Click the batch monitoring entry and view the number of objects that can be configured and the number of objects that are skipped.

3. Set Monitoring Status to On or Off.

4. When you turn it on, select how diagnostic result notifications are handled.

5. Click Apply Settings and view the Succeeded, Skipped, and Failed results item by item.

6. If failed items exist, click Recheck Failed Items, and verify the failed objects before you handle them.

If the page indicates that no result has been obtained, the operation may have already been performed. Verify the object status first, and then use the retry provided in the current dialog box. Do not interpret a failed response as meaning that nothing was saved.

Send diagnostic results to message channels

Add a webhook in the monitoring configuration of the object. Enter a valid HTTP or HTTPS address. You cannot add the same address more than once.

• Node monitoring uses an address list. You can add multiple addresses and test them one by one. The message type and @ mention policy options available in the baseline and rule forms are not provided.

• Baseline and rule monitoring provides message types such as DingTalk, Lark, WeChat, and Custom. For DingTalk and Lark, you can configure @All or @Owner, which are mutually exclusive. Other message types do not provide this @ mention policy.

• After you click Send Test Message, confirm on the receiving end that the message is delivered. The test only verifies the message channel. It does not simulate a task exception and is not equivalent to generating a real diagnostic result.

• Diagnostic result notifications for baselines and rules are sent after the diagnosis is completed and contain the result and a link to the work item. If you leave the field empty, no notification is pushed. This is a different notification from the original alert message.

Message delivery for node monitoring is used to receive analysis conclusions. Do not conclude that monitoring did not run only because you did not receive a group message. Also check the monitoring records and the work items in the Workbench.

Configure and authorize an IM bot

To start handling directly from an IM group after you receive a monitoring notification, you must first configure an IM bot and authorize it to help handle the work items owned by the current user.

Prerequisites:

• A notification webhook is configured for the monitored object, and the target group can receive monitoring notifications that contain work item information.

• An enabled IM bot already exists in the same tenant, and the bot is bound to the Data O&M agent.

• The IM bot has joined the group that receives monitoring notifications. The user who starts the handling is the owner of the target work item and has the required O&M operation permissions.

Procedure:

1. In Operation Center, go to AI Workbench > Workbench and click Agent Collaboration Settings.

2. If no bot is available in the current tenant, click Configure IM Bot, create and enable a bot in the unified IM gateway, and bind the agent type Expert Domain Agent - Data O&M to it.

3. Return to Agent Collaboration Settings, and search for and select the bot that you want to authorize in Collaboration Bots. You can authorize up to 50 IM bots in the same tenant, and you can also select bots created by other members of the same tenant.

   Note

   Agent collaboration settings are isolated per user. Through these settings, the owner of a work item authorizes a collaboration agent to perform related operations on the owner's behalf within the owner's own permission scope.

4. Click Save.

5. In the target group, quote the work item notification sent by the notification bot, @ mention the authorized IM bot, and describe the handling requirements. After the bot replies that it has started the handling, you can continue to view the progress and add instructions in the group.

After Intelligent Monitoring detects a supported exception, the system creates or updates the corresponding work item and triggers an agent analysis. You can view the analysis progress, diagnostic results, and handling recommendations in the work item. Work items are the unified entry for you to handle all automated diagnostic results.

Access work items

You can view work items in two ways:

• From the object: Open the monitoring dialog box of a node, baseline, or rule, switch to Monitoring Records, and click View Details of the corresponding work item.

• Centralized view: In Operation Center, go to AI Workbench > Workbench and search by source, task instance ID, task ID, baseline ID, custom rule ID, work item ID, or title. You can also aggregate work items by node or baseline to view the related instances.

Understand work item details

On the work item details page, you see a structured diagnostic report. When you view a record, also verify the source, data timestamp, and instance cycle to avoid treating another instance of the same node or an issue from a different source as the target of this handling.

• Visibility: The work item list and summary are visible to all users who have permissions to access the workspace.

• Handling permissions: The agent diagnostic details and the follow-up handling conversation are available only to the owner of the current work item.

• Ownership: The owner of a node monitoring work item is the owner of the node by default, regardless of who configured the monitoring.

• Permission isolation: The operation permissions of the agent are exactly the same as those of the currently logged-on user. You cannot act on behalf of another owner to perform operations beyond that owner's permission scope.

Handle work items

After the issue is understood, the owner can perform the handling operations in either of the following two ways.

1. Handle in the Workbench:

   1. Review the plan: View the diagnostic evidence and recommended actions in "Summary".

   2. Confirm interactively: Go to "Agent Diagnostic Details", converse with the agent, supplement necessary information such as the target resource group or the confirmation that a rerun is allowed, and ask the agent to provide a clear handling plan.

   3. Authorize execution: Carefully verify the plan proposed by the agent, such as rerunning an instance or modifying code, confirm the impact scope, and then authorize the agent to execute the plan.

   4. Track progress: Track the execution result of each step in the conversation or in "Activity Log".

2. Handle in an IM tool (such as DingTalk)

   1. Start a request: After you receive a webhook notification, quote the notification in the group chat and @ the authorized collaboration bot (for example, send "Analyze this work item and provide a handling plan").

   2. Confirm the plan: The agent analyzes the context of the work item and proposes a plan.

   3. Authorize execution: The agent performs the subsequent O&M operations only after you confirm the plan in the group chat.

   4. Track progress: You can continue to ask "Check the progress" or "Check the latest run status" in the group to confirm whether the operation succeeded.

Example of handling a code error: When a task fails because of a code error, you can quote the notification and state "Check the cause of the current instance failure, and confirm the code that needs to be modified, the deployment, and the rerun scope". After the agent provides a plan, confirm the execution. If the code modification or deployment is not complete, fix and deploy the code in DataStudio first, and then handle the target failed instance. Do not treat an analysis recommendation or a deployment request as a successful deployment.

Example of handling insufficient resources: When a task fails because of insufficient scheduling resources, the agent can propose plans such as migrating the resource group of the current instance and then rerunning it, or rerunning it in the original resource group. After you confirm the target resource group, check the results of the migration taking effect, the rerun being accepted, the instance running, and the instance running successfully. Migrating the resource group of the current instance is not the same as modifying the node definition. If subsequent instances still use the original resource group, evaluate and modify the node configuration separately.

Verify recovery and update the status

After the handling is complete, the system continues to track the status. The owner should also verify the actual run results:

1. Check whether the target instance ran successfully, and verify the data timestamp, the cycle, and the scope of this handling.

2. If a baseline is involved, check the output of the corresponding baseline instances and key tasks. Do not conclude that the entire baseline met its goal only because one instance succeeded.

3. If a baseline event is involved, check the recovery status and recovery time of the event and the affected baselines.

4. View the latest analysis, the operations that have been performed, and the recovery results in the Summary and Activity Log sections of the work item.

After node monitoring finds that a tracked instance has recovered, it automatically updates the corresponding work item to Closed. The recovery can come from a rerun configured for the node, manual handling by a user, or agent handling confirmed by a user. Automatic closure does not mean that the agent performed the fix on its own.

For work items that are not automatically closed, the owner updates the status after verifying the result. To ignore a work item, first confirm that the issue no longer needs to be handled. Work items from different sources should be evaluated for recovery based on the status of their own resources. Do not apply the automatic closure rule of node monitoring to all of them.

The monitoring switch, the agent status, the work item status, and the instance run status indicate different types of progress. Monitoring only indicates that the enabled configuration has been saved. Pre-processing Completed only indicates that this round of AI analysis is complete. Rerun Accepted does not mean that the instance ran successfully.