DataWorks:AI assistant in practice: Intelligent data O&M through natural language

Overview

The DataWorks AI assistant provides built-in inspection templates for various scenarios, including alerts, abnormal instances, data quality, and data governance. It also supports custom inspection capabilities. This topic uses "monitoring specified task failures" as an example to describe how to configure scheduled inspection tasks through the AI assistant, receive alert notifications on IM platforms such as DingTalk, and enable the AI assistant to automatically fix abnormal issues. You can follow this practice to integrate your existing alert rules in DataWorks with IM platforms such as DingTalk.

DataWorks AI assistant documentation: Use AI assistant services.

DataWorks console: https://dataworks.console.alibabacloud.com/overview.

Scenario and results

Traditional monitoring alerts typically only notify you of task errors, and the handling process still requires PC-side operations or switching between multiple applications. The DataWorks AI assistant supports integration with IM platforms such as DingTalk to proactively push alert messages to your mobile device. You can directly complete diagnostics, remediation, and rerun operations within the IM client, achieving end-to-end closed-loop handling from receiving alerts to resolving issues.

The final results of this practice are as follows:

• User user D uses natural language commands in the DingTalk application DataWorksAIAssistant01 to have the AI assistant service instance AI Assistant subscribe to failure alerts for a specified task.

• When a task fails, the AI assistant automatically pushes the alert, diagnostic results, and remediation plan to user D.

• After user D confirms the plan, the AI assistant automatically performs code modification, deployment, and instance rerun operations on behalf of user D to fix the issue.

Configuration planning

Before you start, plan the following information:

1. AI assistant instance name: The name of the AI assistant service instance created in DataWorks.

2. Execution identity: The identity (RAM user or RAM role) used by the AI assistant to perform operations.

3. IM application information: The name of the enterprise internal application created on the IM platform (such as DingTalk) and the organization it belongs to.

4. Network connection method: The network interaction method between the AI assistant and the IM platform (internal network or public network).

The planning for this example is as follows:

Step 1: Configure task monitoring alerts

• Objective: Define the tasks to monitor and the conditions that trigger alerts to ensure that the DataWorks system can generate alert events available for subscription.

• Problem addressed: The prerequisite for the AI assistant to subscribe to alerts is that a valid monitoring rule exists and can generate alert events when conditions are met. You need to first determine "which task to monitor and what conditions trigger an alert", and then create a custom monitoring rule in DataWorks Operation Center.

• General instructions:

  1. Create a custom monitoring rule in DataWorks Operation Center.

  2. In the rule, specify the objects to monitor and the trigger conditions.

  3. Alert events generated by the rule can be subscribed to by the AI assistant, in addition to being sent through the notification methods you select (SMS, email, etc.). The AI assistant subscribes to alert events regardless of the notification method configured for the alerts.

• Configuration in this example (set an error alert for the Regional Transaction Ranking Analysis task in the Retail E-commerce workspace):

  Configuration item	Description	Example value
  Rule name	A custom name for the alert rule.	Regional Transaction Ranking Analysis Failure Alert
  Rule object	The name of the task to monitor.	Regional Transaction Ranking Analysis 02
  Trigger condition	The task status that triggers an alert (error, incomplete, etc.).	Error
  Alert action	The notification method after an alert is triggered. Select based on your needs.	Select based on your needs (does not affect IM-side subscriptions).

• Result: After the configuration is complete, when the "Regional Transaction Ranking Analysis 02" task encounters an exception, DataWorks triggers an alert event. You can view the alert record in the Alert Information list in Operation Center and receive notifications configured in the alert action (SMS, email, or DingTalk message). This alert event is the content that the AI assistant can subscribe to later.

Step 2: Create an AI assistant service

• Objective: Create an AI assistant service instance so that the AI assistant can operate DataWorks on your behalf.

• Problem addressed: The AI assistant needs a runtime carrier. You must specify whose identity the AI assistant uses to perform operations (which determines permissions), what resources it uses to run (which determines resources), and what capabilities it has (which determines the scope of executable operations).

• General instructions and example configuration: Log on to the DataWorks console and create an AI assistant service instance with the following settings:

  Parameter	Description	Example value
  Instance name	A custom name for the AI assistant service.	AI Assistant
  Use scenario	Select the capability scope of the AI assistant. After you select DataWorks Assistant, the AI assistant has DataWorks O&M-related skills.	DataWorks Assistant
  Resource group	Select the serverless resource group used to run the AI assistant service. The AI assistant service continuously consumes resource group quota after it is started. We recommend that you use a separate resource group from the one used for production tasks.	General-purpose resource group
  Execution identity	The identity used by the AI assistant when it performs DataWorks operations. If you select Current logon user, all operations performed by the AI assistant are equivalent to operations performed by that user.	Current DataWorks logon user (user D)

• Result: After the instance is created, you can view the "AI Assistant" instance in the AI assistant service list on the DataWorks console, and its status is displayed as Running. At this point, the AI assistant has DataWorks O&M capabilities, but it is not yet connected to any IM platform. You cannot interact with the AI assistant through DingTalk yet. The following steps complete the IM integration.

Step 3: Connect the AI assistant to DingTalk

• Objective: Establish a communication channel between the AI assistant and DingTalk to enable real-time interaction in DingTalk.

• Problem addressed: By default, the AI assistant can only be used on the web. By binding it to a DingTalk application, you can enable IM interaction capabilities.

Step 4: Subscribe to alerts through the AI assistant

• Objective: Use natural language commands to have the AI assistant start monitoring alert events for a specified task and verify end-to-end availability.

• Problem addressed: After infrastructure configuration is complete, you need to activate the alert subscription feature of the AI assistant through explicit commands.

• General description: Send a subscription command to the AI assistant in the DingTalk chat window. You can specify the monitoring target and other information in the command: workspace ID, task name, and trigger conditions.

• Command in this example: Create alert inspection. When the alert rule "Regional Transaction Ranking Analysis Failure Alert" in workspace 959211 is triggered, immediately push the alert to this chat window with problem diagnosis and solutions..

• Result: After the command is sent, the AI assistant automatically subscribes to alert information triggered by the specified rule and replies with a confirmation message indicating that monitoring has started. From this point on, whenever the "Regional Transaction Ranking Analysis 02" task triggers a failure alert, the AI assistant proactively pushes a message in the DingTalk chat window. By default, the system scans for alert messages once every minute.

• Keyword in the command: Alert inspection.

• Alert service configuration adjustments:

  • Create an inspection: Create alert inspection, monitor <workspace name>.

  • Adjust inspection interval: Alert service inspection interval 5 minutes.

  • Modify alert source: Alert service alert source channel phone,sms.

  • Modify alert recipients: Alert service alert contacts Zhang San, Li Si (separate multiple recipients with commas; all indicates all recipients).

  • Manage monitored workspaces: Alert service add monitoring <workspace name> / Alert service remove monitoring <workspace name>.

  • Manage push targets: Alert service add push to current group / Alert service remove push <group name>.

Step 5: Handle alerts through the AI assistant

• Objective: The AI assistant pushes alert information and automatically fixes issues, forming a complete closed-loop handling process.

• Problem addressed: Achieve automatic issue remediation through a mobile device.

• General description: When an alert is pushed, it includes the root cause and a remediation plan. After you confirm that the plan is feasible, reply with an execution command to the AI assistant, and the AI assistant automatically performs the remediation operations.

• Result: In this example, the constructed task failure scenario is caused by a partitioned table query that does not specify a partition filter condition, resulting in an error. When the alert is pushed, it includes the root cause and a remediation plan. After you confirm that the plan is feasible, reply with an execution command to the AI assistant. The AI assistant then automatically performs the following operations on behalf of "user D": modify the task code or configuration → deploy to the production environment → rerun the failed instance → resolve the alert. Throughout the entire process, you do not need to leave DingTalk or switch to a PC.

The following content simulates the complete interaction between user D and the AI assistant in an IM client (such as DingTalk).

plain`<br>`L128: FAILED: ODPS-0130071:[0,0] Semantic analysis exception - physical plan generation failed: Table(demo_mp_odps_ol.ods_trade_order) is full scan with all partitions, please specify partition predicates.`<br>

Join the official communication group

First, click the application link to join the "Alibaba Cloud Big Data AI Platform" organization. Then scan the QR code below to join the AI assistant product DingTalk communication group. After joining, you can receive dedicated product technical support.

DingTalk group ID: 149605034971

DingTalk group QR code: