DataWorks:Data Integration DI Agent

Capability overview

DI Agent supports creating and managing all four types of sync tasks through natural language: batch single-table, real-time single-table, batch full-database, and real-time full-database. Instead of filling out configuration forms for data sources, field mappings, and scheduling strategies one by one, you only need to describe your sync requirements, and the Agent completes the entire process from intent understanding to deployment. Key capabilities include:

• Natural language task creation: Describe your sync requirements in one sentence. For example, "Sync the RDS orders database to Hologres in real time, partitioned by day." The Agent automatically completes data source identification, schema detection, field mapping, resource group assignment, and scheduling configuration.

• Intent understanding and multi-turn dialog: The Agent understands vague or incomplete descriptions and asks follow-up questions to fill in key parameters such as partition strategies and incremental conditions, ensuring the generated task configuration is accurate and usable.

• Full lifecycle management: You can manage tasks through conversation by editing, pausing, resuming, deleting, or rerunning them. The conversational interface covers the entire task lifecycle from creation to decommission.

• Batch operations: Create or modify multiple sync tasks in a single conversation. This is useful for full-database migrations and bulk scheduling adjustments.

Usage example

The following example demonstrates how to create a sync task through conversation:

1. Open the DI Agent conversational interface.

2. Enter your sync requirements. For example, "Sync the RDS orders database to Hologres in real time, partitioned by day."

3. The Agent automatically detects the source schema, displays the list of tables to sync, and proceeds after you confirm.

4. The Agent automatically completes field mapping, resource group assignment, and scheduling configuration.

5. Preview the task configuration summary and deploy the task after confirming it is correct.

6. Check the task running status and verify that data is synced as expected.

Capability overview

DI Agent supports embedding large language model inference as an ETL operator in data pipelines to perform batch AI processing on unstructured data in OSS, such as images, videos, audio files, and documents. Processing capabilities include recognition, classification and tagging, transcription, translation, summarization, and vectorization. Processing results are written directly to MaxCompute, Hologres, or Data Lake Formation (DLF), bringing unstructured data into the enterprise data asset system. Key capabilities include:

• Natural language processing logic: Describe your processing intent in natural language, such as "Identify product categories in images and output tags." The Agent automatically generates the corresponding processing pipeline configuration.

• Three-stage pipeline orchestration: The Agent automatically orchestrates a Source → Transform (large model inference) → Sink pipeline without requiring you to write scripts or set up inference services.

• Direct data warehouse output: Processing results are written directly to MaxCompute, Hologres, or DLF without additional data transfer steps.

• Data embedding: Vectorize processed data and write the results to Hologres for downstream RAG use cases.

• Batch and scheduled execution: Process millions of files in batch mode and configure scheduled periodic runs for continuous incremental processing of new data.

Usage example

The following example demonstrates how to process product images in OSS and write the results to MaxCompute:

1. Open the AI data processing entry point.

2. Enter your processing requirements. For example, "Read product images from an AWS S3 bucket → call a large model to identify category tags → write the results to a MySQL table."

3. The Agent automatically orchestrates a three-stage pipeline: Source (S3) → Transform (large model inference) → Sink (MySQL).

4. ChatDB automatically identifies the target table and generates the corresponding SQL statement.

5. View the returned chart results. The visualization type (line chart, bar chart, etc.) is automatically selected.

6. To vectorize data from a feature engineering database into Hologres for RAG use, you can define the embedding task in natural language: "Continue with embedding → call a large model for vectorization → write the results to a Hologres table."

7. The Agent automatically orchestrates a three-stage pipeline: Source (MySQL) → Transform (large model vectorization) → Sink (Hologres).

8. ChatDB automatically identifies the target table and generates the corresponding SQL statement.

9. View the returned results.

Capability overview

When you receive a sync task alert, you can describe the issue or paste the alert information in the Agent conversational interface. The Agent automatically aggregates multi-dimensional information, including task logs, resource monitoring metrics, and source database status, to identify the root cause and provide a remediation plan. This reduces troubleshooting time from hours to minutes. Each diagnosis result is automatically archived as a knowledge base entry, so the next time a similar issue occurs, the existing solution is matched directly, eliminating the need to start from scratch when team members change. Key capabilities include:

• Multi-dimensional information aggregation: Automatically collect and correlate task logs, resource monitoring metrics, source database status, network bandwidth, and other multi-dimensional information, and present the complete context in a single view.

• Root cause identification: Automatically analyze fault root causes based on aggregated information, such as insufficient CUs, source table locks, network bandwidth bottlenecks, or configuration errors, and provide clear conclusions.

• Remediation recommendations with one-click execution: After the root cause is identified, the Agent provides actionable remediation suggestions, such as scaling out resource groups, restarting instances, or adjusting concurrency. One-click execution is supported after you confirm.

• Automatic knowledge accumulation: Each diagnosis result is automatically archived as a knowledge base entry. The next time a similar issue occurs, the existing solution is matched directly.

Usage example

1. After an alert is triggered, open the Agent conversational interface and describe the issue or paste the alert message.

2. The Agent automatically aggregates multi-dimensional information including task logs, resource monitoring metrics, and source-side status.

3. The Agent outputs a root cause analysis report (for example, insufficient CUs causing the consumption rate to fall below the production rate).

4. The Agent provides recommended fixes (for example, scaling up the resource group CU quota or restarting failed instances). After you confirm, the fix is executed.

Capability overview

DI Agent supports proactive scanning of all sync pipelines at configurable intervals, shifting the O&M model from reactive response to proactive governance. Inspection results are presented as structured reports with high-risk tasks prioritized by impact scope and urgency, each accompanied by specific remediation recommendations and expected benefit assessments. Key capabilities include:

• Configurable inspection intervals: Four frequencies are supported — hourly, daily, weekly, and manual-only — to accommodate different business requirements for health check timeliness.

• Full-pipeline proactive scanning: The Agent automatically scans all sync pipelines at each configured interval to proactively identify potential risks without manual item-by-item checks.

• Structured daily health report: Inspection results include a key metrics overview, an anomalous task list, and risk level assessments, enabling O&M teams to grasp the overall health status within 3 minutes.

• Prioritization and remediation recommendations: High-risk tasks are prioritized by impact scope and urgency, with each item accompanied by specific remediation recommendations and expected benefit assessments.

Usage example

1. Go to the intelligent inspection configuration page and select an inspection interval (hourly, daily, or weekly).

2. Configure the inspection scope (all pipelines, or specific workspaces/resource groups).

3. The Agent automatically runs inspections at the configured interval and generates structured reports.

4. Review the key metrics overview and high-risk task list in the report, and address risk items by priority.