Overview
Real-time synchronization tasks in DataWorks Data Integration integrate with Alibaba Cloud CloudMonitor. You can view key runtime metrics in CloudMonitor, including resource utilization, business latency, record counts, synchronization rates, and DML operation statistics. Use these metrics to monitor task health and quickly identify performance bottlenecks or anomalies.
The integration uses the DataWorks → CloudMonitor approach. You only need to enable CloudMonitor to access metric dashboards and configure alert rules at no extra cost, without purchasing a separate Prometheus instance.
Prerequisites
Before you begin, ensure that the following requirements are met:
-
The Alibaba Cloud CloudMonitor service is enabled. Go to the CloudMonitor console to enable it.
-
A DataWorks workspace is created. The DataWorks workspace that contains the real-time synchronization task must be running normally.
-
The real-time synchronization task is running normally.
View metrics
Access the monitoring page
-
Log on to the Alibaba Cloud CloudMonitor console.
-
In the left-side navigation pane, select Cloud Service Monitoring. From the list of cloud services, find and click DataWorks - Data Integration Sync Task.
Filter for the target workspace
On the Data Integration task monitoring page, select the region of your DataWorks workspace from the filter at the top. Then, in the list of workspaces, click the name of your workspace to view its monitoring details.
Supported metrics
DataWorks Data Integration reports the following four categories of metrics to CloudMonitor for real-time synchronization tasks:
1. Basic metrics
-
Description: These metrics monitor the core runtime status of real-time synchronization tasks, such as latency and data transfer volume.
-
Dimensions: In the secondary filter, select Solution Task ID + Subtask ID. For solutions where the source uses database and table sharding, you must also specify the subtask ID to accurately view metrics.
|
Metric name |
Description |
Unit |
|
Business latency DIJobDelayTime |
The end-to-end synchronization latency from the source to the destination. |
ms |
|
Source records read DIJobTotalRecordsRead |
The total number of records read from the source. |
Records |
|
Destination records written DIJobTotalRecordsWrite |
The total number of records written to the destination. |
Records |
|
Source read rate DIJobSpeedRecordsRead |
The rate at which records are read from the source. |
Records/s |
|
Destination write rate DIJobSpeedRecordsWrite |
The rate at which records are written to the destination. |
Records/s |
2. Resource utilization metrics
-
Description: These metrics track the resource consumption of the nodes running real-time synchronization tasks, helping you determine whether to scale out or optimize resource configurations.
-
Dimensions:
-
In the secondary filter, select Solution Task ID + Subtask ID. For solutions where the source uses database and table sharding, you must also specify the subtask ID to accurately view metrics.
-
Worker-related metrics support filtering by Worker ID to display separate curves, which makes it easy to compare resource consumption across different worker nodes.
-
|
Metric name |
Description |
Unit |
|
Master CPU utilization DIJobMasterCpuUtilization |
The CPU utilization of the task master node. |
% |
|
Master memory utilization DIJobMasterMemUtilization |
The heap memory utilization of the task master node. |
% |
|
Master resource utilization DIJobMasterResourceUtilization |
The higher value between the CPU utilization and memory utilization of the master node. |
% |
|
Worker CPU utilization DIJobWorkerCpuUtilization |
The CPU utilization of each worker node. |
% |
|
Worker memory utilization DIJobWorkerMemUtilization |
The memory utilization of each worker node. |
% |
|
Worker resource utilization DIJobWorkerResourceUtilization |
The higher value between the CPU utilization and memory utilization of each worker node. |
% |
3. Task-level DML metrics
-
Description: These metrics track DML operations (insert, update, and delete) in real-time synchronization tasks, providing total operation counts at the task or subtask level.
-
Dimensions: In the secondary filter, select Solution Task ID + Subtask ID. For solutions where the source uses database and table sharding, you must also specify the subtask ID to accurately view metrics.
|
Metric name |
Description |
Unit |
|
Task-level source Insert DIJobTotalDmlInsertRead |
The number of Insert operations read from the source at the task level. |
Records |
|
Task-level source Update DIJobTotalDmlUpdateRead |
The number of Update operations read from the source at the task level. |
Records |
|
Task-level source Delete DIJobTotalDmlDeleteRead |
The number of Delete operations read from the source at the task level. |
Records |
|
Task-level destination Insert DIJobTotalDmlInsertWrite |
The number of Insert operations written to the destination at the task level. |
Records |
|
Task-level destination Update DIJobTotalDmlUpdateWrite |
The number of Update operations written to the destination at the task level. |
Records |
|
Task-level destination Delete DIJobTotalDmlDeleteWrite |
The number of Delete operations written to the destination at the task level. |
Records |
4. Table-level DML metrics
-
Description: These metrics track DML operations (insert, update, and delete) in real-time synchronization tasks, providing operation counts at the table level within a task or subtask.
-
Dimensions:
-
In the secondary filter, select Solution Task ID + Subtask ID + Database Name + Schema Name + Table Name. For sharded-database sources, you must also specify the subtask ID to accurately view metrics.
-
You must specify the exact table name to view the corresponding data. For read metrics, select the source database name, schema name, and table name. For write metrics, select the destination database name, schema name, and table name.
-
Read metrics
|
Metric name |
Description |
Unit |
|
Table-level source Insert DIJobTableDmlInsertRead |
The number of Insert operations read from the source for the specified table. |
Records |
|
Table-level source Update DIJobTableDmlUpdateRead |
The number of Update operations read from the source for the specified table. |
Records |
|
Table-level source Delete DIJobTableDmlDeleteRead |
The number of Delete operations read from the source for the specified table. |
Records |
Write metrics
|
Metric name |
Description |
Unit |
|
Table-level destination Insert DIJobTableDmlInsertWrite |
The number of Insert operations written to the destination for the specified table. |
Records |
|
Table-level destination Update DIJobTableDmlUpdateWrite |
The number of Update operations written to the destination for the specified table. |
Records |
|
Table-level destination Delete DIJobTableDmlDeleteWrite |
The number of Delete operations written to the destination for the specified table. |
Records |
Configure alert rules
For information about how to configure alert rules, see the CloudMonitor documentation: Create an alert rule.
FAQ
-
Q: Why can't I see Data Integration metrics in CloudMonitor?
A: Check the following items:
-
Whether the CloudMonitor service is enabled.
-
Whether a running real-time synchronization task exists in the DataWorks project. Metric reporting depends on the latest engine version. You may need to restart the task to adopt the latest version.
-
Whether you selected the correct region and DataWorks project in the primary filter.
-
-
Q: How do I view subtask metrics for database and table sharding tasks?
A: In the secondary filter, in addition to selecting the solution task ID, you must also select the corresponding subtask ID (for example, each shard subtask in a MySQL sharded-database scenario). The subtask ID comes from the metric reporting dimensions and can be searched in the drop-down list.
-
Q: Why do table-level DML metrics sometimes show no data?
A: Table-level DML metrics require you to specify the exact database name, schema name, and table name. If no corresponding DML operations occurred on the table within the selected time range (for example, no Delete operations during that period), the metric is displayed as empty or zero.