A full and incremental synchronization task for full databases synchronizes the database and table schemas, historical data, and ongoing changes of a source database to a destination such as MaxCompute. This topic describes common O&M operations, troubleshooting methods, tuning recommendations, and alert configuration for these tasks.
Applicability and boundaries
Before you apply this topic, assess your task form and data path.
| Check item | Applicable scope | When it does not apply |
| Task form | Historical full data + real-time incremental + scheduled output or merge | For message sources that write data in real time only, see Real-time full-database synchronization: Operations and tuning. |
| Data path | Database source → MaxCompute, a data lake, or a destination that produces snapshots or partitions | The methods in this topic do not apply when the destination does not support scheduled output or merge. |
The methods in this topic rely on the following task capabilities: full data initialization for newly added tables, the handoff between the full and real-time stages, checkpoint protection when tables are added, and scheduled output or merge nodes. If the task does not support any of these capabilities, downstream results may be incomplete.
This topic applies only to full and incremental synchronization tasks for full databases. For real-time incremental tasks, see Real-time full-database synchronization: Operations and tuning. For tasks that synchronize data only in scheduled batches, see O&M and tuning for offline database synchronization.
Task stages
A full and incremental synchronization task runs through the following stages:
| Stage | Description | O&M focus |
| Schema Migration | Migrates the source database and table schemas to the destination | Source metadata permissions, destination table creation permissions, field type mapping, and table mapping rules |
| Full Data Initialization | Writes the historical data of the source to the destination in a single pass | Split key, full concurrency, source connections, source query performance, and destination write capability |
| Real-time Incremental | Continuously consumes source changes that occur after the full stage starts | Checkpoint delay, failover, checkpoints, DDL, and source log retention |
| Incremental Merge | Periodically merges incremental change data into the destination snapshot table or destination partition. Only some solutions include this stage. | Incremental Merge scheduling, whether upstream incremental data has caught up, data timestamp, destination partition, and SQL execution duration |
The source may keep writing data during full data initialization. After the full stage finishes, the task replays incremental data until it catches up with the source. Completion of the full stage does not mean the whole task is complete. You must also verify that real-time incremental has caught up.
Prerequisites
Before you operate or troubleshoot a full and incremental synchronization task, confirm the following items:
A DataWorks workspace and a resource group for data integration.
Source and destination data source connections configured, with network connectivity between the resource group and both endpoints.
The source account has permissions to read database and table metadata and binary logs or WAL.
The destination account has permissions to create tables, modify tables, and write data.
(If the task includes Incremental Merge) A destination that supports scheduled output or merge nodes. For MaxCompute, the destination table schema must support merge SQL execution.
(If you plan to rerun the task) A Hologres or MaxCompute destination.
Find your task and view execution details
Choose Data Integration > Synchronization Task to open the synchronization task list. The task list shows the basic information and current running status of each task. You can filter tasks by task name, task status, creation time, data source type, and destination type.
Click a task name to open the task execution details page. Use this page as the primary monitoring interface during routine operations and issue investigation. The page shows the following information:
A task progress bar that indicates the current stage: schema migration, full data initialization, real-time incremental, or Incremental Merge. Use it to identify which stage the task is in.
The subtask list of each stage. Use it to inspect individual subtasks and pinpoint which subtask failed or runs slowly.
The real-time synchronization status, including checkpoint delay and throughput. Use these metrics to assess whether real-time incremental keeps pace with source changes.
Links to task logs and monitoring metrics for deeper investigation.
Common O&M operations
Perform the following operations from the Operations column of the task list.
Start and stop a task
To start the task, click Submit and Run. After the task starts, first check whether schema migration and full data initialization run normally, and then check whether real-time incremental has caught up.
To stop the task, click Stop in the Operations column. Before you stop a task, confirm its current stage:
If you stop the task during full data initialization, some full subtasks may need to run again after recovery.
If you stop the task during real-time incremental, recovery depends on the source log retention period.
If you stop a task that includes Incremental Merge, confirm whether the destination partition already contains partial output.
Modify configurations and apply updates
Common changes to a full and incremental synchronization task for full databases include adding tables, removing tables, adjusting table mappings, adjusting resources, and modifying advanced parameters. To modify a task, choose More > Modify Configuration in the Operations column. After you make a change, submit the task and apply the update. The change takes effect only after you submit it.
The following table lists the risk points and recommendations for the most common change types. For resource and parameter tuning guidance, see Tuning recommendations.
| Change type | Risk points | Recommendations |
| Add table | The new table requires schema migration and full data initialization, and then joins real-time incremental after initialization completes | Avoid peak business hours. Confirm that the source log retention period is long enough. Check whether both the full stage and the incremental stage complete for the new table. |
| Remove table | May affect existing table mappings, resource files, and destination data | Confirm that downstream systems no longer depend on the table before removal. If the existing task cannot be submitted successfully, create a new task to take over. |
| Modify table mapping | May affect full writes, real-time writes, and Incremental Merge results | Check the mappings of full tables, incremental tables, and the final destination tables together. |
| Adjust resources | Affects source queries, destination writes, and resource group consumption | Tune by stage. Do not apply the same criteria to the full stage and the real-time stage. |
High-risk operation checklist
Before you run a rerun, a data backfill, a large-batch table addition, a table removal, or a major parameter change, confirm the following items:
Which stage the current problem occurs in.
The scope of the operation: the whole database, specific tables, specific data timestamps, or specific destination partitions.
Whether the source log retention period is long enough.
Whether existing destination data can be overwritten or written again.
Whether an Incremental Merge instance is running or about to run, including instances for the same data timestamp or destination partition.
Whether downstream systems are reading or depend on the affected destination tables, partitions, or data timestamps, and whether downstream nodes must be rerun after the operation completes.
Whether the source and the destination can handle the additional read and write load or higher concurrency.
Rerun a task
Consider a rerun when destination data is corrupted, the real-time task has been failing for an extended period, source logs cannot be replayed, or the task must be initialized again. In the Operations column, choose More > Rerun to force a full and incremental re-initialization of all source tables.
The following limits apply to forced rerun:
Only Hologres and MaxCompute destinations support forced rerun. Sharded full and incremental synchronization tasks do not support forced rerun.
A forced rerun synchronizes columns from the source tables to the destination tables. If a destination table is missing columns that exist in the source table, the missing columns are added.
Before you rerun, complete the checks in High-risk operation checklist.
In a task that includes Incremental Merge, a rerun and a merge must not process the same data timestamp or the same destination partition at the same time. Concurrent runs on the same data timestamp may overwrite partition data or table data.
If a conflict is possible, use one of the following methods:
Pause the rerun, wait for the conflicting Incremental Merge instance to complete, and then perform the rerun.
Freeze the upcoming Incremental Merge instance, wait for the rerun to complete, and then unfreeze the instance.
After the rerun completes, verify that the Incremental Merge instance resumes automatic execution. If data is not generated the next day or the Incremental Merge instance does not resume, manually verify and resume the instance.
Backfill full data
When some tables, some partitions, or some data timestamps are missing data at the destination, use full data backfill to repair the data. Only full-database full and incremental synchronization tasks support full data backfill. Click Backfill All Data in the Operations column and configure the parameters.
Before you run a data backfill, complete the checks in High-risk operation checklist, and confirm the following operation-specific item:
Whether real-time incremental has caught up.
When you backfill multiple data timestamps, run the backfill in batches and verify each batch before you proceed to the next. This prevents interference between the backfill and real-time incremental, Incremental Merge, and downstream scheduling.
Troubleshooting methods
Besides the task status, combine metrics and logs for fine-grained troubleshooting. Typical troubleshooting objects include schema migration, full data initialization, real-time incremental, checkpoints, scheduled instances, data timestamps, destination partitions, and merge SQL. You can view the task progress, subtask status, checkpoint delay, throughput, logs, and monitoring metrics on the task execution details page. For more information, see Find your task and view execution details.
Identify the stage that the problem belongs to, and then go to the corresponding section:
| Symptom | Stage | Section |
| Schema migration fails due to permissions, metadata reading, destination table creation, field types, or table mappings | Schema Migration | Schema migration failure |
| Full data initialization is stuck, does not start, or runs slowly | Full Data Initialization | Full data initialization slow or stuck |
| Real-time incremental latency increases, or real-time incremental cannot catch up after the full stage completes | Real-time Incremental | Real-time incremental latency |
| Scheduled output or Incremental Merge is not generated, or data is incomplete | Incremental Merge | Merge not producing or data incomplete |
| Other symptoms, such as unexpected results after adding or removing tables, DDL errors, dirty data, or conflicts after a rerun or backfill | Multiple stages | FAQ |
Schema migration failure
Schema migration failures usually involve permissions, metadata reading, destination table creation, field types, and table mappings. Check the following items in order:
Check whether the source account has permissions to read database and table metadata.
Check whether the task resource group can connect to the source and the destination.
Check whether the destination account has permissions to create tables, modify tables, and write data.
Check whether the destination database, schema, or namespace exists.
Check whether table name mappings conflict, and whether field types and partition fields are compatible.
Full data initialization slow or stuck
Treat slow or stuck full data initialization as a batch subtask issue. Do not troubleshoot it with real-time latency methods. First confirm which stage is stuck, and then check whether the full subtasks have started.
| Check item | Judgment method | Recommendations |
| Source query performance | The source database has high CPU usage, high I/O, slow SQL queries, lock waits, or many connections | Optimize source queries, avoid peak business hours, and reduce concurrency if necessary. |
| Split key | A few slices take significantly longer than others | Choose a more uniform split key. Do not increase concurrency when no suitable split key exists. |
| Source connections or quota | Logs indicate an insufficient quota or too few connections | Increase source connections or the data source quota as appropriate, and monitor the load of the source database. |
| Full concurrency | Both the source and the destination have spare capacity, but throughput is low | Increase concurrency gradually. Do not raise it to a high value at once. |
| Destination write | The destination is rate-limited, writes slowly, or has too many partitions | Address destination resources, rate limiting, and partition design first. |
| Resource specifications | CPU, memory, or network becomes a bottleneck | Increase resource specifications or compute units, and monitor garbage collection and the failure rate. |
Real-time incremental latency
When you troubleshoot the real-time incremental stage, check the checkpoint delay, read and write throughput, and window wait time first. Then review logs, failover events, checkpoints, JVM metrics, garbage collection, backpressure, and DDL events.
Common causes include:
Large source transactions, or rapid growth of binary logs or WAL.
Partition or shard skew at the source.
Destination write rate limiting, insufficient connections, or slow batch commits.
Checkpoints that take too long or fail frequently.
DDL event handling failures.
Task out-of-memory errors or frequent failover.
If real-time incremental cannot catch up for a long time after full data initialization completes, this usually means that too much incremental data accumulated during the full stage, or that the destination write capability is insufficient. Check both the rate at which the source generates incremental data and the write throughput of the destination.
Check whether the delay keeps decreasing. If it does not, check the read-end checkpoint, the write-end waits, and destination rate limiting separately.
Merge not producing or data incomplete
For a task that includes an Incremental Merge node, do not check only whether the real-time task is running. A real-time task that runs normally does not guarantee that the destination snapshot table or destination partition is produced. Check the following items in order:
Check whether the Incremental Merge scheduled instance is generated, whether it runs, and whether it succeeds.
Check whether the upstream dependencies of the Incremental Merge instance are complete.
Confirm that real-time incremental has caught up to the time range that the Incremental Merge instance requires.
Check the data timestamp, the destination partition, and the SQL execution duration.
If a rerun or data backfill ran recently, confirm whether it processed the same partition.
Check the scheduled instance, the destination partition output, and downstream consumption together.
If source event times or binary logs are out of order, an Incremental Merge instance that is triggered by scheduled time may run before some delayed incremental events arrive. Leave a safety buffer for Incremental Merge scheduling, for example, a delay of 30 minutes, to cover the maximum out-of-order or delay window of the source.
Tuning recommendations
The full stage and the real-time stage have different tuning objectives. The full stage aims for efficient initialization of historical data. The real-time stage aims for continuous stability and low latency. The Incremental Merge stage focuses on the timeliness of partition output and on data integrity.
| Tuning item | Applicable stage | Recommendations |
| Full concurrency | Full Data Initialization | Increase concurrency gradually based on the connection capacity of the source and the write capability of the destination. |
| Maximum number of source connections | Full Data Initialization | Too few connections limit the read speed, and too many connections may affect the stability of the source database. Increase connections gradually and monitor the connection pool utilization of the source database. |
| Batch resource specifications or compute units | Full Data Initialization | Increase the resource specifications or compute units when CPU, memory, or network becomes a bottleneck. The effect is limited when the source or the destination is rate-limited. |
| Real-time resource specifications or compute units | Real-time Incremental | If the checkpoint delay exceeds 5 minutes, consider increasing the compute units by 1 or 2. Monitor failover frequency, checkpoint state, and garbage collection before you make further adjustments. |
| Checkpoint or flush interval | Real-time Incremental | A slightly longer interval reduces frequent commits but increases the delay before new data becomes visible in the destination. If the checkpoint delay is high, verify that the checkpoint interval is not set too low. |
| Incremental Merge scheduling time | Incremental Merge | Leave a buffer for source incremental delay and out-of-order events to avoid merging too early. A 30-minute buffer covers most common delay scenarios. |
Alerts and metrics
Monitor at least the task status, business delay, and failover. Based on the channel capability and the task configuration, also add alerts for resource utilization, write exceptions, DDL notifications, message backlog, and dirty data. For a task that is configured with scheduled output or merge nodes, also monitor scheduled instance failures and destination partition output.
The following alert metrics are available:
| Metric | Metric ID | Description |
| Business Delay | DELAY | The time difference between a source data change and destination write completion |
| Dirty Data | DIRTY_RECORD_COUNT | The number of records that failed validation or write due to data quality issues |
| Failover | FAILOVER_COUNT | The number of failover events triggered by task errors or resource failures |
| Message Backlog | OFFSET_DELAY | The volume of unprocessed messages or log entries waiting to be consumed |
| Task Status | STATUS | The current running status of the synchronization task |
| Unsupported DDL | UNSUPPORTED_DDL | DDL operations that the task cannot handle automatically |
| DDL Notification | DDL_REPORT | Notifications about DDL events that the task processed or skipped |
| Task Resource Utilization | RESOURCE_UTILIZATION | The CPU, memory, and network utilization of task resources |
Recommended metrics by stage
Focus on the following metrics for each stage:
| Stage | Focus metrics |
| Full Data Initialization | Remaining tables, processed tables, remaining splits, processed splits, and full write count |
| Real-time Incremental | Checkpoint delay, read and write throughput, DML and DDL counts, checkpoint duration, and failover count |
| Incremental Merge | Incremental Merge instance status, upstream completion time, data timestamp, destination partition, and SQL execution duration |
| Resources | CPU, memory, garbage collection, network, and resource group utilization |
Responding to alerts
When an alert is triggered, respond as follows:
| Alert condition | Recommended response |
| Business Delay exceeds 5 minutes | Check the checkpoint state and verify that the source log retention window has not been exceeded. Review the real-time incremental throughput and the destination write capacity. |
| Failover count increases | Check the task logs for the root cause of each failover. Common causes include source connectivity loss, destination write errors, and out-of-memory conditions. |
| Dirty Data count increases | Review the dirty data logs to identify the failing field or constraint. Do not increase the alert threshold before you determine whether the dirty data causes missing data, field anomalies, or failed updates and deletes. |
| Unsupported DDL alert | Review the DDL event type and the task DDL policy. Do not ignore an unsupported DDL operation. Verify its impact on data integrity and apply manual schema changes if necessary. |
| Task Status changes to Failed | Check the failure stage on the task execution details page, and then follow the stage-specific guidance in Troubleshooting methods. |
FAQ
The following table lists quick answers to common issues. For stage-based troubleshooting methods, see Troubleshooting methods.
| Issue | Troubleshooting focus | Recommendations |
| After a table is added, only subsequent incremental data exists and no historical data | Whether the new table matches the selection rules; whether full data initialization is enabled or triggered; whether the source log retention covers the changes during the full stage | When historical data is required, confirm that the new table has completed full data initialization. If initialization is not supported or not triggered, complete the data through a data backfill or a separate full synchronization pipeline. |
| After a table is added, the task prompts for confirmation or cannot continue | Whether the task has run the full stage before; whether new tables exist; whether a checkpoint is generated; whether the task runs in a mode in which the full and real-time stages are not decoupled | Wait until the task runs stably and generates a checkpoint, and then submit the new table. Do not expand the synchronization scope while checkpoint protection is missing. |
| Full data initialization is stuck or does not start | Whether schema migration is complete | See Full data initialization slow or stuck. |
| Full data initialization is slow | — | See Full data initialization slow or stuck. |
| The full stage completes, but real-time incremental cannot catch up | — | See Real-time incremental latency. |
| Data inconsistency after a table is removed or a source table is dropped | Whether the task rules still match the removed table; how the DDL policy handles DROP TABLE; whether the destination and downstream systems still depend on the table | Confirm that the business no longer depends on the table before you remove it. Dropping a source table does not automatically clean up the destination. Handle existing destination data separately based on business requirements. |
| Batch refresh of table mappings times out | The number of tables, the number of fields, the regular expression matching scope, and the destination metadata read duration | Narrow the refresh scope and process table mappings in batches. After a large batch refresh, sample the results to confirm added tables, removed tables, and field mappings. |
| Scheduled output or Incremental Merge is not generated, or data is incomplete | — | See Merge not producing or data incomplete. |
| Duplicate data, overwrites, or partition conflicts after a data backfill or rerun | The write mode, existing destination data, and whether a scheduled output or Incremental Merge instance runs for the same partition | Pause or stagger the conflicting instances before a rerun or data backfill. Define the overwrite, append, and cleanup strategies, and then perform the recovery. |
| Task failure or higher latency after a DDL operation | The current DDL type, destination support, the task DDL policy, and destination permissions and field mapping | Check the DDL events and the destination schema change results. Do not ignore an unsupported DDL operation. First determine its impact on data integrity. |
| UPDATE or DELETE results are not as expected | The source primary key, the destination primary key or unique key, primary key mapping, the write mode, and dirty data logs | Compare sample records, source events, task mappings, and destination results first. When the primary key cannot locate a row, updates and deletes may not apply to the correct destination row. |
| Slow writes to MaxCompute or another destination | Partition field granularity, the number of partitions involved in a single checkpoint, Tunnel or commit duration, and destination rate limiting | Reduce the write dispersion caused by partitioning on high-cardinality fields first, and then adjust resources, batch commits, and the destination quota. |
| Increased dirty data alerts | Field type, length, encoding, primary key, non-null constraints, and destination write limits | Do not only raise the threshold. First determine whether the dirty data causes missing data, field anomalies, or failed updates and deletes. |
| Insufficient source log retention | The task stop duration, the binary log or WAL retention policy of the source, and the handoff time between the full and incremental stages | When incremental data cannot be replayed, initialize the task again or backfill data by business scope. Confirm the source log coverage window before the recovery. |