Offline synchronization for an entire database involves multiple stages, such as schema migration, full synchronization, and offline incremental synchronization. The large number of tables and high density of scheduled instances complicates O&M. This topic describes core O&M operations, such as starting and stopping tasks, modifying configurations, and rerunning tasks to backfill data. It also provides troubleshooting methods and tuning recommendations to help you efficiently manage these tasks. For information about how to configure these tasks, see Configure a task for offline synchronization for an entire database.
Applicability
Before you troubleshoot issues with offline synchronization for an entire database, assess the task type, channel capabilities, and troubleshooting focus to confirm that this topic applies to your scenario.
Criteria | Applicable scenarios | Inapplicable scenarios/Key focus |
Task type | Offline batch synchronization for an entire database or multiple tables: one-time full synchronization, periodic full synchronization, or periodic incremental synchronization that uses field values to define incremental conditions. | Real-time CDC or message consumption tasks. Offline troubleshooting methods do not apply. |
Typical channels | Source: Offline read from databases, data warehouses, or files. Destination: MaxCompute, Hologres, Hive, DLF, Elasticsearch, StarRocks, and more. | Do not treat a task as an offline incremental synchronization task if a stable incremental field is not available. |
Troubleshooting focus | Scheduled instance, business date, WHERE conditions, | Does not involve real-time log offsets or failover. |
Offline synchronization for an entire database relies on offline read and write capabilities. It does not depend on Binlog, WAL, or Oplog, and does not provide real-time log offset recovery semantics.
Task stages
Offline synchronization for an entire database typically includes the following stages. Each stage has different O&M priorities.
Stage | Description | O&M focus |
Schema migration | Reads table schemas from the source and creates or updates tables at the destination. | Metadata permissions, destination table creation permissions, field type mappings, and table name mappings. |
One-time full synchronization | Writes historical data from the source to the destination in batches. | Split key, offline concurrency, source connections, destination write capacity, and resource specifications. |
Periodic offline incremental synchronization | Synchronizes new or changed data based on the scheduling cycle. | Scheduling time, upstream dependencies, business date parameters, destination partitions, and output names. |
Offline incremental synchronization depends on a field that identifies incremental data, such as an update timestamp or an auto-incrementing field. If the source table does not have a stable incremental field, you must typically perform periodic full synchronization or modify the source table before you configure offline incremental synchronization.
An offline synchronization task for an entire database usually generates multiple offline subtasks. As the number of tables grows, the pressure on scheduled instances, resource consumption, and destination writes increases significantly.
O&M operations
Start and stop tasks
After you submit a one-time full synchronization task, monitor the full synchronization subtasks to ensure they are running properly. For periodic offline tasks, you must also check whether scheduled instances are generated as expected, upstream dependencies are met, and the business date is correct.
Before you stop a task, confirm whether data from some tables or partitions has already been written to the destination. If a task that writes to the destination uses an overwrite or partition-clearing strategy, pay attention to idempotence and concurrent overwrite issues when you rerun the task.
For general task scheduling and management operations such as pausing, resuming, and backfilling data, see Offline synchronization task O&M.
Modify configurations
Common modifications for offline synchronization tasks for an entire database include adding or deleting tables, and adjusting table mappings, partition rules, scheduling configurations, concurrency, and resources.
Change type | Potential risks | Recommendation |
Add tables | Adding a table requires schema migration and scheduled instance generation. | After you submit the change, check that a corresponding offline task or instance is generated for the new table. |
Delete tables | May affect existing scheduling nodes and downstream dependencies. | Before deleting, confirm that downstream consumers no longer depend on the corresponding output. |
Modify partition rules | May cause data to be written to a new partition or overwrite the wrong partition. | Before submission, confirm the business date parameters and destination partition expressions. |
Modify scheduling | May affect upstream dependencies and downstream output times. | After modification, check whether periodic instances are generated at the new time. |
Adjust concurrency and resources | May increase the pressure on source connections, destination writes, and the resource group. | Adjust gradually and monitor the load on the source, destination, and resource group. |
Rerun tasks and backfill data
Rerunning tasks or backfilling data can fix missing destination partitions. Before you perform these operations, confirm the following:
Which tables and business date ranges need to be fixed.
The destination write mode (append, overwrite, or truncate and write).
Whether other instances are currently writing to the same destination table or partition.
Whether downstream tasks have already consumed incorrect data.
Whether to rerun downstream tasks after the current task is rerun.
If the destination write mode clears partitions, do not run multiple tasks that write to the same partition simultaneously. If necessary, pause conflicting instances first or stagger their execution times.
For more information about how to use the backfill data feature, see Backfill instance O&M.
Troubleshooting
Table mapping refresh failure or unselectable tables
Common causes include insufficient source permissions, network issues for the resource group, an excessive amount of source metadata, an overly large table filter scope, too many selected tables, or insufficient destination permissions. Troubleshoot in the following order:
Retest source and destination connectivity by using the resource group from the task configuration.
Confirm that the source account has permissions to read database, table, and field metadata.
If the table filter scope is large or many tables are selected, first narrow the scope to verify connectivity.
Check whether the destination account has table creation or write permissions.
Check for metadata interface timeouts if there is an excessive number of source tables.
Schema migration failures
If schema migration fails, first check destination table creation permissions, whether the destination database or schema exists, whether field types are compatible, and whether table name mappings conflict. For partitioned tables, also confirm that the partition fields, partition hierarchy, and partition expressions meet destination constraints.
Full synchronization task runs slowly
Item to check | How to identify | Recommended action |
Slow source query | High CPU, I/O, slow SQL queries, or lock waits on the source database. | Optimize the source query, avoid peak business hours, and reduce concurrency if necessary. |
Uneven splits | A few subtasks take significantly longer to complete. | Choose a more suitable split key, or process large tables separately. |
Low concurrency | Low throughput despite low load on the source and destination. | Gradually increase offline concurrency and resource specifications. |
Insufficient connections | Logs indicate insufficient connections or quota. | Increase the maximum number of source connections or the data source quota, and monitor source database stability. |
Slow destination write | Throttling, too many partitions, or slow batch commits on the destination. | Address destination resource issues, throttling, and partition design first. |
Insufficient resources | CPU, memory, or network usage is approaching its limit. | Increase resource specifications or CUs, and monitor garbage collection (GC) and failure rates. |
When you tune offline synchronization for an entire database, do not significantly increase concurrency initially. An increase in concurrency also increases the number of source connections, write pressure on the destination, and consumption of the resource group. If the bottleneck is at the source or destination, increasing concurrency usually amplifies the issue.
For solutions to common slow data synchronization scenarios, see Accelerate or limit the speed of offline synchronization.
Periodic instances produce no output
If a periodic offline instance produces no output or produces incorrect output, check the following items in order:
Whether periodic scheduling is enabled and the scheduling time is correct.
Whether upstream dependencies are complete.
Whether the business date parameter is correct.
Whether the output name and destination partition expressions are correct.
Whether there are write failures or permission issues at the destination.
Whether downstream tasks depend on the wrong output or partition.
The data range for an offline incremental instance is determined by its incremental condition and scheduling parameters. When you troubleshoot a periodic instance, check whether the Incremental Condition Field exists, whether the Field Type is comparable, and whether the Scheduling Parameters are correctly passed to the destination partition or WHERE filter condition.
Tuning recommendations
Tuning item | Recommendation |
Total CUs | Increase the total resources for a large number of tables or parallel subtasks. If the source or destination is throttled, increasing CUs alone has a limited effect. |
CUs per Task | Increase per-task resources for wide tables, large columns, or a large volume of data in a single table. |
Offline Concurrency | Increase gradually based on the source connection capacity and destination write capacity. |
Max Source Connections | Adjust in coordination with offline concurrency. Setting this too high may affect source database stability. |
Rate Limit Configuration | If rate limiting is enabled, throughput is actively reduced. Before tuning, confirm whether you need to disable or increase the rate limit. |
Partition design | Avoid overly fine-grained partitions, which can lead to small files, too many partitions, or excessive write overhead. |
For offline synchronization tasks for an entire database that involve many tables, group your observations by large tables, wide tables, and smaller tables. A few large tables can determine the overall completion time, and average throughput does not reflect the real bottleneck.
For more information about the relationship between concurrency and rate limiting, see Relationship between concurrency and rate limiting in offline synchronization.
Alerts and monitoring
For offline synchronization tasks for an entire database, focus on alerts for scheduling and offline instances. Alerts usually need to be configured on the generated periodic subtasks or offline instances in the O&M Center. You should not rely solely on the status of the main solution task.
Configure the following alert rules:
Instance Failed: Alerts immediately when a task execution fails.
Instance Timed Out: Alerts when a task runs longer than expected.
Upstream Dependency Incomplete: Alerts when a dependent upstream task does not complete on time.
Periodic Instance Not Generated: Alerts when an instance is not generated after the scheduling cycle begins.
Target Partition Not Generated: Alerts when an expected destination partition is not produced on time.
Source/Destination Connection Failed: Alerts for data source connection exceptions.
If the timeliness of data for downstream tasks is critical, you must configure alerts for instance timeouts and output validation. Configuring only failure alerts does not detect cases where a task runs for a long time without producing timely output.
For specific instructions on how to configure alert rules, see Configure a task for offline synchronization for an entire database > Step 6: Configure Alert Rules.
Checklist for high-risk operations
Before you perform large-scale operations such as rerunning tasks, backfilling data, deleting tables, modifying partition rules, or significantly increasing concurrency, confirm the following: the affected tables and business dates; the destination write mode and its potential to overwrite or clear data; any concurrent writes to the same destination; whether downstream tasks have consumed the data; the source and destination's capacity to handle higher concurrency; and whether downstream tasks also need to be rerun.
FAQ
The following questions are from accumulated troubleshooting cases for periodic tasks that synchronize an entire database. When you troubleshoot, first confirm the task type, source, and destination, and then cross-check the page configuration, scheduled instances, run logs, and destination results.
Stopping old instances after re-adding a table
Troubleshooting: After you delete a table and apply the update, subsequent periodic scheduling nodes are disabled. You should not assume that already generated or running instances will stop automatically.
Mitigation: If you need to stop writes immediately, check the status of the corresponding periodic instance in the O&M Center. If necessary, manually stop or pause the relevant instances before you add, rerun, or backfill data for the table.
Incorrect destination for new tables
Troubleshooting: Check whether the schema, database name, or table name mapping rules have changed. Confirm whether the new table used the default mapping.
Mitigation: Check and configure the schema, database name, or table name mapping rules to prevent new tables from being written to different destination locations.
Table mapping refresh failure or unselectable tables
Troubleshooting: Check resource group connectivity, source metadata query latency, the number of selected tables, and source account permissions.
Mitigation: First, narrow the database and table scope to verify. Check the data source connection parameters and the source metadata query performance. If an object is unselectable, the supported scope displayed on the page prevails.
Inconsistent incremental condition results
Troubleshooting: Confirm whether the incremental condition has been modified, whether Scheduling Parameters are calculated based on the business date or run time, and whether the destination partition and filter condition use the same set of parameters.
Mitigation: Check the incremental condition, Scheduling Parameters, and destination partition expression at the same time. When you backfill data, confirm the business date range.
Periodic instances waiting for upstream dependencies
Troubleshooting: Check whether an extra Upstream Node ID or Upstream Node Name Rule is configured in the Advanced Parameters.
Mitigation: Clear any unnecessary upstream dependency configurations, republish the task, and then observe the instance dependencies.
Missing or abnormal destination partition output
Troubleshooting: Confirm whether the partition column actually exists, the partition expression is correct, and the partition hierarchy, case, and spacing meet the destination's requirements.
Mitigation: Check the destination table metadata and periodic parameters. If necessary, use a single table or a small number of tables to verify partition writes first.
Output name conflicts or special characters
Troubleshooting: Check whether the output name rule for the scheduling node generates duplicate outputs or contains special characters that are not supported by the destination scheduler.
Mitigation: Adjust the output name rule to avoid using special characters such as $. After you save and publish, check the downstream dependencies.
Views or special objects are unselectable
Troubleshooting: Confirm whether the source object type is supported by the current channel for offline synchronization for an entire database.
Mitigation: Only the objects that are selectable on the page are supported. If not supported, use a single-table offline synchronization task, or materialize the view results into a physical table before synchronization.
splitPk is ineffective or causes uneven data distribution
Troubleshooting: Confirm whether the split key field type is supported by the data source and channel, whether the data distribution is even, and whether there are a large number of null values.
Mitigation: Use a supported split key field with a more even distribution. You can process large tables separately for individual tuning.
For more information about the limitations of split keys, see Configure a task for offline synchronization for an entire database > Quotas and limits.
Learn more
Configure a task for offline synchronization for an entire database — The complete process for configuring a task for offline synchronization for an entire database
Accelerate or limit the speed of offline synchronization — Solutions for common slow data synchronization scenarios
Relationship between concurrency and rate limiting in offline synchronization — Description of how to configure concurrency and rate limiting in channel controls