Offline synchronization for an entire database involves multiple stages, such as schema migration, full synchronization, and offline incremental synchronization. The large number of tables and high density of scheduled instances complicates O&M. This topic describes core O&M operations, such as starting and stopping tasks, modifying configurations, and rerunning tasks to backfill data. It also provides troubleshooting methods and tuning recommendations to help you efficiently manage these tasks. For information about how to configure these tasks, see Configure a task for offline synchronization for an entire database.
Usage notes
Before you troubleshoot issues with offline synchronization for an entire database, assess the task type, channel capabilities, and troubleshooting focus to confirm that this topic applies to your scenario.
Criteria | Applicable scenarios | Inapplicable scenarios/Key focus |
Task type | Offline batch synchronization for an entire database or multiple tables: one-time full synchronization, periodic full synchronization, or periodic incremental synchronization that uses field values to define incremental conditions. | Real-time CDC or message consumption tasks. Offline troubleshooting methods do not apply. |
Typical channels | Source: Offline read from databases, data warehouses, or files. Destination: MaxCompute, Hologres, Hive, DLF, Elasticsearch, StarRocks, and more. | Do not treat a task as an offline incremental synchronization task if a stable incremental field is not available. |
Troubleshooting focus | Scheduled instance, business date, WHERE conditions, splitPk, source partitions, field mappings, destination partitions, write mode, and idempotence for reruns. | Does not involve real-time log offsets or failover. |
Offline synchronization for an entire database relies on offline read and write capabilities. It does not depend on Binlog, WAL, or Oplog, and does not provide real-time log offset recovery semantics.
Task stages
Offline synchronization for an entire database typically includes the following stages. Each stage has different O&M priorities.
Stage | Description | O&M focus |
Schema migration | Read table schemas from the source and create or update tables at the destination. | Metadata permissions, table creation permissions at the destination, column type mappings, and table name mappings. |
One-time full synchronization | Batch write historical data from the source to the destination. | Split key, offline concurrency, source connection count, destination write capability, and resource specifications. |
Periodic offline incremental synchronization | Synchronize new or changed data based on the schedule. | Schedule time, upstream dependencies, business date parameters, destination partitions, and output names. |
Offline incremental synchronization relies on a field that can be used to identify incremental data, such as an update time field or an auto-increment field. If no stable incremental field is available in the source table, you typically must choose periodic full synchronization or modify the source table before configuring offline incremental synchronization.
Offline tasks for an entire database typically generate multiple offline subtasks. The more tables involved, the more noticeable the impact on scheduled instances, resource consumption, and write pressure at the destination.
O&M operations
Start and stop
After you submit a one-time full synchronization task, focus on whether all full subtasks are running properly. For periodic offline tasks, also check whether scheduled instances are generated as expected, whether upstream dependencies are met, and whether the business date is correct.
Before you stop a task, check whether data has already been written to some tables or partitions at the destination. If the task uses an overwrite or partition-cleanup write strategy, pay attention to idempotence and concurrent overwrite issues when you rerun the task.
For general task scheduling and management operations such as pausing, resuming, and backfilling data, see O&M for scheduled instances.
Modify configurations
Common modifications for offline tasks for an entire database include adding tables, removing tables, adjusting table mappings, adjusting partition rules, adjusting schedule settings, and adjusting concurrency and resources.
Change type | Risk | Recommendation |
Add tables | New tables require schema migration and scheduled instance generation. | After you submit the change, verify that the new tables have corresponding offline tasks or instances generated. |
Remove tables | May affect existing scheduling nodes and downstream dependencies. | Before you remove tables, confirm that downstream tasks no longer depend on the corresponding outputs. |
Modify partition rules | May cause data to be written to incorrect partitions or overwrite wrong partitions. | Before you submit the change, verify the business date parameters and destination partition expressions. |
Modify schedules | May affect upstream dependencies and downstream output timing. | After the modification, verify that scheduled instances are generated at the new times. |
Adjust concurrency and resources | May increase source connection load, destination write pressure, and resource group consumption. | Adjust incrementally and monitor the load on the source, destination, and resource group. |
Rerun and backfill data
Offline tasks for an entire database are often rerun or used to backfill data to repair missing partitions at the destination. Before you proceed, confirm the following:
The tables and business date range that need to be repaired.
The write mode at the destination (append, overwrite, or truncate-then-write).
Whether other instances are currently writing to the same destination table or partition.
Whether downstream tasks have already consumed incorrect data.
Whether downstream tasks need to be rerun after the current rerun.
If the write mode at the destination involves partition cleanup, do not run multiple tasks on the same partition concurrently. If necessary, pause conflicting instances or stagger execution times.
For more information about backfilling data, see O&M for backfill instances.
Troubleshooting
Table mapping refresh failure or tables not selectable
Common causes include insufficient source permissions, resource group network connectivity issues, excessive source metadata volume, overly broad database/table filter scope, too many selected tables, and insufficient destination permissions. Troubleshoot in the following order:
Use the resource group specified in the task configuration to re-test connectivity to the source and destination.
Verify that the source account has permissions to read database, table, and column metadata.
If the database/table filter scope is broad or too many tables are selected, narrow the scope first to verify.
Check whether the destination account has table creation or write permissions.
Check whether metadata API timeouts occur when the source has a large number of tables.
Schema migration failure
When schema migration fails, first check destination table creation permissions, whether the destination database or schema exists, whether column types are compatible, and whether table name mappings conflict. For partitioned tables, also verify that partition columns, partition levels, and partition expressions comply with destination requirements.
Slow full synchronization task
Item | How to identify | Recommendation |
Slow source queries | High CPU, I/O, slow SQL, or lock waits on the source database. | Optimize source queries, avoid peak business hours, and reduce concurrency if necessary. |
Uneven data splitting | A few subtasks take significantly longer than others. | Choose a more suitable split key, or split large tables into separate tasks. |
Low concurrency | Low throughput despite low load on both the source and destination. | Gradually increase offline concurrency and resource specifications. |
Insufficient connections | Logs indicate insufficient connections or quota. | Increase the maximum number of connections on the source or the data source quota, and monitor source database stability. |
Slow destination writes | Throttling at the destination, too many partitions, or slow batch commits. | Address destination resource issues, throttling, and partition design first. |
Insufficient resources | CPU, memory, or network approaching upper limits. | Increase resource specifications or CUs, and monitor GC and failure rates. |
When tuning offline synchronization for an entire database, do not significantly increase concurrency at the outset. Increasing concurrency raises source connection count, destination write pressure, and resource group consumption. If the bottleneck is at the source or destination, increasing concurrency typically only amplifies the problem.
For general solutions to slow data synchronization, see Troubleshoot slow data synchronization.
Scheduled instance not producing output
When a periodic offline instance does not produce output or produces unexpected output, check the following in order:
Whether periodic scheduling is enabled and the schedule time is as expected.
Whether upstream dependencies have completed.
Whether business date parameters are correct.
Whether output names and destination partition expressions are correct.
Whether write failures or permission issues exist at the destination.
Whether downstream tasks depend on incorrect outputs or partitions.
Offline incremental instances rely on both the incremental conditions and scheduling parameters to determine the data range. When troubleshooting scheduled instances, check whether the incremental condition field exists, whether the field type is comparable, and whether scheduling parameters are correctly passed to the destination partition or WHERE filter conditions.
Tuning recommendations
Tuning item | Recommendation |
Total resource CUs | Increase total resources when the number of tables or parallel subtasks is large. Increasing CUs alone has limited effect when throttling occurs at the source or destination. |
Per-task CUs | Increase per-task resources for wide tables, large fields, or tables with large data volumes. |
Offline concurrency | Gradually increase based on the source connection capacity and destination write capability. |
Maximum source connections | Adjust in coordination with offline concurrency. Setting this value too high may affect source database stability. |
Throttling configuration | When throttling is enabled, throughput is actively limited. Before tuning, determine whether you need to disable or increase the throttling limit. |
Partition design | Avoid overly granular partitions that lead to small files, excessive partitions, or high write overhead. |
For offline tasks involving a large number of tables, monitor large tables, wide tables, and small tables separately. A few large tables may determine the overall completion time, and average throughput does not reflect the actual bottleneck.
For information about the relationship between concurrency and throttling, see Relationship between concurrency and throttling for batch synchronization.
Alerts and monitoring
For offline tasks for an entire database, focus on scheduling and offline instance alerts. Alerts typically need to be configured after you locate the generated scheduled subtasks or offline instances in Operation Center. Do not rely solely on the overall task status of the solution.
We recommend that you configure the following monitoring rules:
Instance failure: Trigger an alert immediately when a task fails.
Instance timeout: Trigger an alert when a task runs longer than expected.
Upstream dependency not completed: The upstream task that the current task depends on did not finish on time.
Scheduled instance not generated: No instance is generated after the scheduled time arrives.
Destination partition not produced: The expected destination partition is not produced on time.
Source or destination connection failure: The data source connection is abnormal.
If offline tasks for an entire database are time-sensitive for downstream consumption, configure instance timeout and output validation rules. Configuring only failure alerts cannot detect cases where a task runs for an extended period without producing output on time.
For information about how to configure alert rules, see Configure a task for offline synchronization for an entire database > Step 6: Configure alert rules.
High-risk operation checklist
Before you perform large-scale reruns, backfill data, remove tables, modify partition rules, or significantly increase concurrency, confirm the following: which tables and business dates are affected, whether the write mode at the destination will overwrite or clean up partitions, whether other instances are writing to the same destination table or partition, whether downstream tasks have already consumed the affected data, whether the source and destination can handle higher concurrency, and whether downstream tasks need to be rerun.
FAQ
The following questions are primarily derived from historical troubleshooting of periodic offline tasks for entire databases. When troubleshooting, first identify the task type, source, and destination, and then cross-validate by checking the page configuration, scheduled instances, runtime logs, and destination results.
If I remove a table and then add it back, will the old incremental instances stop?
How to check: After you remove the table and apply the update, subsequent periodic scheduling nodes are undeployed. However, do not assume that instances already generated or running will stop automatically.
Recommended action: If you need to stop writes immediately, check the status of the corresponding scheduled instances in Operation Center. Manually stop or pause the relevant instances if necessary, and then add the table back, rerun, or backfill data.
Target schema or database name does not match expectations after adding a table
How to check: Verify whether the schema, database name, or table name mapping rules have changed, and confirm whether the new table used the default mapping.
Recommended action: Check and configure the schema, database name, or table name mapping rules to prevent new tables from being written to an unintended destination.
Table mapping refresh is slow, times out, or tables are not selectable
How to check: Check resource group connectivity, source metadata query duration, the number of selected tables, and source account permissions.
Recommended action: Narrow the database/table scope first to verify. Check data source connection parameters and source metadata query performance. If an object is not selectable, refer to the supported objects on the page.
Periodic incremental condition produces results inconsistent with the configuration
How to check: Verify whether the incremental conditions have been modified, whether scheduling parameters are calculated based on the business date or runtime, and whether the destination partition and filter conditions use the same set of parameters.
Recommended action: Check the incremental conditions, scheduling parameters, and destination partition expressions together. When backfilling data, confirm the business date range.
Scheduled instance keeps waiting for upstream
How to check: Check whether additional upstream node IDs or upstream node name rules are configured in the advanced parameters.
Recommended action: Clear unnecessary upstream dependency configurations, redeploy the task, and then monitor the instance dependency relationships.
Destination partition not produced or written to an incorrect partition
How to check: Verify that the partition column actually exists, the partition expression is correct, and the partition level, case sensitivity, and spacing comply with the destination requirements.
Recommended action: Cross-check the destination table metadata and periodic parameters. If necessary, verify partition writes with a single table or a small number of tables first.
Downstream output name conflicts or contains special characters
How to check: Check whether the scheduling node output name rules produce duplicate outputs or contain special characters not supported by the target scheduling system.
Recommended action: Adjust the output name rules and avoid special characters such as $. After you save and deploy the changes, verify downstream dependencies.
Views or special objects are not selectable
How to check: Verify whether the source object type is supported by the current offline synchronization channel for the entire database.
Recommended action: Use the selectable objects on the page as the reference. If an object is not supported, use single-table batch synchronization instead, or materialize the view results into a physical table before synchronization.
splitPk does not take effect or causes uneven splitting
How to check: Verify whether the split key column type is supported by the data source and the synchronization channel, whether the data distribution is even, and whether a large number of null values exist.
Recommended action: Switch to a supported split key with a more even distribution. Large tables can be split into separate tasks for individual tuning.
For information about split key limitations, see Configure a task for offline synchronization for an entire database > Quotas and limits.
References
Configure a task for offline synchronization for an entire database — The complete configuration process for offline synchronization tasks for an entire database.
Troubleshoot slow data synchronization — General solutions for slow data synchronization scenarios.
Relationship between concurrency and throttling for batch synchronization — Configuration details for concurrency and throttling in Channel Control.