DataWorks Data Integration supports three methods for synchronizing data from external sources into MaxCompute: batch synchronization, real-time synchronization, and integrated synchronization. Choose based on your latency requirements, data volume, and pipeline complexity.
Choose a synchronization method
| Method | Best for |
|---|---|
| Batch synchronization | Periodic full or incremental loads |
| Real-time synchronization | Continuous incremental sync with low-latency delivery |
| Integrated synchronization | Complex end-to-end pipelines combining batch, real-time, and processing tasks |
How to choose:
-
Start with batch synchronization for periodic loads from one or more tables.
-
Use real-time synchronization when you need continuous data consistency between a source database and MaxCompute.
-
Choose integrated synchronization when your pipeline would otherwise require manually orchestrating multiple batch, real-time, and processing tasks.
Batch synchronization
Batch synchronization uses readers and writers to move structured and semi-structured data from a source into MaxCompute. You can sync from a single table or from tables across sharded databases into a single MaxCompute table.
Prerequisites
-
A MaxCompute data source added on the Data Sources page in the DataWorks console. See Add a MaxCompute data source.
-
A network connection established between a resource group for Data Integration and your data source. See Establish a network connection between a resource group and a data source.
Configure a batch synchronization node
Choose one of the following configuration methods:
Real-time synchronization
Real-time synchronization captures incremental changes from source databases and delivers them to MaxCompute continuously, keeping your MaxCompute tables in sync with the source. Conversion plug-ins cleanse the source data before multiple writers deliver it to the destination simultaneously.
A single real-time synchronization task supports the following topologies:
-
A single table to a single MaxCompute table
-
Tables in sharded databases to a single MaxCompute table
-
Multiple tables in a database to multiple MaxCompute tables
Prerequisites
-
A MaxCompute data source added on the Data Sources page in the DataWorks console. See Add a MaxCompute data source.
-
An exclusive resource group for Data Integration purchased with appropriate specifications. See Create and use an exclusive resource group for Data Integration.
-
A network connection established between a resource group for Data Integration and your data source. See Establish a network connection between a resource group and a data source.
-
A MaxCompute environment configured for your data source. See Prepare a MaxCompute environment.
No optimal value is provided for the concurrency of synchronization tasks in which exclusive resource groups of MaxCompute run. Set concurrency based on your data volume and target synchronization time. To minimize synchronization time, purchase resource specifications that support the maximum number of concurrent threads. For per-task resource requirements, see Performance metrics.
Configure a real-time synchronization node
Choose one of the following configuration methods:
Integrated synchronization
Integrated synchronization provides preconfigured solutions for complex pipelines that combine batch synchronization, real-time synchronization, and data processing tasks. Instead of building and orchestrating these tasks separately, you can configure the entire pipeline with a few clicks.
Prerequisites
-
A MaxCompute data source added on the Data Sources page in the DataWorks console. See Add a MaxCompute data source.
-
An exclusive resource group for Data Integration purchased with appropriate specifications. See Create and use an exclusive resource group for Data Integration.
-
A network connection established between a resource group for Data Integration and your data source. See Establish a network connection between a resource group and a data source.
-
A MaxCompute environment configured for your data source. See Prepare a MaxCompute environment.
No optimal value is provided for the concurrency of synchronization tasks in which exclusive resource groups of MaxCompute run. Set concurrency based on your data volume and target synchronization time. To minimize synchronization time, purchase resource specifications that support the maximum number of concurrent threads. For per-task resource requirements, see Performance metrics.
Configure an integrated synchronization solution
Choose one of the following solutions: