Data Integration is a stable, efficient, and elastically scalable data synchronization platform that reliably moves and synchronizes data at high speed between heterogeneous data sources across complex network environments.
Process
Access Data Integration from a desktop computer using Google Chrome version 69 or later.
The general development workflow for Data Integration is as follows:
Configure a data source, prepare a resource group, and establish network connectivity between the data source and the resource group.
Select a batch or real-time sync type based on your use case, and follow the UI guide to complete the resource and task configuration.
Use data preview and trial runs to debug the task. After successful debugging, submit and deploy the task.
This begins the continuous operations phase, where you monitor sync status, set up alerts, and optimize resources, forming a complete management cycle.
To use AI-powered conversations instead of traditional form-based configuration for creating Data Integration tasks and performing intelligent O&M throughout the entire workflow, DataWorks also provides Data Integration AI Native capabilities (DI Agent), which support natural language conversation-based task creation, intelligent diagnostics, and periodic inspections. For more information, see Data Integration DI Agent.
Synchronization methods
DataWorks Data Integration provides synchronization methods that combine three dimensions: latency, scope, and data policy. For detailed explanations and recommendations, see Supported data sources and synchronization solutions.
Latency: Batch or real-time. Batch synchronization uses periodic scheduling to migrate data on an hourly or daily basis. Real-time synchronization captures source data changes through change data capture (CDC), achieving second-level latency.
Scope: Single table, full database, or sharding. Supports fine-grained transfer of a single table, as well as batch migration and merging of an entire database or sharded databases.
Data policy: Full, incremental, or full and incremental. Full migration moves all historical data. Incremental synchronization processes only new or changed data. The full and incremental mode combines both, offering batch, real-time, and near-real-time implementations based on data source characteristics and latency requirements.
Method | Description |
Batch | Uses a periodic batch scheduling mechanism with hourly or daily tasks to perform full or incremental migration of source data to the destination. |
Real-time | Uses a streaming processing engine to capture source data changes in real time (CDC logs), achieving data synchronization with second-level latency. |
Single table | Transfers data for a single table, with support for fine-grained field mapping, transformation rules, and control configurations. |
Full database | Migrates table schemas and data from multiple tables within a source database instance to the destination in a single operation, with support for automatic table creation. A single task can synchronize multiple tables, reducing the number of tasks and resource consumption. |
Sharding | Writes data from multiple source tables that share the same schema into a single destination table, automatically identifies sharding routing rules, and merges the data. |
Full | Migrates all historical data from the source table at once, typically used for data warehouse initialization or data archiving. |
Incremental | Synchronizes only new or changed data from the source (such as |
Full and incremental | Performs a one-time full synchronization of historical data, and then automatically transitions to incremental data writes. Data Integration supports full and incremental synchronization for various scenarios. Select the appropriate option based on the data source characteristics and latency requirements of your source and destination.
|
Basic concepts
Concept | Description |
Data synchronization | Data synchronization refers to reading data from a source, performing extraction and filtering, and writing the data to a destination. Data Integration focuses on transferring data that can be abstracted into logical two-dimensional table structures. It does not provide data stream consumption or ETL transformation capabilities. Data Integration supports only the at-least-once delivery guarantee. Exactly-once delivery is not supported. This means that duplicate data may occur after synchronization. You can rely only on primary keys and destination capabilities to ensure data deduplication. |
Field mapping | Field mapping defines the read/write correspondence between source and destination data in a sync task. When you configure field mapping, carefully check the type compatibility between source and destination fields to avoid conversion errors that may produce dirty data or cause task failures. Common risks include:
|
Concurrency | Concurrency is the maximum number of threads that can read from or write to the data store in parallel during a data sync task. |
Throttling | Throttling is the transfer speed limit that a Data Integration sync task can reach. |
Dirty data | Dirty data refers to data that is invalid, incorrectly formatted, or encounters synchronization exceptions. When a single record fails to be written to the destination, it is classified as dirty data (for example, a source If a task fails due to dirty data, data that has been successfully written is not rolled back. Data Integration uses a batch write mechanism. The ability to roll back a failed batch depends on whether the destination supports transactions. Data Integration itself does not provide transaction support. |
Data source | A data source is a standardized configuration unit in DataWorks for connecting to external systems. It provides unified read/write endpoint definitions for Data Integration tasks through preset connection templates for various heterogeneous data sources such as MaxCompute, MySQL, and OSS. |
Data consistency | Data Integration supports only the at-least-once delivery guarantee. Exactly-once delivery is not supported. This means that duplicate data may occur after synchronization. You can rely only on primary keys and destination capabilities to ensure data deduplication. |
Features and core value
The capabilities of DataWorks Data Integration are reflected in its extensive connectivity, flexible synchronization methods, high performance, convenient development and O&M, and comprehensive security management.
Extensive data ecosystem connectivity
Break down data silos and enable data aggregation and migration.
Rich data source support: Covers a wide range of data source types, including relational databases, big data storage systems, NoSQL databases, message queues, file storage services, and SaaS applications.
Complex network compatibility: By configuring Network connectivity configuration, you can use the Internet, VPCs, Express Connect, or Cloud Enterprise Network (CEN) to enable data transfer across hybrid cloud and multi-cloud architectures.
Flexible and versatile synchronization methods
Meet synchronization requirements ranging from batch to real-time, from single-table to full-database, and from full to incremental.
Batch synchronization: Supports various batch sync scenarios including single-table, full-database, and sharding. Provides data filtering, column pruning, and transformation logic for large-scale T+1 periodic ETL loading.
Real-time synchronization: Captures data changes from data sources such as MySQL, Oracle, and Hologres in near-real time and writes them to real-time data warehouses or message queues to support real-time business decisions.
Full and incremental integration: Provides batch full-database, real-time full-database, and full-database full and incremental sync solutions. The first execution performs full data initialization, and subsequent runs automatically switch to incremental sync. This simplifies the initial data loading and subsequent update processes, delivering full migration, incremental capture, and automatic full-to-incremental transition capabilities.
Elastic scalability and performance
Adaptive resource scheduling that provides highly reliable data transfer for core business operations.
Elastic resources: Serverless resource groups support on-demand elastic scaling and pay-as-you-go billing to effectively handle traffic fluctuations.
Performance management: Supports concurrency control, throttling, dirty data handling, and distributed processing to ensure stable synchronization under varying workloads.
Low-code development and intelligent O&M
Reduce the development complexity and O&M costs of data synchronization through visual configuration and streamlined workflows.
Low-code development: Codeless UI provides a visual configuration interface that allows you to configure most sync tasks through simple point-and-click operations without writing code. Code editor supports advanced configuration through JSON scripts to meet complex requirements such as parameterization and dynamic column mapping.
Full-stack O&M: Batch sync tasks can be integrated into DAG workflows, with support for scheduling orchestration, monitoring, and alerting.
Comprehensive security management
Integrates multi-layered security mechanisms to ensure data controllability and compliance throughout the entire data flow lifecycle.
Centralized management: A unified data source management center that supports permission control over data sources and isolation between development and production environments.
Security protection: Complies with RAM access control, and supports role-based authentication and data masking.
Billing
The costs of Data Integration tasks mainly include resource group fees, scheduling fees, and public network traffic fees. Data Integration tasks depend on resource groups, whose costs are charged by the resource group. Some batch and full-database batch sync tasks involve scheduled runs and incur scheduling fees. If data sources transfer data over the Internet, public network traffic fees also apply. For billing details, see Core billing scenarios.
Network connectivity
Network connectivity between data sources and resource groups is a prerequisite for successful execution of Data Integration tasks. You must ensure network connectivity between them; otherwise, tasks will inevitably fail.
Data Integration supports data synchronization between heterogeneous data sources across complex network environments, including the following scenarios:
Data synchronization across Alibaba Cloud accounts or regions.
Hybrid cloud and on-premises IDC connectivity.
Multiple network channel configurations such as Internet, VPC, and CEN.
For detailed network configuration solutions, see Overview.
References
After you configure data sources, you can create sync tasks in Data Integration or Data Development to transfer and migrate data. For more information, see: