Real-time synchronization of incremental data among various data sources is supported. The real-time synchronization feature allows you to synchronize incremental data from a single table or all tables in a database to a destination. You can configure a real-time synchronization node based on the data source types that support real-time synchronization. This topic describes the data source types that support real-time synchronization of incremental data from a single table or all tables in a database.
Background information
To run a real-time synchronization node to synchronize incremental data from a source, you must perform the following steps:- Configure data sources. Before you configure a real-time synchronization node, you must configure the data sources that you need to use. This way, you can select the data sources when you configure a real-time synchronization node. For information about the data source types that support real-time synchronization, see Supported data source types and read and write operations. Note For information about the items that you need to understand before you configure a data source, see Overview.
- Purchase an exclusive resource group for Data Integration whose specifications meet your business requirements. For more information, see Create and use an exclusive resource group for Data Integration.
- Establish network connections between the exclusive resource group for Data Integration and the data sources. For more information, see Establish a network connection between a resource group and a data source.
- Configure the settings that are required to prepare databases. You must create an account that can be used to access a database in the source and an account that can be used to access a database in the destination. You must also grant the accounts the permissions required to perform specific operations on the databases based on your configurations for data synchronization. For more information, see Overview.
- Configure a real-time synchronization node that is used to synchronize incremental data from a single table or all tables in a database in real time. For more information, see Create a real-time synchronization node to synchronize incremental data from a single table or Configure a real-time synchronization node to synchronize all incremental data from a database.
Note In a DataWorks workspace in standard mode, real-time synchronization does not support the development environment. Therefore, when you configure a real-time synchronization node in such a workspace, you must select data sources that are added to the production environment.
Data source types that support real-time synchronization of incremental data from a single table
A real-time synchronization node that is used to synchronize incremental data from a single table in real time supports input, output, and conversion plug-ins. For information about how to configure such a node, see Create a real-time synchronization node to synchronize incremental data from a single table.Note A real-time synchronization node that is used to synchronize incremental data from a single table in real time supports the following types of conversion plug-ins: data filter, string replacement, and data masking. You can use the plug-ins to process data in the source based on your business requirements and synchronize the processed data to the destination. For more information, see the following topics:
| Data source type | Plug-in for reading incremental data from a single table in real time | Plug-in for writing incremental data that is read from a single table in real time |
|---|---|---|
| AnalyticDB for MySQL 3.0 | - | Configure AnalyticDB for MySQL Writer |
| DataHub | Configure DataHub Reader | Configure DataHub Writer |
Elasticsearch | - | Configure Elasticsearch Writer |
Hologres | - | Configure Hologres Writer |
| Kafka | Configure Kafka Reader | Configure Kafka Writer |
| LogHub (SLS) | Configure LogHub Reader | - |
| MaxCompute | - | Configure MaxCompute Writer |
| MySQL | Configure MySQL Reader | - |
| PolarDB | Configure PolarDB Reader | - |
Data source types that support real-time synchronization of incremental data from all tables in a database
Note To prevent the data synchronization node from failing during data synchronization because of insufficient permissions on the related databases, we recommend that you prepare the database environments in advance.
| Destination type | Source database environment |
|---|---|
| Synchronize incremental data from all tables in a database to MaxCompute in real time | |
| Synchronize incremental data from all tables in a database to Hologres in real time |
|
| Synchronize incremental data from all tables in a database to AnalyticDB for MySQL 3.0 in real time | |
| Synchronize incremental data from all tables in a database to DataHub in real time | |
| Synchronize incremental data from all tables in a database to Kafka in real time |