Real-time synchronization of incremental data among various data sources is supported. The real-time synchronization feature allows you to synchronize incremental data from a single table or all tables in a database to a destination. You can configure a real-time synchronization node based on the data source types that support real-time synchronization. This topic describes the data source types that support real-time synchronization of incremental data from a single table or all tables in a database.

Background information

To run a real-time synchronization node to synchronize incremental data from a source, you must perform the following steps:
  1. Configure data sources. Before you configure a real-time synchronization node, you must configure the data sources that you need to use. This way, you can select the data sources when you configure a real-time synchronization node. For information about the data source types that support real-time synchronization, see Supported data source types and read and write operations.
    Note For information about the items that you need to understand before you configure a data source, see Overview.
  2. Purchase an exclusive resource group for Data Integration whose specifications meet your business requirements. For more information, see Create and use an exclusive resource group for Data Integration.
  3. Establish network connections between the exclusive resource group for Data Integration and the data sources. For more information, see Establish a network connection between a resource group and a data source.
  4. Configure the settings that are required to prepare databases. You must create an account that can be used to access a database in the source and an account that can be used to access a database in the destination. You must also grant the accounts the permissions required to perform specific operations on the databases based on your configurations for data synchronization. For more information, see Overview.
  5. Configure a real-time synchronization node that is used to synchronize incremental data from a single table or all tables in a database in real time. For more information, see Create a real-time synchronization node to synchronize incremental data from a single table or Configure a real-time synchronization node to synchronize all incremental data from a database.
Note In a DataWorks workspace in standard mode, real-time synchronization does not support the development environment. Therefore, when you configure a real-time synchronization node in such a workspace, you must select data sources that are added to the production environment.

Data source types that support real-time synchronization of incremental data from a single table

A real-time synchronization node that is used to synchronize incremental data from a single table in real time supports input, output, and conversion plug-ins. For information about how to configure such a node, see Create a real-time synchronization node to synchronize incremental data from a single table.
Note A real-time synchronization node that is used to synchronize incremental data from a single table in real time supports the following types of conversion plug-ins: data filter, string replacement, and data masking. You can use the plug-ins to process data in the source based on your business requirements and synchronize the processed data to the destination. For more information, see the following topics:
Data source typePlug-in for reading incremental data from a single table in real timePlug-in for writing incremental data that is read from a single table in real time
AnalyticDB for MySQL 3.0-Configure AnalyticDB for MySQL Writer
DataHubConfigure DataHub ReaderConfigure DataHub Writer

Elasticsearch

-Configure Elasticsearch Writer

Hologres

-Configure Hologres Writer
KafkaConfigure Kafka ReaderConfigure Kafka Writer
LogHub (SLS)Configure LogHub Reader-
MaxCompute-Configure MaxCompute Writer
MySQLConfigure MySQL Reader-
PolarDBConfigure PolarDB Reader-

Data source types that support real-time synchronization of incremental data from all tables in a database

Note To prevent the data synchronization node from failing during data synchronization because of insufficient permissions on the related databases, we recommend that you prepare the database environments in advance.
Destination typeSource database environment
Synchronize incremental data from all tables in a database to MaxCompute in real time
Synchronize incremental data from all tables in a database to Hologres in real time
Synchronize incremental data from all tables in a database to AnalyticDB for MySQL 3.0 in real time
Synchronize incremental data from all tables in a database to DataHub in real time
Synchronize incremental data from all tables in a database to Kafka in real time