You can use the reader, writer, and conversion plug-ins for various data sources to synchronize data in real time. This topic describes the plug-ins for data sources that support real-time synchronization.

Plug-in type Plug-in name References
Reader MySQL Binlog Reader Configure MySQL Binlog Reader
DataHub Reader Configure DataHub Reader
LogHub Reader Configure LogHub Reader
Kafka Reader Configure Kafka Reader
PolarDB Reader Configure PolarDB reader
PostgreSQL Reader For more information about how to configure a PostgreSQL data source for batch and real-time synchronization, see Configure a source PostgreSQL data source.
Writer MaxCompute Writer Configure MaxCompute Writer
Hologres Writer Configure Hologres Writer
AnalyticDB for MySQL Writer Configure AnalyticDB for MySQL Writer
DataHub Writer Configure DataHub Writer
Kafka Writer Configure Kafka Writer
Elasticsearch Writer Configure Elasticsearch Writer
Conversion Data Filtering Configure Data Filtering
String Replace Configure String Replace
Data Masking Configure data de-identification
Note
  • You cannot run a real-time synchronization node on the node configuration tab. Instead, you must run a real-time synchronization node in the production environment after you save and commit the node.
  • Real-time synchronization nodes can be run only on an exclusive resource group for Data Integration. For more information, see DataWorks exclusive resources.

Limits

If you want to synchronize data in real time from a relational database, such as a MySQL, a PolarDB for MySQL, an Oracle, or a PolarDB-X database, note that the following operations are not supported:
  • Online DDL operations are not supported. This feature allows you to perform data definition language (DDL) operations on tables, such as creating an index, without blocking data manipulation language (DML) operations and SELECT queries that run concurrently with the DDL operations. In this case, errors and data quality issues occur when data is written to the destination in real-time synchronization.
  • If a column is inserted to or deleted from the source between the start time of the real-time synchronization node and the current time, errors and data quality issues occur when data is written to the destination.