You can use the reader, writer, and conversion plug-ins for various data sources to synchronize data in real time. This topic describes the plug-ins for data sources that support real-time synchronization.
| Plug-in type | Plug-in name | References |
|---|---|---|
| Reader | MySQL Binlog Reader | Configure MySQL Binlog Reader |
| DataHub Reader | Configure DataHub Reader | |
| LogHub Reader | Configure LogHub Reader | |
| Kafka Reader | Configure Kafka Reader | |
| PolarDB Reader | Configure PolarDB reader | |
| PostgreSQL Reader | For more information about how to configure a PostgreSQL data source for batch and real-time synchronization, see Configure a source PostgreSQL data source. | |
| Writer | MaxCompute Writer | Configure MaxCompute Writer |
| Hologres Writer | Configure Hologres Writer | |
| AnalyticDB for MySQL Writer | Configure AnalyticDB for MySQL Writer | |
| DataHub Writer | Configure DataHub Writer | |
| Kafka Writer | Configure Kafka Writer | |
| Elasticsearch Writer | Configure Elasticsearch Writer | |
| Conversion | Data Filtering | Configure Data Filtering |
| String Replace | Configure String Replace | |
| Data Masking | Configure data de-identification |
Note
- You cannot run a real-time synchronization node on the node configuration tab. Instead, you must run a real-time synchronization node in the production environment after you save and commit the node.
- Real-time synchronization nodes can be run only on an exclusive resource group for Data Integration. For more information, see DataWorks exclusive resources.
Limits
If you want to synchronize data in real time from a relational database, such as a
MySQL, a PolarDB for MySQL, an Oracle, or a PolarDB-X database, note that the following
operations are not supported:
- Online DDL operations are not supported. This feature allows you to perform data definition language (DDL) operations on tables, such as creating an index, without blocking data manipulation language (DML) operations and SELECT queries that run concurrently with the DDL operations. In this case, errors and data quality issues occur when data is written to the destination in real-time synchronization.
- If a column is inserted to or deleted from the source between the start time of the real-time synchronization node and the current time, errors and data quality issues occur when data is written to the destination.