This topic describes the fields that Data Integration uses to synchronize data in real time.
The following table describes the record format for real-time synchronization of MySQL data in Data Integration.
_sequence_id_ | _operation_type_ | _execute_time_ | _before_image_ | _after_image_ | Field 1 | Field 2 | Field 3 |
The record ID of an incremental event. The value is unique and auto-incrementing. | The operation type (I, D, or U). | The timestamp of the data. | Indicates whether the record contains the data before the change (Y/N). | Indicates whether the record contains the data after the change (Y/N). | Actual data field 1. | Actual data field 2. | Actual data field 3. |
When Data Integration synchronizes data in real time from sources such as MySQL, Loghub, and PolarDB to DataHub or Kafka, it adds five additional columns to the destination. These columns are used for operations such as metadata management, sorting, and deduplication.
Parameter | Type | Description |
_sequence_id_ | STRING | The auto-increment ID used to merge full and incremental data. |
_operation_type_ | STRING | The operation type. Valid values are:
|
_execute_time_ | LONG | The timestamp when the data was generated. This is the binlog timestamp. |
_before_image_ | STRING | Indicates whether the record contains the data before the update. Valid values are Y and N. |
_after_image_ | STRING | Indicates whether the record contains the data after the update. Valid values are Y and N. |
For different operation types such as INSERT, UPDATE, and DELETE, the _before_image_ and _after_image_ fields in incremental data records are defined as follows:
If the operation type is INSERT, the generated record contains the inserted data. The value of _before_image_ is N, and the value of _after_image_ is Y.
If the operation type is UPDATE, Data Integration splits the operation into two records: one for the data before the update and one for the data after the update. These two incremental data records have the same values for _sequence_id_, _operation_type_, and _execute_time_.
For the data record that stores the original data, the value of _before_image_ is Y, whereas the value of _after_image_ is N. For the data record that stores the updated data, the value of _before_image_ is N, whereas the value of _after_image_ is Y.
If the operation type is DELETE, the incremental data record contains the deleted data. For this record, the value of _before_image_ is Y, and the value of _after_image_ is N.