This topic describes the fields that Data Integration uses to synchronize data in real time.
The following table describes the format of a data record synchronized by Data Integration
from a MySQL or Oracle database in real time.
| _sequence_id_ | _operation_type_ | _execute_time_ | _before_image_ | _after_image_ | Field 1 | Field 2 | Field 3 |
|---|---|---|---|---|---|---|---|
| The position of the data record. | The type of the operation. Valid values: I, D, and U. | The timestamp when the data record was generated. | Indicates whether the data record stores the original data. Valid values: Y and N. | Indicates whether the data record stores the updated data. Valid values: Y and N. | Field 1 in the source database. | Field 2 in the source database. | Field 3 in the source database. |
When Data Integration synchronizes data from databases such as MySQL, Oracle, LogHub
and PolarDB to DataHub or Kafka in real time, Data Integration adds five fields to
data records in the destination data store. These fields are used for operations such
as metadata management, sorting, and de-duplication. The following table describes
the fields that Data Integration adds to the destination data store.
| Field | Type | Description |
|---|---|---|
| _sequence_id_ | STRING | The position of the synchronized data record in the binary log file. It consists of the name of the binary log file and the offset of the data record. |
| _operation_type_ | STRING | The type of the operation. Valid values:
|
| _execute_time_ | LONG | The timestamp in the binlog file, indicating when the data record was generated. |
| _before_image_ | STRING | Indicates whether the data record stores the original data. Valid values: Y and N. |
| _after_image_ | STRING | Indicates whether the data record stores the updated data. Valid values: Y and N. |
In incremental data records generated for the INSERT, UPDATE, and DELETE operations,
the _before_image_ and _after_image_ fields are set:
- INSERT: An incremental data record is generated after an INSERT operation. This data record stores the new data. For this data record, the value of _before_image_ is N, whereas the value of _after_image_ is Y.
- UPDATE: Two incremental data records are generated for an UPDATE operation. One stores
the original data, and the other stores the updated data. The two data records have
the same values for _sequence_id_, _operation_type_, and _execute_time_.
For the data record that stores the original data, the value of _before_image_ is Y, whereas the value of _after_image_ is N. For the data record that stores the updated data, the value of _before_image_ is N, whereas the value of _after_image_ is Y.
- DELETE: An incremental data record is generated after the DELETE operation. This data record stores the original data. For this data record, the value of _before_image_ is Y, whereas the value of _after_image_ is N.