This topic describes the fields that Data Integration uses to synchronize data in real time.
The following table describes the format of a data record synchronized by Data Integration
from a MySQL or Oracle database in real time.
| _sequence_id_ | _operation_type_ | _execute_time_ | _before_image_ | _after_image_ | Field 1 | Field 2 | Field 3 |
|---|---|---|---|---|---|---|---|
| The ID of the incremental data record. The ID must be unique and increasing. | The type of the operation. Valid values: I, D, and U. | The timestamp when the data record was generated. | Indicates whether the data record stores the original data. Valid values: Y and N. | Indicates whether the data record stores the updated data. Valid values: Y and N. | Field 1 in the source database. | Field 2 in the source database. | Field 3 in the source database. |
When Data Integration synchronizes data from databases such as MySQL, Oracle, LogHub
and PolarDB to DataHub or Kafka in real time, Data Integration adds five fields to
data records in the destination data store. These fields are used for operations such
as metadata management, sorting, and de-duplication. The following table describes
the fields that Data Integration adds to the destination data store.
| Field | Type | Description |
|---|---|---|
| _sequence_id_ | STRING | The auto-increment ID that is used to merge full data and incremental data. |
| _operation_type_ | STRING | The type of the operation. Valid values:
|
| _execute_time_ | LONG | The timestamp in the binlog file. The timestamp indicates the time when the data record was generated. |
| _before_image_ | STRING | Indicates whether the data record stores the original data. Valid values: Y and N. |
| _after_image_ | STRING | Indicates whether the data record stores the updated data. Valid values: Y and N. |
In incremental data records generated for the INSERT, UPDATE, and DELETE operations,
the _before_image_ and _after_image_ fields are configured:
- INSERT: An incremental data record is generated after an INSERT operation is performed. This data record stores the new data. For this data record, the value of _before_image_ is N, whereas the value of _after_image_ is Y.
- UPDATE: Two incremental data records are generated for an UPDATE operation. One stores
the original data, and the other stores the updated data. The two data records have
the same values for _sequence_id_, _operation_type_, and _execute_time_.
For the data record that stores the original data, the value of _before_image_ is Y, whereas the value of _after_image_ is N. For the data record that stores the updated data, the value of _before_image_ is N, whereas the value of _after_image_ is Y.
- DELETE: An incremental data record is generated after a DELETE operation is performed. This data record stores the original data. For this data record, the value of _before_image_ is Y, whereas the value of _after_image_ is N.