This topic describes the fields that Data Integration uses to synchronize data in real time.

The following table describes the format of a data record synchronized by Data Integration from a MySQL or Oracle database in real time.
_log_file_name_offset_ _operation_type_ _execute_time_ _before_image_ _after_image_ Field 1 Field 2 Field 3
The position of the data record. The type of the operation. Valid values: I, D, and U. The timestamp when the data record is generated. Indicates whether the data record stores the original data. Valid values: Y and N. Indicates whether the data record stores the updated data. Valid values: Y and N. Field 1 in the source database. Field 2 in the source database. Field 3 in the source database.
When Data Integration synchronizes data from relational databases such as MySQL and Oracle databases to DataHub or Kafka in real time, Data Integration adds five fields to data records in the destination data store. These fields are used for operations such as metadata management, sorting, and de-duplication. The following table describes the fields that Data Integration adds to the destination data store.
Field Data type Description
_log_file_name_offset_ STRING The position of the synchronized data record in the binary log file. It consists of the name of the binary log file and the offset of the data record.
_operation_type_ STRING The type of the operation. Valid values:
  • I: INSERT.
  • D: DELETE.
  • U: UPDATE.
_execute_time_ LONG The timestamp in the binlog file, indicating when the data record was generated.
_before_image_ STRING Indicates whether the data record stores the original data. Valid values: Y and N.
_after_image_ STRING Indicates whether the data record stores the updated data. Valid values: Y and N.
In incremental data records generated for the INSERT, UPDATE, and DELETE operations, the _before_image_ and _after_image_ fields are set as follows:
  • INSERT: An incremental data record is generated after an INSERT operation. This data record stores the new data. For this data record, the value of _before_image_ is N, whereas the value of _after_image_ is Y.
  • UPDATE: Two incremental data records are generated for an UPDATE operation. One stores the original data, and the other stores the updated data. The two data records have the same values for _log_file_name_offset_, _operation_type_, and _execute_time_.

    For the data record that stores the original data, the value of _before_image_ is Y, whereas the value of _after_image_ is N. For the data record that stores the updated data, the value of _before_image_ is N, whereas the value of _after_image_ is Y.

  • DELETE: An incremental data record is generated after the DELETE operation. This data record stores the original data. For this data record, the value of _before_image_ is Y, whereas the value of _after_image_ is N.