Hologres:Use Realtime Compute for Apache Flink or Blink to consume Hologres binary log data in real time

Take note of the following items before you consume Hologres binary logs:

• Hologres V0.9 and later allow you to consume binary logs. Hologres V1.3.21 and later allow you to configure an engine whitelist. If you configure an engine whitelist for an instance whose version is earlier than V1.3.21, the binary logs of the instance cannot be consumed. If the version of your Hologres instance is earlier than V1.3.21, you can join the Hologres DingTalk group for technical support. For more information, see Obtain online support for Hologres.

• Hologres allows you to consume binary logs at the table level. Both row-oriented tables and column-oriented tables are supported. If the version of your Hologres instance is V1.1 or later, you can also consume binary logs of tables that use the row-column hybrid storage mode. After you enable the binary logging feature, column-oriented tables have larger overheads than row-oriented tables in theory. Therefore, we recommend that you enable the binary logging feature for row-oriented tables if data is frequently updated.

• For more information about the support for the binary logging feature and how to enable and configure this feature, see Subscribe to Hologres binary logs.

• Only Alibaba Cloud Realtime Compute for Apache Flink allows you to consume Hologres binary logs. If you use Realtime Compute for Apache Flink to consume Hologres binary logs in HoloHub mode, only simple data types are supported. In Realtime Compute for Apache Flink that uses Ververica Runtime (VVR) 6.0.3 and later, Hologres binary logs can be consumed in Java Database Connectivity (JDBC) mode. Compared with the HoloHub mode, the JDBC mode supports more data types. For more information about data type mappings, see Data type mappings between Realtime Compute for Apache Flink and Hologres. For more information about required permissions for the JDBC mode and HoloHub mode, see Permissions.

• You cannot consume binary logs of parent partitioned tables.

• In Hologres V2.0 and minor versions later than V2.0, the HoloHub mode is supported when specific conditions are met. In Hologres V2.1 and later, the HoloHub mode is no longer supported. Only the JDBC mode is supported. Before you upgrade your Hologres instance, check the Realtime Compute for Apache Flink deployments that use the HoloHub mode and upgrade the VVR version of the Realtime Compute for Apache Flink deployments. For more information, see Change the HoloHub mode to the JDBC mode in this topic.

In this mode, the binary logs consumed in a source table are transferred to a downstream node as regular data of Realtime Compute for Apache Flink. The change type of all data is INSERT. You can process data of a type that is specified by the hg_binlog_event_type field based on your business requirements. After the binary logging feature is enabled for a Hologres table, you can execute the following DDL statement to create a source table to consume binary logs in non-change data capture (CDC) mode in Realtime Compute for Apache Flink:

create table test_message_src_binlog_table(
  hg_binlog_lsn BIGINT,
  hg_binlog_event_type BIGINT,
  hg_binlog_timestamp_us BIGINT,
  id INTEGER,
  title VARCHAR,
  body VARCHAR
) with (
  'connector'='hologres',
  'dbname'='<yourDbname>',
  'tablename'='<yourTablename>',
  'username'='<yourAccessID>',
  'password'='<yourAccessSecret>',
  'endpoint'='<yourEndpoint>',
  'binlog' = 'true',
  'binlogMaxRetryTimes' = '10',
  'binlogRetryIntervalMs' = '500',
  'binlogBatchReadSize' = '100'
);

• The three fields that are prefixed with binlog are system fields. You cannot change the names or data types of these fields.

• Other fields correspond to user fields and must be all lowercase.

In this mode, each row of the binary log data consumed in a source table is automatically assigned an accurate Flink RowKind type, such as INSERT, DELETE, UPDATE_BEFORE, or UPDATE_AFTER, based on the type that is specified by the hg_binlog_event_type field. This way, the binary logs can be mirrored to the source table. This is similar to the CDC feature in MySQL and PostgreSQL.

After the binary logging feature is enabled for a Hologres table, you can execute the following DDL statement to consume binary logs in the CDC source table in real time in Realtime Compute for Apache Flink:

create table test_message_src_binlog_table(
  id INTEGER,
  title VARCHAR,
  body VARCHAR
) with (
  'connector'='hologres',
  'dbname'='<yourDbname>', // The name of the Hologres database.
  'tablename'='<yourTablename>',// The name of the Hologres table.
  'username'='<yourAccessID>',// The AccessKey ID of the current account.
  'password'='<yourAccessSecret>',// The AccessKey secret of the current account.
  'endpoint'='<yourEndpoint>',// The VPC endpoint of your Hologres instance.
  'binlog' = 'true',
  'cdcMode' = 'true',
  'binlogMaxRetryTimes' = '10',
  'binlogRetryIntervalMs' = '500',
  'binlogBatchReadSize' = '100'
);