Build a Hologres Source Table for Real-Time Flink Ingestion - Hologres

Use a Hologres table as a data source in a Flink exclusive mode job. By default, the connector scans the entire table once in batch mode and stops — it does not pick up data written after the scan begins. For continuous ingestion from Hologres, see Subscribe to Hologres binary logs.

Scan source: Bounded

Prerequisites

Before you begin, ensure that you have:

A Flink exclusive mode job running version 3.6 or later
A Hologres instance with a VPC endpoint in ip:port format
An Alibaba Cloud AccessKey ID and AccessKey secret

Flink exclusive mode versions earlier than 3.6 do not support Hologres as a data source. To use an older version, obtain the Hologres connector JAR file from the Hologres exchange DingTalk group by contacting technical support. For details, see How do I get more online support?.

Limitations

The Holo-blink connector supports reading a Hologres source table and joining it with a dimension table.
By default, only row-store tables are supported as a source. To read from a column-store table, set bulkread='true'.

Row-store tables with a primary key must also have the primary key configured as a clustering key.

begin;
create table test(a int primary key, b text, c text[], d float8, e int8);
call set_table_property('test', 'orientation', 'row');
call set_table_property('test', 'clustering_key', 'a');
commit;

DDL definition

Use the following Data Definition Language (DDL) statement to create a Hologres source table:

CREATE TABLE holo_dim_table
(
    pk VARCHAR
    ,seller_id VARCHAR
    ,seller_bc_type VARCHAR
    ,seller_tag  VARCHAR
    ,PRIMARY KEY (pk)
) WITH (
    type = 'hologres',
    `endpoint` = '<yourEndpoint>',   -- VPC endpoint of the Hologres instance, in ip:port format
    `username` = '<yourUsername>',   -- AccessKey ID of your Alibaba Cloud account
    `password` = '<yourPassword>',   -- AccessKey secret of your Alibaba Cloud account
    `dbname` = '<yourDbname>',       -- Hologres database name
    `tablename` = '<yourTablename>', -- Hologres table name
    `bulkread` = 'true'              -- Set to true to read from a column-store table
);

Placeholder	Description	Where to find it
`<yourEndpoint>`	VPC endpoint of the Hologres instance, in `ip:port` format	Hologres console > instance page > Network Information
`<yourUsername>`	AccessKey ID of your Alibaba Cloud account	AccessKey Management console
`<yourPassword>`	AccessKey secret of your Alibaba Cloud account	AccessKey Management console
`<yourDbname>`	Name of the Hologres database	—
`<yourTablename>`	Name of the Hologres table	—

WITH clause parameters

Parameter	Required	Default	Description
`type`	Yes	`hologres`	Type of the data source.
`dbname`	Yes	—	Name of the Hologres database.
`tablename`	Yes	—	Name of the Hologres table.
`username`	Yes	—	AccessKey ID of your Alibaba Cloud account.
`password`	Yes	—	AccessKey secret of your Alibaba Cloud account.
`endpoint`	Yes	—	VPC endpoint of the Hologres instance, in `ip:port` format. Find this in the Network Information section of the instance page in the Hologres console.
`bulkread`	Yes	—	Set to `true` to read from a column-store table. Set to `false` to read from a row-store table only.

Batch read mode

By default (bulkread=false), the connector reads from a row-store table. If you need to read from a column-store table, set bulkread='true'.

When using a row-store table with a primary key, you must configure the primary key as a clustering key. See the example in Limitations.

Data type mapping

For more information about the data type mapping between Flink exclusive mode and Hologres, see Data types.

Real-time consumption of Hologres binary logs

Flink exclusive mode 3.7 and later supports real-time consumption of Hologres binary logs through the Hologres connector. For details, see Consume Hologres binary logs in real time using Flink/Blink.

What's next

Subscribe to Hologres binary logs — consume Hologres data in real time instead of batch mode
Data types — full data type mapping between Flink exclusive mode and Hologres