Fully managed Flink - Hologres - Alibaba Cloud Documentation Center

Fully managed Flink is a real-time computing service provided by Alibaba Cloud. It is a fully managed service that is developed based on Apache Flink. Hologres is highly compatible with fully managed Flink. You can use Hologres source tables, dimension tables, result tables, and catalogs in fully managed Flink. This allows you to build a one-stop real-time data warehouse by using fully managed Flink. This topic describes how to use Hologres instances as upstream and downstream storage services to store data in fully managed Flink.

Hologres source tables, dimension tables, and result tables

Fully managed Flink supports Hologres source tables, dimension tables, and result tables. For more information, see Hologres connector.

Real-time consumption of binary logs

Fully managed Flink consumes binary log data in real time by using Hologres connectors. Fully managed Flink that uses VVR 6.0.3 or later supports streaming source tables in Java Database Connectivity (JDBC) mode. Fully managed Flink supports more data types in JDBC mode than those in HoloHub mode and supports custom accounts in JDBC mode. For more information, see Use Realtime Compute for Apache Flink to consume binary logs in real time.

Hologres catalog

Fully managed Flink supports Hologres catalogs. You can read Hologres metadata in the fully managed Flink console without the need to register Hologres tables. This improves job development efficiency and ensures table schema accuracy. For more information, see Manage Hologres catalogs.

Fully managed Flink supports schema evolution and database synchronization based on Hologres catalogs. For more information, see CREATE TABLE AS statement and CREATE DATABASE AS statement.

Hologres DataStream connector

If you use Hologres DataStream connectors to read data from or write data to Hologres, you need to use Hologres DataStream connectors to connect Hologres to fully managed Flink. For more information, see Hologres DataStream connector.

Mappings between data types

For more information about the mappings between data types in fully managed Flink and Hologres, see Data types.

Note

When you deploy SQL jobs in fully managed Flink, you must define data types to those that are supported by fully managed Flink, regardless of whether you use Hologres source tables, dimension tables, or result tables in the SQL jobs. You can define data types to those that are supported by Hologres only when you create an internal table in Hologres.

If you want to use fully managed Flink to write JSON data to Hologres, you must define the data types of columns for the source table and result table in the SQL job to the VARCHAR type, and define the data types of columns for the Hologres internal table to the JSONB type.

Hologres internal table: Set the data type for message to JSONB.

BEGIN ;
DROP TABLE IF EXISTS holo_internal_table;
CREATE TABLE IF NOT EXISTS holo_internal_table
(
    id BIGINT NOT NULL,
    message JSONB NOT NULL
);
CALL set_table_property('holo_internal_table', 'distribution_key', 'id');
COMMIT ;

Flink job: Set the data type for message in the source table and result table to VARCHAR.

CREATE TEMPORARY TABLE randomSource (
    id BIGINT,
    message VARCHAR
  )
WITH ('connector' = 'datagen');

CREATE TEMPORARY TABLE sink_holo (
    id BIGINT,
    message VARCHAR
  )
WITH (
    'connector' = 'hologres',
    'endpoint' = '',
    'username' = '',
    'password' = '',
    'dbname' = '',
    'tablename' = 'holo_internal_table'
  );

INSERT INTO sink_holo
SELECT
  1,
  '{"k":"v"}'
FROM
  randomSource;

Connector release note

For more information about the connector release notes, see Hologres connector release note.

FAQ

For more information about common issues and the solutions to the issues when you use fully managed Flink, see Troubleshoot Blink and Flink issues.