Flink CDC source and sink modules - Realtime Compute for Apache Flink

This topic describes the source and sink modules for Flink Change Data Capture (CDC) data ingestion jobs and lists the supported connectors.

Connector parameter configuration

You can configure the parameters for the source and sink connectors in a Flink CDC data ingestion job. The supported connectors and their parameters are described below.

# Source module
source:
  type: mysql # Or another connector identifier
  name: MySQL Source
  # Other parameters. Use key: value pairs.

# Sink module
sink:
  type: paimon # Or another connector identifier
  name: Paimon Sink
  # Other parameters. Use key: value pairs.

Supported connectors

Connector	Supported type
Connector	Source	Sink
MySQL Note Connects to ApsaraDB RDS for MySQL, PolarDB for MySQL, and self-managed MySQL.	√	×
Streaming Lakehouse Paimon	×	√
Message Queue for Kafka	√ Note Requires Ververica Runtime (VVR) 8.0.10 or later.	√
Upsert Kafka	×	√
StarRocks	×	√
Hologres	×	√
Simple Log Service (SLS)	√ Note Requires Ververica Runtime (VVR) 11.1 or later.	×
MongoDB	√ Note Requires Ververica Runtime (VVR) 11.2 or later.	×
MaxCompute	×	√ Note Requires Ververica Runtime (VVR) 11.1 or later.
SelectDB	×	√ Note Requires Ververica Runtime (VVR) 11.1 or later.
Postgres CDC (public preview)	√ Note Requires Ververica Runtime (VVR) 11.4 or later.	×
Print	×	√

Reuse connection information from an existing Catalog

Starting from Ververica Runtime (VVR) 11.5, you can reuse connection information from existing Catalogs in your Flink CDC data ingestion jobs. You can reference a built-in Catalog created on the Data Management page to automatically retrieve connection properties, such as the URL, username, and password. This reduces manual configuration effort.

Syntax

source:
  type: mysql
  using.built-in-catalog: mysql_rds_catalog
  
sink:
  type: paimon
  using.built-in-catalog: paimon_dlf_catalog

In the source and sink modules, use the using.built-in-catalog syntax to reference your built-in Catalog.

For example, in the code above, the metadata for the mysql_rds_catalog Catalog already includes required parameters such as hostname, username, and password. You do not need to specify these parameters again in the YAML job.

Limits

The following connectors support reusing connection information from a Catalog:

MySQL (source)
Kafka (source)
Upsert Kafka (sink)
StarRocks (sink)
Hologres (sink)
Paimon (sink)
SLS (source)

Note

Catalog parameters that are incompatible with the CDC YAML syntax do not take effect. For more information, see the parameter list for each connector.