Apache Flink CDC

This topic describes how to use Apache Flink Change Data Capture (CDC) connectors in Realtime Compute for Apache Flink and how to rename them.

Important

Apache Flink CDC connectors are open-source community projects that follow the Apache 2.0 license. Their service support and Service Level Agreement (SLA) differ from those of the commercial connectors included in Realtime Compute for Apache Flink:

If you encounter issues such as configuration failures, job failures, or data loss, you must find solutions in the open-source community. The Realtime Compute for Apache Flink team does not provide technical support for issues related to Apache Flink CDC connectors.
You are responsible for the SLA of any service that uses Apache Flink CDC connectors.

Available CDC connectors

CDC connector	Description
MySQL CDC PostgreSQL CDC (public preview) MongoDB CDC (public preview)	Realtime Compute for Apache Flink includes commercial versions of these connectors. You do not need to manually install the open-source Apache Flink CDC packages for them.
OceanBase CDC Oracle CDC SQL Server CDC TiDB CDC Db2 CDC Vitess CDC	To learn how to use these community-supported Apache Flink CDC connectors, see Use a community CDC connector. Note When you use an Apache Flink CDC connector or a custom connector, if a Realtime Compute for Apache Flink built-in connector or a custom connector with the same name already exists, you must change the default connector name to avoid conflicts with built-in connectors. For SQL Server CDC and Db2 CDC, you must modify the default community connector name and then repackage the connector. For example, rename `sqlserver-cdc` to `sqlserver-cdc-test`. For more information, see Change the connector name.

Version mapping between CDC and VVR

VVR version	CDC version
vvr-6.0.7-flink-1.15	release-2.4
vvr-8.0.11-flink-1.17 ~ vvr-11.3-jdk11-flink-1.20	release-3.4
vvr-11.4-jdk11-flink-1.20+	release-3.5

Use a community CDC connector

SQL job

On the Apache Flink CDC page, click the target community release version. We recommend that you use the latest stable version.

Note
To avoid compatibility issues, select a release version that corresponds to your VVR version. For version mapping, see Version mapping between CDC and VVR.
On the target CDC connector page, download the JAR package.

In the SQL Client JAR section, click the download link for the corresponding JAR file, such as flink-sql-connector-mongodb-cdc-3.0.0.jar.

Note
You can also download the corresponding JAR file from the Maven Central Repository.
Log on to the Realtime Compute for Apache Flink console.
In the Actions column of the target workspace, click Console.
In the left-side navigation pane, click Connectors.
On the Connectors page, click Create Custom Connector and upload the downloaded JAR file.

For more information, see Manage custom connectors.
When you write an SQL job, use the name of the Apache Flink CDC connector as the value for the connector parameter.

For information about the parameters that are supported by each connector, see the Apache Flink CDC documentation.

JAR job

To use the corresponding connector in a JAR job, you need to declare the following dependency in the pom.xml file.
```
<dependency>
  <groupId>com.ververica</groupId>
  <artifactId>flink-connector-${the-connector-name}-cdc</artifactId>
  <version>${the-community-connector-version}</version>
</dependency>
```
The Maven Central Repository contains only release versions, not snapshot versions. To use a snapshot version, clone the GitHub repository and compile the JAR package.
In your code, import the corresponding connector implementation class, and use it as described in the documentation.
Important
Note the difference between the artifacts with different artifact IDs: flink-connector-xxx and flink-sql-connector-xxx.
- flink-connector-xxx: Contains only the connector code. You must declare other dependencies yourself.
- flink-sql-connector-xxx: Bundles all dependencies into a single JAR file that you can use directly.
Select an artifact based on your requirements. For example, to create a custom connector in the Flink development console, use flink-sql-connector-xxx.

Rename a connector

This section demonstrates how to rename a connector, using the community SQL Server CDC connector as an example.

Clone the GitHub repository and switch to the branch for the specified version.

Change the identifier of the SQL Server CDC connector factory class.

//com.ververica.cdc.connectors.sqlserver.table.SqlServerTableFactory
@Override
public String factoryIdentifier() {
    return "sqlserver-cdc-test";
}

Compile and package the flink-sql-connector-sqlserver-cdc submodule.
On the Connectors page, click Create Custom Connector and upload the packaged JAR file.

For more information, see Manage custom connectors.
When you write an SQL job by following the steps in SQL job, pass the name of the connector, connector, as the value for the sqlserver-cdc-test parameter.