Use DataStream connectors to read from and write to data sources - Realtime Compute for Apache Flink

This topic shows you how to use DataStream connectors to read from and write to data sources.

DataStream connector dependencies and usage

To read from or write to data sources with the DataStream API, use the corresponding DataStream connector to connect to Realtime Compute for Apache Flink. The VVR DataStream connectors are available in the Maven central repository to use directly in your job development.

Important

Use only the connectors that are explicitly listed as supporting the DataStream API in Supported connectors. Connectors not on this list are unsupported, and their interfaces may change without notice.

DataStream connectors are commercially encrypted and cannot be run directly. For local debugging and execution, see Run and debug jobs that contain connectors locally.

You can use a connector in one of the following ways:

(Recommended) Additional dependencies

In your job's Maven pom.xml file, add the required connector as a project dependency with the provided scope.
Note
- ${vvr.version} specifies the engine version for the job's runtime environment. For example, if your job runs on the vvr-8.0.9-flink-1.17 engine, its corresponding Flink version is 1.17.2. We recommend that you use the latest engine. For more information about specific versions, see Engine versions.
- Because the connector's JAR package is introduced as an additional dependency, you do not need to bundle this dependency into the JAR package. Therefore, you need to declare the scope as provided.
```

  <dependency>
      <groupId>com.alibaba.ververica</groupId>
      <artifactId>ververica-connector-mysql</artifactId>
      <version>${vvr.version}</version>
      <scope>provided</scope>
  </dependency>
```

If you need to develop new connectors or extend the functionality of existing connectors, your project also requires a dependency on the common connector package flink-connector-base or ververica-connector-common.

<!-- Base dependency for Flink connector public interfaces -->
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-base</artifactId>
    <version>${flink.version}</version>
</dependency>
<!-- Base dependency for Alibaba Cloud connector public interfaces -->
<dependency>
    <groupId>com.alibaba.ververica</groupId>
    <artifactId>ververica-connector-common</artifactId>
    <version>${vvr.version}</version>
</dependency>

Deploy a JAR job and add the corresponding connector JAR package to the Additional Dependencies section. You can upload a connector that you developed or a connector provided by Realtime Compute for Apache Flink.

For example, add ververica-connector-mysql-1.17-vvr-8.0.9.jar and ververica-connector-kafka-1.17-vvr-8.0.9.jar.

Project dependency

In your job's Maven pom.xml file, add the connectors you need as project dependencies. The following example shows how to add the Kafka and MySQL connectors.
Note
- ${vvr.version} is the engine version of the job runtime environment. For example, if your job runs on the vvr-8.0.9-flink-1.17 engine, the corresponding Flink version is 1.17.2. We recommend that you use the latest engine. For more information, see Engine.
- Because the connectors are packaged directly into the job JAR as project dependencies, they must be in the default compile scope.
```
        
        <dependency>
            <groupId>com.alibaba.ververica</groupId>
            <artifactId>ververica-connector-kafka</artifactId>
            <version>${vvr.version}</version>
        </dependency>
        
        <dependency>
            <groupId>com.alibaba.ververica</groupId>
            <artifactId>ververica-connector-mysql</artifactId>
            <version>${vvr.version}</version>
        </dependency>
```

If you need to develop new connectors or extend the functionality of existing connectors, your project also requires a dependency on the flink-connector-base or ververica-connector-common common connector package.

        <!-- Flink connector public interfaces -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-base</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <!-- Base dependency for Alibaba Cloud connector public interfaces -->
        <dependency>
            <groupId>com.alibaba.ververica</groupId>
            <artifactId>ververica-connector-common</artifactId>
            <version>${vvr.version}</version>
        </dependency>

Important

To prevent dependency conflicts, note the following:

${flink.version} is the Flink version that the job runs on. This version must be the same as the Flink version of the VVR engine that you select on the job deployment page. For example, if you select the vvr-8.0.9-flink-1.17 engine on the deployment page, the corresponding Flink version is 1.17.2. We recommend that you use the latest engine. For details about specific versions, see Engine Versions.
For Flink-related dependencies, set the scope to provided by adding <scope>provided</scope> to the dependency. This mainly includes non-Connector dependencies in the org.apache.flink group that start with flink-.
In the Apache Flink source code, only call methods annotated with @Public or @PublicEvolving. Realtime Compute for Apache Flink guarantees compatibility only for these public APIs.
If a DataStream API is supported by a built-in Flink connector, we recommend using its corresponding built-in dependency.

Realtime Compute for Apache Flink:DataStream connector setup

DataStream connector dependencies and usage

(Recommended) Additional dependencies

Project dependency

Related documentation