All Products
Search
Document Center

Realtime Compute for Apache Flink:Supported connectors

Last Updated:Feb 29, 2024

This topic describes the types of tables and the connectors that are supported by fully managed Flink.

Table types

Alibaba Cloud Realtime Compute for Apache Flink allows you to use Flink SQL to define a table that provides the mappings between the upstream and downstream storage, or use the DataStream API to access the upstream and downstream storage to perform read and write operations. Realtime Compute for Apache Flink supports the following types of tables that are defined by using Flink SQL:

Source table

  • A source table serves as a data input table of a Flink deployment and triggers a stream processing.

  • A source table cannot be used as a dimension table and must be used as a driving table to promote subsequent computing. The computing results of a source table trigger the computing chain.

  • In most cases, a source table can be used for conversion and computing of up to tens of millions or even hundreds of millions of data records.

  • A source table provides streaming data inputs to trigger and push the data processing of Realtime Compute for Apache Flink. The continuous stream of new data can be from storage such as a message queue service or database change logs.

  • A source table contains key fields that can be joined and associated, such as user ID and order ID as primary keys.

Dimension table

  • A dimension table is an auxiliary table, which is used as an extension of the data of a source table.

  • A dimension table cannot be used as a driving table. It can only be used as a supplement for a source table. A dimension table does not drive computing.

  • In most cases, a dimension table can store only a small amount of data, such as gigabytes or terabytes of data. A dimension table can be a static table or a low-throughput streaming table.

  • A dimension table provides supplemental information to business data, such as user names, product details, and region information.

  • A dimension table can be joined with a source table to enrich the data of the source table to form a more detailed wide table.

Result table

  • A result table is an output data table of a Flink deployment.

  • A result table stores the final result data after computing and conversion, such as aggregation results and data obtained after filtering.

  • The final result data that is stored in a result table can be exported to external systems, such as databases, message queues, and file systems, for subsequent analysis.

  • A result table is the final output of the entire deployment processing chain. It stores the output of the computing.

For example, the following source table and dimension table are provided for data analysis in a deployment:

  • Source table: an order table that contains the columns such as user ID, order ID, and order amount.

  • Dimension table: a user information table that contains columns such as user ID, user name, and address. This is a static table.

When the deployment runs, Realtime Compute for Apache Flink reads real-time order data from the source table and joins the order data stream with the user information in the dimension table. Realtime Compute for Apache Flink aggregates data to obtain the total order amount by region and writes the aggregation results to the result table.

In this deployment, the order table is used as the source table, the user information table is used as the dimension table, and the statistical result table is used as the final output of the deployment. The order table cannot be used as a dimension table and must be used as a driving table for data input. The user information table cannot be used as a driving table and can only be used as a dimension table to provide additional order data.

Supported connectors

Connector

Table type

Running mode

API type

Data update or deletion in a result table

Source table

Dimension table

Result table

Apache Kafka connector

×

Streaming mode

SQL API and DataStream API

Data in a result table cannot be updated or deleted. Data can only be inserted into a result table.

Hologres connector

Streaming mode and batch mode

SQL API and DataStream API

Supported.

Simple Log Service connector

×

Streaming mode

SQL API

Data in a result table cannot be updated or deleted. Data can only be inserted into a result table.

MySQL connector

Streaming mode

SQL API and DataStream API

Supported.

ApsaraDB RDS for MySQL connector

×

Streaming mode and batch mode

SQL API

Supported.

MaxCompute connector

Streaming mode and batch mode

SQL API and DataStream API

Data in a result table cannot be updated or deleted. Data can only be inserted into a result table.

ApsaraDB for Redis connector

×

Streaming mode

SQL API

Supported.

Upsert Kafka connector

×

Streaming mode and batch mode

SQL API

Supported.

Elasticsearch connector

Streaming mode and batch mode

SQL API and DataStream API

Supported.

AnalyticDB for MySQL V3.0 connector

×

Streaming mode and batch mode

SQL API

Supported.

ClickHouse connector

×

×

Streaming mode and batch mode

SQL API

Supported.

Print connector

×

×

Streaming mode and batch mode

SQL API

Supported.

Blackhole connector

×

×

Streaming mode and batch mode

SQL API

Supported.

ApsaraDB for HBase connector

×

Streaming mode

SQL API

Supported.

Datagen connector

×

×

Streaming mode and batch mode

SQL API

N/A.

ApsaraMQ for RocketMQ connector

×

Streaming mode

SQL API and DataStream API

Data in a result table cannot be updated or deleted. Data can only be inserted into a result table.

Tablestore connector

Streaming mode

SQL API

Supported.

JDBC connector

Streaming mode and batch mode

SQL API

Supported.

StarRocks connector

×

Streaming mode and batch mode

SQL API and DataStream API

Supported.

PostgreSQL CDC connector (public preview)

×

×

Streaming mode

SQL API

N/A.

AnalyticDB for PostgreSQL connector

×

Streaming mode and batch mode

SQL API

Supported.

Lindorm connector

×

Streaming mode

SQL API

Supported.

OSS connector

×

Streaming mode and batch mode

SQL API and DataStream API

Data in a result table cannot be updated or deleted. Data can only be inserted into a result table.

Faker connector

×

Streaming mode and batch mode

SQL API

N/A.

Apache Iceberg connector

×

Streaming mode and batch mode

SQL API

Supported.

TSDB for InfluxDB connector

×

×

Streaming mode

SQL API

Not supported.

Apache Paimon connector

Streaming mode and batch mode

SQL API

Supported.

Hudi connector

×

Streaming mode and batch mode

SQL API and DataStream API

Supported.

Tair connector

×

×

Streaming mode

SQL API

Supported.

OceanBase connector (public preview)

×

Streaming mode and batch mode

SQL API

Supported.

MongoDB connector (public preview)

Streaming mode

SQL API and DataStream API

Supported.

PolarDB for Oracle 1.0 connector

×

×

Streaming mode and batch mode

SQL API

Supported.