Realtime Compute for Apache Flink:April 9, 2026 release

AI Function multimodal capabilities

• Multimodal data processing: Added new built-in functions for PDF-to-image conversion, file content retrieval (from OSS and MNS), image clarity detection (based on OpenCV), image compression, and Base64 image pass-through. These functions support image modal data inference by calling Vision Language Models (VLMs) such as Qwen-VL.

• MNS connector: Added a new connector for Message Service (MNS) that allows you to subscribe to OSS change events, to complete the end-to-end real-time AI processing pipeline.

SQL enhancements

• Variant type support: Added the Variant type and its field access syntax (variant.field and variant['key']). This type supports conversion to and from basic types and can be written to Paimon sinks.

• New built-in functions: Added support for hash functions such as MD5 in data ingestion transforms. CDC YAML now includes a parse_json function to convert JSON strings to the Variant type.

Data ingestion (CDC YAML)

The data ingestion feature is now generally available, exiting public preview.

The YAML connectors for Paimon, StarRocks, Hologres, MySQL, and Kafka are now generally available.

The YAML connectors for Doris, OceanBase, MaxCompute, SLS, MongoDB, Postgres, and Fluss have entered public preview.

• Merge multiple columns into a data lake: You can now merge multiple fields from an upstream JSON source with different names or casing into a single target column. This feature supports rules for regular expression matching, case normalization, and custom mappings.

• Append-only writes to partitions: The Paimon sink now supports writing to partitioned tables without a primary key (for append-only use cases), removing the requirement to include the partition key in the primary key.

• Transform enhancements:

  • You can now completely clear a primary key or partition key by passing a null value.

  • You can now define complex table name routing logic by using regular expressions.

• End-to-end Variant support: CDC YAML now supports field access and type conversion for the Variant type and allows writing Variant data to Paimon.

• Source enhancements:

  • Added a new polardbx-cdc source that supports subscribing to binlogs with high parallelism and at the table level.

  • The SLS source now allows you to enforce specific field parsing types.

  • The Kafka source can now split a single message into multiple records based on fields and write them to different target tables (field routing). It also supports custom partitioners.

• Sink enhancements:

  • The Paimon sink now allows you to separately configure the parallelism for commit nodes.

  • The MaxCompute sink now supports DATETIME type mapping and an optimized commit logic to reduce QPS consumption.

  • The Iceberg sink now supports built-in catalog references and automatic retrieval of connection information, such as URLs and credentials, for configuration reuse.

Connectors

• Kafka:

  • The sink now supports writing three-part table IDs (Database.Schema.Table) in Debezium JSON format.

  • Resolved an issue where a stateful restart after changing topics would consume from both the old and new topics simultaneously. Instead, the connector now throws an incompatible state exception.

• MySQL CDC:

  • Improved the error message for expired GTIDs to clearly state the root cause.

  • Added the consumer server ID to logs to simplify troubleshooting.

• PolarDB-X: Now officially supported as a CDC YAML source (in public preview).

• OceanBase: Refactored the JDBC sink write logic to support manual transaction rollbacks and connection pool reuse. This resolves disconnection issues caused by wait_timeout.

• Elasticsearch: The source table and dimension table now support version 8.x (compatible with the ES7 client).

• Doris: Improved the error message for incorrect port configurations.

Data lakehouse integration

• Iceberg:

  • The sink now reports the numRecordsOutOfSinkPerSecond (OUT RPS) metric.

  • Added support for configuring Hadoop-related parameters to improve connection flexibility.

  • Data ingestion jobs can now write to Data Lake Formation (DLF) Iceberg.

• Hologres:

  • The Binlog source table now supports consuming from the LATEST offset.

  • The connector catalog now supports index information such as secondary indexes and prefix scan keys.

  • Added support for reading the varchar[] array type.

  • Optimized parameter detection caching to prevent initialization timeouts when there are many tables.

  • The sink now supports parallelism greater than the number of shards when sink.reshuffle-by-holo-distribution-key.enabled is configured.

• MaxCompute:

  • The catalog now supports paginated queries, which resolves freezing issues in the metadata center.

  • The optimized YAML sink commit logic reduces OOM errors caused by QPS limits.

• Hive: The catalog now allows you to specify a storage format (such as Parquet) when creating a table.

• Paimon: Added support for the Lance file format.

Observability

• Adds local disk usage metrics (geminiDB.disk_space_*).

• Adds Gemini Native memory metrics (geminiDB.native_memory_usage / limit).

• Adds Auto-pilot operator parallelism limit metrics (such as sourceParallelismUpperBound).

• Suppresses non-essential WARN logs, such as alerts when a format does not support snapshots, to reduce noise.