Realtime Compute for Apache Flink Release Notes (November 21, 2024) - Realtime Compute for Apache Flink

The November 21, 2024 release of Realtime Compute for Apache Flink includes major features and key bug fixes.

Important

This upgrade is rolling out in stages. For the latest upgrade schedule, see the announcements on the right side of the Realtime Compute for Apache Flink console. New features are available after your account is upgraded. If you require an expedited upgrade, please submit a ticket, and we will review your request.

Overview

The November 21, 2024 release of Realtime Compute for Apache Flink includes platform upgrades, engine updates, connector updates, performance optimizations, and bug fixes.

Platform updates

This platform update improves user experience, system stability, and operational efficiency. Key highlights include:

Hybrid billing: This mode combines the benefits of subscription and pay-as-you-go, allowing you to add an elastic resource limit on top of your fixed resources. This approach helps you avoid over-provisioning for short-term peaks and mitigates the risk of failing to acquire resources when relying solely on elasticity. When used with the platform's auto-tuning capabilities, this mode converts resource savings into cost benefits.
Console homepage redesign: The redesigned homepage now provides an overview of resources and deployments, along with quick access to frequently used features.
Draft renaming: You can now change the name of an existing draft.
Improved version management: As new Flink versions are released, older versions gradually reach their end of support (EOS). To ensure smooth deployment upgrades, the platform now saves your most recently used EOS versions, enabling rollbacks if necessary.

Engine updates

This release introduces VVR 8.0.10, an enterprise-grade Flink engine based on Apache Flink 1.17.2. It includes optimizations and enhancements built on top of the latest community bug fixes. Key changes include:

New capabilities

JDK 11 support: The engine now supports JDK 11, expanding your runtime environment options. Compatibility between minor VVR versions that run on JDK 8 and JDK 11 is not guaranteed.
SelectDB connector (public preview): Lets you write data to ApsaraDB for SelectDB result tables.

Note

These new features are experimental and are not covered by a service level agreement (SLA). Use them with caution in production environments.

Enhancements

Enhanced SQL semantics: You can now perform processing-time temporal joins. This lets you join each record in a fact table with the latest data in a dimension table. Unlike event time, which is based on when an event actually occurred, processing time is based on when data arrives in the processing system.
New built-in SQL function: Introduces the PERCENTILE(expr, percentage[, frequency]) function.
Enhanced CDC data ingestion for YAML deployments: You can now use the Kafka source connector in YAML deployments, with support for Debezium JSON and Canal JSON formats. The Apache Paimon and StarRocks connectors now handle upstream TRUNCATE TABLE and DROP TABLE events from MySQL. The engine now supports merging Decimal and Timestamp fields with different precisions in sharded database and table scenarios.
Enhanced authorization for CTAS/CDAS: You can now use Apache Paimon with DLF 2.0 as the metastore for CREATE TABLE AS (CTAS) and CREATE DATABASE AS (CDAS) statements.
Enhanced StarRocks connector: The StarRocks connector now supports the BIGINT UNSIGNED and VARBINARY data types.

Performance improvements

Optimized integrated full and incremental CDC data ingestion: The engine now uses unified batch and stream processing to optimize real-time ingestion of both full and incremental data from MySQL CDC to Apache Paimon, significantly improving data lake ingestion performance.
Optimized fully managed storage: This improvement reduces the time and cost required to restore from a savepoint and improves storage efficiency.

User experience enhancements

Connectors: The platform now masks unnecessary Debezium configuration parameters to prevent misuse. We have optimized the timeout parameter for the Hologres connector to reduce retries during deployment, improving the reliability of writes.
SQL: We have improved the tips and suggestions for the SinkMaterializer operator in the SQL precheck for better clarity.

Security enhancement

Apache Paimon catalog security enhancement: After you create a catalog, the platform masks the value of the fs.oss.accessKeySecret parameter to protect your credentials.

The following table details key features and related documentation. This release is rolling out in stages. Once available in your region, we encourage you to upgrade the engine for your deployments. For instructions, see Upgrade the engine version of a deployment. We look forward to your feedback.

Key features

Feature	Description	References
Engine support for JDK 11	The engine fully supports JDK 11, allowing developers to use new features to optimize Java applications, expand runtime environment options, and improve platform flexibility and compatibility.	Develop a JAR draft Develop a Python draft UDSF
Keyed-Ordered mode for dimension table joins	We introduced Key-Ordered mode to meet the business requirement of asynchronously fetching data from external systems and processing it in UpsertKey order. This mode addresses the limitations of the existing Ordered and Unordered modes.	Key parameters
Enhanced CDC data ingestion for YAML deployments	YAML deployments now support Kafka as a source, which allows Flink deployments to process Kafka data streams configured through YAML more flexibly.	Kafka connector Develop a YAML draft for CDC data ingestion (public preview)
Optimized SLS connector	A backoff strategy now enhances the stability and reliability of the SLS connector.	N/A
Enhanced StarRocks connector	Supports the `BIGINT UNSIGNED` and `VARBINARY` data types. The connector can automatically expand the length of the `CHAR` type to three times its original size to accommodate encoding differences between MySQL and StarRocks.	StarRocks connector
Enhanced SQL semantics	The engine now supports temporal joins using processing time, which associates fact table data with the latest data in a dimension table.	Processing-time temporal join
New built-in SQL function	This release adds the `PERCENTILE` function.	Supported functions
Optimized Hive catalog	You can now create a Hive catalog in a workspace that uses fully managed storage, directly upload related configuration files during creation, and manage the file lifecycle.	Manage Hive catalogs
Enhanced authorization for CTAS/CDAS	You can now use Apache Paimon with DLF 2.0 in CTAS and CDAS statements.	CREATE TABLE AS (CTAS) CREATE DATABASE AS (CDAS)
Console homepage redesign	The redesigned console homepage provides an overview of resources and deployments, quick access to frequently used features, and links to common documentation, so you can quickly check deployment status, access core features, and find necessary documentation.	N/A
New hybrid billing mode	Combines the flexibility of pay-as-you-go with the cost-effectiveness of subscription to provide a more optimized billing strategy.	Hybrid billing
Optimized log archiving	The platform periodically cleans up expired archived logs for a deployment to reduce extra storage costs.	View the logs of a historical deployment
Support for SelectDB connector	ApsaraDB for SelectDB is a next-generation, cloud-native, real-time data warehouse service based on Apache Doris. You can use the SelectDB connector to write data to ApsaraDB for SelectDB.	SelectDB connector

Fixed issues

Connector issues

MySQL CDC data synchronization: Fixed an issue that could cause data loss when transitioning from a full read to an incremental, binlog-based read from OSS.
Tair data writes: Fixed a defect in the Tair connector's buffered writer in VVR 8.0.9 that resulted in no data output.
OSS write performance: Improved OSS write performance in VVR 8.0.7 and later versions.
YAML deployment timestamp: Fixed a time zone conversion issue for the timestamp type when writing data to Apache Paimon in a YAML deployment.
Dimension table data matching: Fixed a data matching failure in dimension tables for connectors, such as MaxCompute and Table Store (OTS), when a dimension table with a primary key used the SHUFFLE_HASH, REPLICATED_SHUFFLE_HASH, or SKEW strategy with the Cache All policy.

SQL issues

Source Merge configuration: Fixed a startup failure that occurred when table.optimizer.source-merge.enabled was set to true.
Minibatch Interval configuration: Fixed an issue in VVR 8.0.7 where the minibatch interval configuration did not take effect.

Compatibility and dependency issues

Connector class loading: Fixed a connector class not found exception at startup for deployments using a built-in connector with an additional JAR dependency.
IntelliJ IDEA local run: Fixed a ClassNotFoundException MySqlSourceReaderMetrics error that occurred when running a MySQL CDC JAR package locally in IntelliJ IDEA.

Dynamic configuration issues

Fixed an issue where dynamic configuration updates would occasionally fail to take effect.