This topic describes the major updates and bug fixes of the Realtime Compute for Apache Flink version released on July 22, 2024.
This upgrade rolls out incrementally using a canary release strategy. The new features in this release are available only after the upgrade completes for your account. Check the latest upgrade schedule on the right side of the Realtime Compute for Apache Flink console. To request an earlier upgrade, submit a ticket.
Overview
This release delivers 13 new features and 7 bug fixes across the platform and engine layers.
Platform highlights: Custom roles with fine-grained permissions, UI improvements to job details and resource configuration, and MaxCompute catalog creation from the console.
Engine highlights: Ververica Runtime (VVR) 8.0.8, based on Apache Flink 1.17.2, with connector enhancements for Elasticsearch, Hologres, StarRocks, Simple Log Service (SLS), MySQL, and MaxCompute; new SQL built-in functions; and performance and security improvements.
After the upgrade completes for your account, upgrade the VVR engine to 8.0.8. For instructions, see Upgrade the engine version of a deployment.
New features
|
Feature |
Description |
References |
|
Custom roles |
Create custom roles and assign fine-grained permissions based on your business requirements. This gives you precise control over who can perform which job-related operations, reducing the risk of over-permissioned accounts. |
|
|
UI optimization: job details |
View the parameter configuration used to start a job directly from the deployment details page: click the Status tab, then click Job Details in the Actions section. This eliminates the need to search through configuration history when debugging a running job. |
N/A |
|
UI optimization: resource configuration |
The maximum allowed values for resource parameters are now displayed in the Resources section of the Configuration tab. This prevents configuration errors caused by exceeding parameter limits. |
N/A |
|
MaxCompute catalog creation on the console |
MaxCompute catalogs can be created on the GUI, which facilitates the configuration and management of Flink jobs and improves data management and development efficiency. |
N/A |
|
StarRocks connector: JSON support |
The StarRocks connector now supports the JSON data type. This resolves the exception that occurred when writing JSON data from MySQL to StarRocks. |
|
|
Elasticsearch connector enhancements |
|
|
|
Simple Log Service (SLS) connector: startup modes |
Consume SLS data starting from the latest or earliest offset. This gives you control over whether a new job picks up only new data or reprocesses historical data. |
|
|
PyFlink Docker image upgrade |
The base Docker image for PyFlink is upgraded to improve compatibility with different Python and glibc versions. This reduces friction when running PyFlink jobs in containerized environments. |
N/A |
|
URL_DECODE and URL_ENCODE functions |
Two built-in SQL functions for URL encoding and decoding are added. These eliminate the need for custom UDFs when processing URL-encoded strings in your SQL jobs. |
|
|
MySQL connector: ApsaraDB RDS for MySQL endpoint support |
Configure ApsaraDB RDS for MySQL endpoints in the MySQL connector. MySQL Change Data Capture (CDC) can now read binary logs stored in the associated Object Storage Service (OSS) bucket. |
|
|
Hologres connector: per-column deletion policies |
Configure deletion policies independently for each column in a Hologres sink table. For example, delete entire rows, or set non-primary-key column values to null without affecting other columns in partial update scenarios. |
|
|
Custom partitioner for dimension table joins |
Configure the shuffle strategy for dimension table join operations. This lets you optimize data distribution across parallel tasks and improve overall join throughput. |
|
|
MaxCompute connector: Arrow format and dynamic shard allocation |
Read MaxCompute source data in Arrow format with dynamic shard allocation. This significantly improves read throughput and reduces the data writing workload on Flink clusters. |
Performance improvements
-
Apache Paimon dimension table joins: Join performance for Apache Paimon dimension tables is improved.
-
MySQL CDC writes to Hologres: The speed of writing both full and incremental data to Hologres using MySQL CDC is improved.
-
MaxCompute source table reads: Read speed from MaxCompute source tables is improved, and the data writing workload on Flink clusters is reduced.
Security improvements
Passwords in the job topology are now encrypted, preventing sensitive credentials from being exposed in job graphs.
Fixed issues
-
Data conflicts occur when multiple streams write to the same Apache Paimon table during partial updates.
-
When MySQL CDC encounters out-of-memory (OOM) errors while parsing a large number of binary logs, the system silently retries without surfacing the error. Starting from this release, the system throws an exception and triggers a job failover.
-
When GeminiStateBackend encounters OOM errors, the system silently retries without surfacing the error. Starting from this release, the system throws an exception and triggers a job failover.
-
The consumption status of the ApsaraMQ for RocketMQ connector is not displayed in the ApsaraMQ for RocketMQ console.
-
If a Hologres source table schema changes (for example, due to a TRUNCATE operation) in a job started from state data, snapshot restoration fails.
-
The
java.lang.NoClassDefFoundError: StringUtilserror may occur when using the StarRocks connector. -
All issues fixed in Apache Flink 1.17.2. For the full list, see the Apache Flink 1.17.2 Release Announcement.