This topic describes the major updates and bug fixes in the Realtime Compute for Apache Flink version released on January 15, 2025.
The version upgrade is gradually released to users. For more information, see the latest announcement in the Realtime Compute for Apache Flink console. You can use the new features in this version only after the upgrade is complete for your account. To apply for an expedited upgrade, submit a ticket.
Overview
This release includes updates of the platform, engine, and connectors, along with bug fixes.
Platform updates
Platform updates focus on improving deployment and application development capabilities, O&M efficiency, and system stability. Below are the highlights:
Hive dialect is supported in batch SQL script development, improving interoperability with Apache Hive and enabling seamless migration of Hive workloads to Flink.
Data backfilling is supported in Workflows to fill in missing data and fix errors in historical data.
The upgraded network architecture supports the search for a specific deployment based on the source or destination system's IP address.
When purchasing a workspace, you do not need to select the zone. If you set the deployment model to single zone, the optimal zone is automatically chosen. If you set the deployment model to cross zone, your jobs will fail over to the secondary zone when faults occur in the primary zone. This helps ensure job continuity and high availability.
After a deployment stabilizes in autopilot mode using the stable strategy, you can view, edit, and save the generated tuning plan.
Runtime parameters for a deployment now support namespace variables, enhancing security by eliminating the need to use plaintext AccessKey pairs and credentials.
Engine updates
Ververica Runtime (VVR) 8.0.11 is officially released to provide an enterprise-grade engine based on Apache Flink 1.17.2. It also provides optimizations and enhancements beyond the latest bug fixes in Apache Flink. VVR 8.0.11 includes the following updates:
Engine
The pre-installed Python version is upgraded to 3.9, with significant improvements in performance, functionality, and security. If you upgrade the VVR version to 8.0.11, you must re-test your Python script, re-deploy it, and restart the Python deployment.
Connectors
Hologres: Consuming data from a Hologres partitioned table is now supported; Metadata columns are supported to enrich table information. Streaming writes are optimized to support conditional updates and aggressive commit mode.
MaxCompute: Updating data in specific column of Delta tables is supported.
StarRocks: When a Flink CHAR field is mapped to a StarRocks CHAR field, the StarRocks field's length is automatically extended. Now, the extension is increased to handle special characters, such as emojis.
Materialized table: When batch mode is enabled, the engine will dynamically choose between incremental or full updates according to specific situations, with the former being the preferred approach.
Features
Item | Description | References |
Hive syntax compatibility | Hive dialect is supported to develop the batch SQL script, enabling seamless migration of Hive workloads to Flink. | |
Optimized Workflows | Data backfilling is supported in Workflows to fill in missing data and fix errors in historical data. | |
Easy deployment search | The upgraded network architecture supports deployment searches based on the source or destination system's IP address and port. This can help you easily find a specific deployment, especially when Flink is handling requests from numerous external systems. | |
Streamlined workspace creation | Deployment model configuration is supported when you purchase a workspace.
| |
New use case of namespace variables | Namespace variables are supported in a deployment's runtime parameter configurations. | |
Optimized automatic tuning | Autopilot mode offers two tuning plan options: schedule-based and fixed-resource. You can view, edit, save, and apply the generated tuning plan. | |
Enhanced Hologres connector |
| |
| ||
Enhanced MaxCompute connector |
| |
StarRocks data mappings | When the CHAR data type is mapped from Flink to StarRocks, the field length is increased to four times that of the original. | |
Incremental updates for materialized tables | When batch execution mode is enabled, materialized tables support incremental updates. | |
Python version upgrade | The Python version is upgraded from 3.7.9 to 3.9.21. If you want to upgrade the VVR engine version for your Python deployment to 8.0.11 or later, run a compatibility test on the Python script, re-deploy it, and restart the deployment. |
Fixed issues
Connector issues
Fixed an issue where a null pointer exception was reported upon the MySQL connector startup.
Fixed an issue where performance degradation occurred when data is being written to a MySQL table lacking the primary key using the MySQL connector. This issue had been introduced after the VVR version was upgraded.
Fixed an issue where the reuse of a Kafka source caused a misalignment between JSON messages in Canal format and metadata columns.
Fixed an issue where the following exception occured upon startup of the ApsaraDB for HBase connector:
No length info found when processingnull.Fixed an issue where using the Simple Log Service Catalog reported the error:
AssertionError: Conversion to relational algebra failed.
SQL issues
Fixed an issue where the window was not triggered due to a delayed watermark emittance.
Fixed an issue where adding a Bit (1) column to a table created using the CTAS statement reported the following error:
ValidationException: Binary string length must be between 1 and 2147483647.
Stability issues
Fixed an issue where a deployment was recovered from the wrong checkpoint after being abnormally terminated with exit code 137.