Realtime Compute for Apache Flink - Ververica Runtime (VVR) 8.0.7 release
May 29 2024
Realtime Compute for Apache FlinkContent
Target customers: all enterprise customers. New features: · Real-time lakehouse: The Apache Paimon connector SDK is upgraded to support the data lake format used by Apache Paimon 0.9. · SQL optimization: State time-to-live (TTL) hints are introduced to separately configure the state TTL for regular join and group aggregation operators. This helps control the state sizes of the operators. Named parameters are supported in user-defined functions (UDFs) to improve development efficiency and reduce maintenance costs. · Connector optimization: The MongoDB connector has completed its public preview and is generally available (GA). The MongoDB connector can be used to create change data capture (CDC) source tables, dimension tables, and result tables. The MySQL and ApsaraDB for Redis connectors are also optimized: MySQL CDC: · The op_type virtual column is supported to pass the operation types (+ I, +U/-U,-D) of the changed data to downstream systems. This allows you to implement business logic and data cleaning policies based on different operation types. · The read performance is optimized for MySQL tables whose primary keys are of the Decimal type. Parallel processing of data change records (SourceRecords) is supported in large-size tables to improve efficiency. · The source reuse feature is introduced. After you enable this feature, if a Flink deployment contains MySQL CDC source tables that have the same configurations (excluding the database name, table name, and server ID), the tables can be merged to reduce the pressure on the MySQL server caused by database connection and change listening. · Buffered execution is supported to improve the processing performance by several times. To enable buffered execution, configure the sink.ignore-null-when-update parameter. · ApsaraDB for Redis: Different DDL statements are supported for non-primary keys to improve code readability when you use the ApsaraDB for Redis connector to create a dimension or result table that contains data of the HashMap type. Prefixes and delimiters can be configured for keys to meet data governance requirements. · Catalog management: The view information is no longer displayed in a MySQL catalog. This prevents data operation errors because a MySQL view is a logical structure and cannot read and write data. · Security: Hadoop 2.x is supported for Hadoop clusters that enable Kerberos authentication. Sensitive information, such as connector configurations, is masked in logs.
Help Document
https://www.alibabacloud.com/help/flink/product-overview/2024-05-29-version
