Realtime Compute for Apache Flink - Materialized tables
Dec 20 2024
Realtime Compute for Apache FlinkContent
Target customers: all enterprise customers. New features/specifications: Realtime Compute for Apache Flink serves as a unified stream and batch processing platform, providing a comprehensive technical solution that meets the diverse data timeliness needs of businesses. To this end, Realtime Compute for Apache Flink is introducing materialized tables, a feature dependent on Apache Paimon as an integrated stream-batch storage. Different from the traditional way of separately defining streaming and batch job logic, materialized tables allow you to define data freshness in Flink SQL. Flink will attempt to refresh data at the defined interval. This approach streamlines ETL processes, seamlessly transitions jobs between stream and batch modes, offers cascading update capabilities, and significantly enhances data update efficiency. Flink's materialized tables facilitate seamless integration of streaming and batch processing. This feature allows for processing streaming and historical data on the data lake, creating a new paradigm for data development. By unifying the data, metadata, and data processing layers, materialized tables effectively address issues like data duplication, inconsistent data processing logic, and the use of different engines for various tasks. Materialized tables are ideal for scenarios where the Lambda architecture cannot ensure consistent data processing logic, real-time statistics are required for offline BI reports, and real-time dashboard applications rely on historical data for accuracy.
Help Document
https://www.alibabacloud.com/help/flink/product-overview/2024-12-20-version?spm=a2c63.p38356.help-menu-45029.d_0_0_0_0.123761022kAg0r