Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog FLIP-29: Table API Enhancements in Flink

FLIP-29: Table API Enhancements in Flink

FLIP-29: Table API Enhancements in Flink

Introduction

Imagine a basic kitchen with just a multipurpose knife and a single pot. While you can handle basic cooking tasks, creating sophisticated dishes feels like an uphill battle. This mirrors the early days of Flink's Table API - it could handle basic data processing, but when faced with complex scenarios, the tools felt inadequate.

FLIP-29 is comparable to a complete kitchen renovation, equipping Flink's Table API with a suite of handy "utensils" that make data processing more intuitive. This improvement was implemented in Flink 1.9, so let's explore what this kitchen makeover brought to the table.

Why Was a Renovation Needed?

Let's first look at a diagram showing different data processing scenarios:

_2026_11_09_15_19_16

Much like a kitchen needs various tools for different ingredients, data processing requires different operators for various types of data. The original Table API had some limitations:

  1. When you needed to transform one data record into multiple records - like slicing a potato into french fries - you had to use a cumbersome combination of join and select operations.
  2. When aggregating multiple data records into one - similar to combining ingredients into a single dish - the original aggregate functions could only output one result, as if all dishes had to be served on a single plate.
  3. When transforming data into new formats - like dicing ingredients - you had to write multiple UDFs, akin to using the same knife repeatedly for different cuts.

What Did the Renovation Bring?

This upgrade brought five new tools to the Table API's "kitchen":

_2026_11_09_15_19_27

1. map: Simplified Data Transformation

Like having a food processor that can slice carrots into multiple shapes at once, the new map operator lets you transform a single row into multiple columns:

val result = table
  .map(new MultiOutputFunction())  // Transform to multiple columns
  .select('name, 'age, 'address)   // Select needed columns

2. flatMap: One-to-Many Data Transformation

Similar to a dough divider that can split one piece into many, flatMap allows you to transform a single row into multiple rows:

val result = table
  .flatMap(new OneToManyFunction())  // One row to many
  .select('id, 'value)              // Select needed columns

3. aggregate: Flexible Multi-Row Aggregation

Resembling a blender that can combine various ingredients into a smoothie, the new aggregate operator enables multi-row aggregation with multiple output columns:

val result = table
  .groupBy('category)               // Group by category
  .aggregate(new MultiColumnAgg())  // Aggregate to multiple columns
  .select('category, 'avg, 'max)   // Select needed columns

4. flatAggregate: Unrestricted Aggregation

Like a wok that can both combine ingredients and portion them out, flatAggregate allows you to transform multiple input rows into multiple output rows:

val result = table
  .groupBy('category)                    // Group by category
  .flatAggregate(new Top3Function())     // Calculate top 3 per group
  .select('category, 'value, 'rank)      // Select needed columns

5. TableAggregateFunction: Power Aggregation Tool

Comparable to a multi-function food processor, TableAggregateFunction provides a complete interface for precise control over the aggregation process:

  1. createAccumulator: Prepare the workspace
  2. accumulate: Add new ingredients
  3. retract: Remove ingredients (particularly useful in stream processing)
  4. merge: Combine intermediate results
  5. emitValue: Output the final product

Usage Tips

Here are some practical tips for using these new tools:

  1. Time Attribute Handling After using map and flatMap, time attributes are lost - similar to how ingredients need new freshness labels after processing.

  2. Grouping and Windowing When using aggregate and flatAggregate, pay attention to group key handling:

    val result = table
  .window(Tumble.over("10.minutes").on('rowtime).as('w))
  .groupBy('w, 'category)
  .aggregate(new MultiColumnAgg())
  .select('w.start, 'category, 'value)
  3. State Management In stream processing, manage states carefully - like keeping your workspace clean while cooking.

Summary

FLIP-29's renovation of Flink Table API is comparable to equipping a kitchen with modern appliances. Through the addition of map, flatMap, aggregate, flatAggregate operators, and the powerful TableAggregateFunction, data processing became more flexible and efficient.

These new tools not only make code more concise and elegant but also handle more complex data processing scenarios. Just as a well-equipped kitchen can produce more varied dishes, the upgraded Table API can better handle various data processing requirements.

This feature was implemented in Flink 1.9, significantly enhancing Flink's data processing capabilities. However, like a chef choosing the right tool for different dishes, these powerful new tools should be used appropriately based on specific scenarios.

Big Data Data analysis Apache flink Realtime Compute for Apache Flink
0 0 0
Share on

Read previous post:

FLIP-28 & FLIP-32: Renovating the Flink Table Building

Apache Flink Community

219 posts | 60 followers

You may also like

Comments

Apache Flink Community

219 posts | 60 followers

Related Products

More Posts by Apache Flink Community

See All