Imagine a building constructed many years ago: it started as an architect's thesis project, using the trendy architectural style and materials of that time. As time passed, this building became an important landmark in the city, but its original design and materials began to show their limitations. FLIP-28 and FLIP-32 are like a renovation plan, aiming to gradually update its internal structure while keeping the building operational, making it fit for future development needs.
This building, constructed with Scala, faces several major issues:
The building's main structure uses a special material (Scala) that forces all subsequent repairs and expansions to use the same material. This material has unique properties, such as the ability to carve decorations directly into walls (analogous to Scala's implicit conversions) and automatic component matching (analogous to type inference) - features that seemed attractive during initial design.
However, as the building's users increased, this special material revealed unexpected problems. Most tenants (developers) are more familiar with a common material (Java), and some features of the special material actually expose internal structures that should remain hidden (like Scala classes automatically getting getter/setter methods). More troublesome is that even simple renovations (developing new features) become extremely difficult because ordinary construction workers (Java developers) aren't familiar with this special material's properties.
The connector modules are like the building's plumbing system, which should use standard parts (Java) but requires various adapters and encapsulation layers to be compatible with the special material. This not only increases system complexity but also affects overall performance. For example, when Scala code needs to interact with Java code, additional conversion layers are often needed to handle differences between the two language type systems.
The building's layout resembles a mixed-use complex without clear zoning, where office areas, commercial spaces, and equipment rooms are all intermingled. DataSet and DataStream APIs are like two completely different office areas, both tightly coupled with the core functional area (Table API). This layout not only makes daily operations difficult but also complicates renovation work.
The original optimizer is like the building's elevator system, struggling to meet increasing user demands and usage scenarios. The newly developed Blink optimizer is like a modern elevator system, but installing it requires a complete overhaul of the existing shafts and control systems.
FLIP-32 proposes a carefully designed renovation plan to transform the building into a modern modular structure:
Like modern commercial buildings typically divided into podium, main tower, and top facilities, the new Table module is also organized into clear hierarchical layers:
This hierarchical diagram shows the core components and their relationships in the new architecture. Just as each floor in a modern building has its specific purpose, each layer in the new architecture has clearly defined responsibilities. The top-level Table Environment acts like the building's management center, coordinating work across all parts. The API layer is like the main office area, providing various services to users. The Common layer resembles the building's infrastructure, providing necessary support for the entire system. The Planner is like an intelligent elevator system, efficiently coordinating and processing various requests.
Each layer has its clear functional positioning:
| Layer | Function | Analogy |
|---|---|---|
| Common Layer | Basic interfaces and tools | Building infrastructure |
| API Layer | Core programming interfaces | Main office area |
| Planner Layer | Query optimization and execution | Smart elevator system |
| Environment Layer | Runtime environment | Building management system |
Like modern buildings using standardized modules for assembly, the new architecture adopts a modular design philosophy. This design allows different parts to be upgraded and maintained independently:
// Example: Using the new modular API
TableEnvironment tEnv = TableEnvironment.create(TableConfig.builder()
.asStreamingExecution()
.build());
// Unified query interface, implementation details abstracted away
tEnv.from("MyTable")
.select("*")
.insertInto("MyOtherTable");This code demonstrates the simplicity and uniformity of the new architecture. Like a modern building using a unified smart control system, the new Table API provides a unified interface, allowing users to handle different types of data operations in the same way.
The new optimizer system is like installing an intelligent dispatch elevator management system:
This flowchart shows how the query optimization system works. Like a modern building's elevator dispatch system, when passengers make lift requests (analogous to user queries), the smart dispatch center (optimizer interface) selects the most appropriate elevator group for service. The traditional elevator group (Flink optimizer) and new elevator group (Blink optimizer) can coexist, with the smart dispatch center selecting the most suitable group based on factors like passenger count, destination floor, and waiting time (analogous to query characteristics).
This pluggable design allows the building to operate two elevator systems simultaneously while gradually replacing old elevators with new ones without disrupting daily operations. Users don't need to know which elevator system they're using; they just press the button (submit a query) and reach their destination (get query results).
This renovation was completed in Flink 1.14, bringing significant improvements:
FLIP-28 and FLIP-32 demonstrate how to complete a major architectural upgrade while keeping the system operational. Through clear module division, standardized interface design, and gradual update strategy, this renovation not only solved the original architecture's problems but also laid a solid foundation for future development. This renovation approach provides valuable lessons for upgrading other large-scale systems.
FLIP-27: Source Interface Redesign: Building a More Efficient Data Delivery System
219 posts | 60 followers
FollowApache Flink Community China - September 27, 2019
Apache Flink Community China - September 27, 2019
Apache Flink Community China - September 16, 2020
Apache Flink Community - April 25, 2025
Apache Flink Community China - May 14, 2021
Apache Flink Community China - April 19, 2022
219 posts | 60 followers
Follow
Realtime Compute for Apache Flink
Realtime Compute for Apache Flink offers a highly integrated platform for real-time data processing, which optimizes the computing of Apache Flink.
Learn More
Big Data Consulting for Data Technology Solution
Alibaba Cloud provides big data consulting services to help enterprises leverage advanced data technology.
Learn More
Big Data Consulting Services for Retail Solution
Alibaba Cloud experts provide retailers with a lightweight and customized big data consulting service to help you assess your big data maturity and plan your big data journey.
Learn More
Quick BI
A new generation of business Intelligence services on the cloud
Learn MoreMore Posts by Apache Flink Community