Imagine managing a warehouse for a large shopping mall. This warehouse stores various items - from product manuals to delivery slips, from product images to transaction records. Over time, several issues emerge: sometimes the same file is stored multiple times; sometimes files that are no longer needed continue occupying space; worse yet, sometimes file contents are accidentally modified without anyone noticing. These issues make warehouse management chaotic and inefficient.
In Apache Flink, the BLOB (Binary Large Object) storage system is like such a warehouse, responsible for managing various binary files. These files might be user-uploaded JAR files, large messages passed between tasks, or TaskManager log files. Before Flink 1.4, this "warehouse" faced similar issues.
Let's understand the new BLOB storage architecture through diagrams:
Like anti-counterfeit tags in stores, the new system performs checksum verification when reading or copying files. This ensures file contents haven't been accidentally modified, improving system reliability.
Imagine a store's inventory management system where product usage status needs real-time tracking and updating. The new BLOB storage system employs smart reference counting to accurately track each file's current usage. When the system detects a file is no longer used by any task, instead of immediate deletion, it waits for a period. During this time, if new tasks need the file, it's retained. It's like a store's "near-expiry products section" where products aren't discarded immediately upon expiration but observed for a while in a special area.
The new BLOB storage system adopts a layered storage system similar to large chain stores. It includes three core components:
BlobServer is like the central warehouse of a large shopping mall and is responsible for the core storage-management work. It manages two important storage spaces: one local warehouse for fast access and another backup warehouse for disaster recovery. Each file follows the <path>/<jobId>/<BlobKey> path convention when stored, keeping files clearly and orderly organized. When a new file is uploaded, BlobServer first stores it in the local warehouse and then immediately synchronizes it to the backup warehouse. This ensures data safety without affecting performance. When a file is requested, it looks in the local warehouse first and searches the backup warehouse only if the file is not found locally.
BlobCache is like the warehouse manager at each branch of the shopping mall. Every branch has its own small warehouse and uses the same storage-path structure as the central warehouse. Although branch warehouse managers can view the central backup warehouse, they have read-only access and cannot modify its contents directly. When a file is needed, the branch manager first checks local inventory. If it is not there, the manager asks the central warehouse to transfer it. To save space, each branch manager can decide when to clean up files that are no longer in use.
BlobClient acts as a warehouse service specialist responsible for handling various file-access requests. Like a professional logistics dispatcher, it establishes a dedicated connection with BlobServer after receiving an upload or download request, ensuring secure and efficient file transfer. It records the details of every request precisely and supervises the entire transfer until it confirms that the file has arrived safely.
When a new BlobServer takes over, it follows four steps:
This process is like an orderly warehouse handover.
Different file types have different purposes and lifecycle-management approaches. JAR files primarily store users' program code; their lifecycle is usually aligned with the job and they are cleaned up only after the job ends. RPC messages are used for communication between tasks; these files typically exist only briefly and can be deleted after use. Log files are mainly used for system monitoring and follow an on-demand storage and timely cleanup strategy to avoid occupying excessive storage space.
When TaskManagers need to exchange very large messages, the sender first stores the message content in BlobServer. The receiver then downloads the content from BlobServer. After the message has been processed, the system decrements its reference count. Once all receivers confirm completion, the messages enter the cleanup queue and await scheduled cleanup.
When a user wants to view TaskManager runtime logs through the Web interface, the Web UI first sends a log-view request. The TaskManager then uploads the relevant log files to BlobServer. The Web UI downloads the logs from BlobServer and displays them to the user. Because log files are usually needed only temporarily, they can be cleaned up after viewing instead of being retained long term.
For Flink 1.4 and later versions, these improvements are enabled by default. Important considerations:
blob.retention.interval parameter defaults to 30 minutes (reduced from 1 hour)FLIP-19 completely overhauled Flink's BLOB storage system, introducing file verification, smart reference counting, and layered storage design. Like a modern intelligent warehouse, it ensures both file safety and efficient space management. The improvement resolves concurrency and cleanup issues in the original architecture while laying groundwork for future extensions (like large RPC message handling). Now a standard feature in Flink 1.4 and later versions, it provides robust support for Flink's stable operation.
210 posts | 60 followers
FollowApsaraDB - July 29, 2026
Alex - September 18, 2018
ApsaraDB - June 27, 2025
francisndungu - December 12, 2018
Alibaba Clouder - March 2, 2021
Apache Flink Community - April 18, 2025
210 posts | 60 followers
Follow
Realtime Compute for Apache Flink
Realtime Compute for Apache Flink offers a highly integrated platform for real-time data processing, which optimizes the computing of Apache Flink.
Learn More
Big Data Consulting for Data Technology Solution
Alibaba Cloud provides big data consulting services to help enterprises leverage advanced data technology.
Learn More
Big Data Consulting Services for Retail Solution
Alibaba Cloud experts provide retailers with a lightweight and customized big data consulting service to help you assess your big data maturity and plan your big data journey.
Learn More
Hologres
A real-time data warehouse for serving and analytics which is compatible with PostgreSQL.
Learn MoreMore Posts by Apache Flink Community