By Alibaba Cloud Messaging Team
Recently, Alibaba Cloud ApsaraMQ for Kafka officially launched its native data lake ingestion capability. By natively integrating Apache Iceberg with Alibaba Cloud Object Storage Service (OSS) Table Bucket, users can route real-time streaming data from Kafka topics directly into the data lake with a single click. Bypassing the need to deploy ETL tools like Spark, Flink, or Kafka Connect, this achieves an end-to-end unification of "real-time streaming + lakehouse analytics." This capability entered Public Beta in June 2026, with General Availability (GA) slated for September.
In today's landscape, where cloud-native architectures and the lakehouse paradigm have become mainstream, combining Kafka with data lakes to build real-time analytical capabilities is an undeniable trend. However, the traditional "Kafka-to-data-lake" pipeline is notoriously cumbersome. Every topic requires standalone Spark or Flink jobs for consumption, transformation, and ingestion. Any schema changes must be manually synchronized to both the ETL logic and the Iceberg table structures. Furthermore, "invisible operational overhead" such as small file compaction, expired snapshot cleanup, and partition optimization severely exacerbates platform complexity and human resource costs.
The native data lake ingestion capability introduced by ApsaraMQ for Kafka is designed specifically to shatter these bottlenecks. Users simply enable the data lake ingestion configuration for a designated topic via the console, and the Kafka broker automatically handles data format conversion, schema extraction, partition routing, and metadata commits in the background. The entire process requires absolutely zero ETL code and eliminates the need to maintain external compute clusters.
Under the traditional model, N topics usually dictate N sets of Spark/Flink jobs, N sets of monitoring alerts, and N sets of resource scheduling policies. ApsaraMQ for Kafka's ingestion capability embeds the data transformation and writing processes directly into the broker layer. By utilizing built-in worker components, it automatically converts Kafka records into Parquet format and writes them to the OSS Table Bucket, completely freeing users from deploying any external compute tasks. Backed by exactly-once semantics, it ensures that no data is duplicated or lost during the transit process.
One of the core challenges in data governance is schema fragmentation and inconsistency. ApsaraMQ for Kafka's ingestion capability deeply integrates with Kafka Schema Registry, centrally managing schema definitions that were previously scattered across producer code, Flink jobs, and Iceberg tables. When a producer sends data, the system automatically validates it against the registered schema. When a schema evolves, the system extracts the new version information and synchronizes the updates to the Iceberg table structure, ensuring uninterrupted continuous data ingestion. This is empowered by Iceberg's native support for schema evolution, allowing for column additions, deletions, and type modifications without the need to rewrite datasets or disrupt downstream tasks.
To cater to diverse business scenarios, ApsaraMQ for Kafka's ingestion supports two write modes:
• Append Mode: Ideal for append-only scenarios like logs and event streams. Data is written to Iceberg tables via append operations, supporting highly efficient batch commits.
• CDC Mode: Designed for Database Change Data Capture (CDC) scenarios, natively supporting Upsert operations. Users simply specify the Key fields, and the system efficiently executes data inserts, deletes, and modifications based on Iceberg's incremental file mechanism without rewriting the entire table.
Additionally, the system provides flexible transformation strategies, including raw archiving and structured archiving (Flatten/Debezium), satisfying a wide spectrum of requirements from simple log persistence to complex CDC synchronization.
Through simple configurations, users can define multi-column partitioning strategies (e.g., partitioned by date) for their Iceberg tables. Query engines can then prune partitions based on query conditions, avoiding full-table scans and significantly boosting analytical efficiency. Simultaneously, once data is written to the OSS Table Bucket, it automatically inherits OSS's backend maintenance capabilities. Crucial tasks like small file compaction, expired snapshot cleanup, and orphan file removal are all executed in the background at the storage layer. This consumes zero user compute resources or business bandwidth, thoroughly liberating enterprises from the tedious grind of data lake operations.
The underlying storage for ApsaraMQ for Kafka's ingestion capability relies on the Table Bucket service launched by Alibaba Cloud OSS. Built upon OSS's massive, secure, and highly cost-effective storage foundation, OSS Table Bucket natively implements Apache Iceberg table semantics while remaining fully compatible with the Iceberg REST API and S3 Table API.
Thanks to end-to-end deep optimizations based on the OSS underlying technical architecture, Table Bucket delivers a TPS more than 10 times higher than self-built Iceberg solutions on generic buckets during large-scale read/write scenarios. Moreover, Table Bucket seamlessly integrates with the OSS Accelerator, allowing on-demand scaling of throughput bandwidth to effortlessly handle high-concurrency read peaks like LLM training and real-time analytics. The exact same copy of table data can be directly read and written by mainstream query engines like Spark, Flink, Trino, and StarRocks, putting a definitive end to the siloed architecture of "ETL replication + multiple data copies."
During major e-commerce promotions, hundreds of thousands of order, payment, and logistics events are generated every second. Platforms typically have to maintain a massive fleet of Flink jobs to consume various business-line Kafka topics and write them into Iceberg tables. Job scheduling, resource contention, and fault recovery drain enormous energy from data teams. Furthermore, schema changes (such as temporarily adding a subsidy field during a sale) require coordinating multiple teams to synchronously modify ETL logic, where a single misstep can cause data outages.
By adopting ApsaraMQ for Kafka's native ingestion capability, topics across all business lines can enable data lake ingestion with a single click. The broker handles conversion and writing automatically, reducing the Flink job count to zero. Business dashboards leap from "hour-level refreshes" to "minute-level refreshes," allowing operations teams to track GMV, conversion rates, and inventory levels in real time, while data teams can finally focus their energy on metric modeling and strategy optimization. Coupled with OSS Table Bucket's automated small file compaction and partition pruning, ad-hoc query performance is improved by more than 10x compared to self-built solutions.
Financial institutions frequently need to synchronize MySQL data from trading systems to data lakes in real time for risk control modeling and regulatory reporting. Under the traditional architecture, change events captured by Debezium must pass through the gauntlet of "Kafka-Flink-Iceberg." Upsert logic must be manually implemented within Flink jobs; a failure at any node can trigger data inconsistencies, with repair windows lasting hours.
Using the CDC Mode of ApsaraMQ for Kafka's ingestion capability, the change stream captured by Debezium is routed directly into a Kafka topic. The system automatically identifies Key fields and writes to the Iceberg table using Upserts. Iceberg's atomic commit mechanism guarantees exactly-once semantics, completely eliminating the risks of data duplication and loss. Schema evolution is handled automatically by the system—when a business table adds a field, the downstream Iceberg table scales synchronously without human intervention. This slashes the operational cost of the entire synchronization pipeline and shrinks data freshness latency from "hour-level" to "minute-level."
Millions of connected vehicles generate billions of records of battery status, driving behavior, and fault diagnostics daily. Traditionally, this data converges via in-vehicle gateways into Kafka, requiring dedicated Spark Streaming clusters for real-time cleansing and persistence. The storage layer often utilizes a tiered architecture of "Kafka for hot data + HDFS for cold data." Not only does cross-system data movement introduce high latency, but the notorious small-file issue of HDFS perpetually plagues operations teams.
By enabling ApsaraMQ for Kafka's ingestion capability, vehicle sensor data streams flow directly into the OSS Table Bucket, stored in Parquet format and partitioned by "vehicle model + date." The automated operations mechanism of OSS Table Bucket continuously executes small file compaction and snapshot cleanup in the background, freeing the ops team from manual file governance. Simultaneously, structured data in the data lake can be directly read by Spark MLlib for battery health prediction and fault warning model training, accelerating the model iteration cycle from weeks to mere days.
During AI model training, data teams must simultaneously manage massive troves of road-test images, video clips, point cloud data, and their corresponding annotation metadata. Previously, unstructured data lived in object storage, structured metadata lived in databases, and vector retrieval relied on independent vector databases. With three types of data scattered across disparate systems, data correlation and version backtracking were exceedingly difficult.
Leveraging Alibaba Cloud OSS's comprehensive "object + vector + table" data storage architecture, raw road-test data is written to OSS Object Buckets, embeddings and retrieval indices are written to OSS Vector Buckets, while vehicle annotation information and sensor metadata collected in real time by Kafka are routed into the OSS Table Bucket via the ingestion capability. These three buckets share a unified account, permission, and audit framework. Consequently, filtering training samples and tracing versions can be executed on a single unified platform, multiplying data preparation efficiency and laying a rock-solid foundation for rapid model iteration.
Security operations teams handle over tens of billions of system access logs, API call records, and security audit events daily. Under legacy architectures, Kafka retains raw logs for the short term (usually 7 days). Data exceeding the retention period must be migrated to low-cost storage via scheduled jobs. Format conversions and field mapping during migration are highly error-prone, often rendering critical security events untraceable.
With ApsaraMQ for Kafka's ingestion capability, security teams route audit log topics directly into the OSS Table Bucket. Raw logs are archived long-term in a structured format, drastically slashing storage costs. Iceberg's Time Travel capability allows security engineers to backtrack to the data state of any historical point in time. Paired with standard SQL, they can rapidly pinpoint anomalous access paths within petabytes of logs. Even when facing sudden security incident investigations, cross-month or cross-year data retrieval can be completed in minutes, truly delivering peace of mind for security tracing with all data at your fingertips.
The evolution of Kafka is, at its core, a history of continuous decoupling and opening up of data infrastructure. In the early days, compute and storage were tightly coupled on a single physical machine, where scaling meant expanding the entire machine. Later, hot and cold data began to be tiered, with historical data gradually offloaded to low-cost storage—yet the broker remained a stateful intermediary node. Today, as object storage becomes the default data foundation, Kafka is evolving from a singular message pipeline into a dual-mode data portal capable of both "real-time consumption" and "direct analytics."
The evolution of ApsaraMQ for Kafka has always orbited a single, core proposition: How to make data flow more efficiently, economically, and openly. The launch of the native data lake ingestion capability is not just an active response to this industry trend; it is a major strategic milestone in Alibaba Cloud's deep integration of "streaming data portals + open data lake storage."
Looking ahead, Alibaba Cloud will continue to deepen the synergy among ApsaraMQ for Kafka, OSS, stream computing, and other data infrastructures. By providing enterprises with a one-stop data platform encompassing data collection, real-time processing, and lakehouse analytics, we are committed to helping our customers unlock the absolute full potential of their data in the AI era.
Stop Manually Copying Skills: A Management Solution for the Multi-Agent Era
AgentTeams and Claude Tag Both Enter Group ChatMode: Is it a New Paradigm or a New Narrative?
744 posts | 60 followers
FollowAlibaba Cloud Native Community - May 18, 2026
Alibaba EMR - October 12, 2021
Apache Flink Community China - August 12, 2022
ApsaraDB - February 29, 2024
Apache Flink Community - July 28, 2025
Apache Flink Community - March 20, 2025
744 posts | 60 followers
Follow
Data Lake Storage Solution
Build a Data Lake with Alibaba Cloud Object Storage Service (OSS) with 99.9999999999% (12 9s) availability, 99.995% SLA, and high scalability
Learn More
Data Lake Formation
An end-to-end solution to efficiently build a secure data lake
Learn More
Big Data Consulting for Data Technology Solution
Alibaba Cloud provides big data consulting services to help enterprises leverage advanced data technology.
Learn More
Big Data Consulting Services for Retail Solution
Alibaba Cloud experts provide retailers with a lightweight and customized big data consulting service to help you assess your big data maturity and plan your big data journey.
Learn MoreMore Posts by Alibaba Cloud Native Community
