By Hongjian Fei
Adapted from the Apache 2026 talk “Real-Time Lakehouse Pipelines: Native Lake Ingestion and Embedded Compute in Disaggregated Kafka.”
Disaggregating storage and compute changes Kafka in obvious ways: brokers get lighter, storage scales independently, and capacity and compute each scale elastically on their own.
The deeper change is this: Kafka’s standard protocol and real-time data plane stay intact, while long-term persistence is no longer tightly bound to Broker local disks. Once the data foundation decouples storage from compute, and as open table formats, Catalogs, and managed runtimes mature, the same topic data can natively flow into two different value paths:
• One is Table: continuously arriving messages settle into Iceberg open tables, turning real-time data into an asset many engines can reuse for the long term.
• The other is SQL: messages are filtered, windowed, aggregated, and routed the moment they arrive, continuously producing business-facing real-time results.
But “native” does not mean the work is skipped. The Table side still handles record mapping, schema evolution, CDC, atomic commits, exactly-once, and small-file governance. The SQL side still needs its own runtime, resource isolation, failure recovery, and job-level observability. What actually changes is who carries these capabilities long term, and how large an external responsibility boundary users face.
This article uses ApsaraMQ for Kafka's topic message lake ingestion and built-in stream computing as examples to answer three questions:
In many real-time workloads, the same event carries two kinds of value at once:
• The first value is in the present. An order updates metrics the moment it is created. A risk event is judged on arrival. Logs enter Kafka and trigger alerts through real-time aggregation. This path prizes low latency, continuous compute, and fast results.
• The second value settles into the future. Data must land in open tables for offline analysis, feature engineering, model training, historical lookback, audit, and cross-engine queries. This path prizes open formats, long-term readability, and stable governance.
The platform therefore needs two paths that are independent of each other yet share one data foundation:
• The TABLE path turns messages into open tables continuously, forming long-lived reusable data assets;
• The SQL path puts events to work the moment they arrive (filtering, windowing, aggregating, routing) to produce real-time results continuously.
This is not about cramming two kinds of work into one runtime; it is about giving them a common data starting point with clear responsibility boundaries on each side. Table cares whether “this table stays correct for the long run.” SQL cares whether “this computation runs continuously, independently, and observably.”
In the past, lake ingestion usually existed as a standalone data pipeline: provision a compute cluster, deploy Flink or Spark, maintain connectors, then wire in object storage and a Catalog. Flexible, yes, but users end up maintaining the relationships among the messaging system, the compute runtime, the table format, and the storage system all at once.
Turning continuous lake ingestion from “a pipeline users build themselves” into “a relationship the platform provides” requires at least three preconditions.
First, Kafka storage and compute are decoupled. Messages are still written and consumed over the familiar Kafka protocol, but persistence no longer clings to a single Broker’s local disk. Once underlying storage scales independently, the same data is easier to organize into different forms.
Second, open formats come first. Data must not land as a pile of isolated files. An open table format like Iceberg provides table-level semantics (Snapshot, Manifest, schema, and partition evolution), while a Catalog lets many compute engines discover and read those tables consistently.
If these concepts are new, think of them as a “table version index chain.” The Catalog discovers tables, manages namespaces, and holds the entry point to table metadata. Table metadata points to the current Snapshot, a complete, visible table version at a moment in time. The Snapshot organizes, via Manifests, the DataFiles, DeleteFiles, and statistics that version references: DataFiles hold row data, while DeleteFiles record which rows should be invalidated. A query engine walking this chain reads a versioned, schema-defined table, not a pile of objects it must interpret itself.
Third, the runtime goes serverless. What users want to express is “keep writing this topic to this table,” not “watch over an extra cluster for me forever.” Resource scheduling, failure recovery, scaling, and version maintenance should be absorbed by the platform as much as possible.
Together these three conditions change the shape of lake ingestion: it need not always be an external compute job; it can also be a managed configuration between a Kafka topic and an open table.
If you look only at the first write, continuous lake ingestion is easy to misread as “messages to files.” But a long-running table carries at least five classes of responsibility.
• Small-file governance: Kafka data arrives continuously by partition, batch, and time. Batches too small, partitions too many, or commits too frequent all manufacture large numbers of small files, amplifying the cost of metadata reads, query planning, storage requests, and background compaction.
• Schema evolution: business fields get added; required fields may be relaxed to optional; numeric precision may widen. The ingestion path must judge which changes are compatible, how to update the table structure, and how to block and alert when they are not.
• CDC reconstruction: for database change logs, insert, update, and delete are not three ordinary message types; they describe the current state of a table. The ingestion system must translate event semantics into DataFiles and DeleteFiles, and reconstruct a consistent view at read time.
• Atomic commits: uploaded files do not mean a new version is live. Only when a set of DataFiles, DeleteFiles, and metadata commits atomically as one Snapshot can downstream readers see a complete new version.
• Failure recovery: a job may crash after writing files but before committing, or fail at a progress checkpoint. Recovery must allow safe replay while ensuring the same batch never takes logical effect twice.
None of these five disappears because the capability is native; what changes is that they move from user-visible external jobs into the platform, absorbed uniformly by it.
Common approaches to Kafka lake ingestion fall into roughly three classes.
• The connector path sits close to the messaging system. Deployment is direct, suiting teams with an existing Connect setup and relatively clear transformation logic. Users typically still watch the connector cluster, job concurrency, version compatibility, and table commit behavior.
• The ecosystem compute-platform path is the most flexible. Engines like Flink and Spark carry complex cleansing, joins, stateful compute, and multi-sink writes, with mature engineering ecosystems. In exchange, users operate an extra runtime and must understand how compute checkpoints, sink commits, and table Snapshots relate.
• The native lake ingestion path targets a narrower need. When the job is mainly to persist Kafka data into open tables continuously and reliably, having the platform manage format conversion, schema awareness, transactional commits, recovery, and table maintenance can markedly shrink the system boundary users must watch.
No path is absolutely better: complex joins, custom operators, and multi-path processing still suit professional compute platforms, while standardized continuous ingestion is better handled as a built-in product capability. When choosing, the key question is not “whose pipeline diagram is shorter.” It is “which part of the complexity the team wants to own long term.”
In the deck’s illustrative pipeline, the traditional external approach exposes 7 user-visible nodes: Kafka, record mapping, schema, file layout, commit, recovery, and Iceberg plus object storage. Any upgrade, transient fault, or config change can propagate across systems.
In native mode, 3 user-visible nodes remain: Kafka topic, Iceberg table, and object storage. Record mapping, schema, file layout, commit, and recovery still exist; they simply move inside the platform.
In current product terms, users enable a message lake ingestion configuration on an existing topic, writing data continuously into an Iceberg table in an OSS Table Bucket. This does not copy out a new Kafka topic; it establishes a managed table relationship on the original data source.
The most important value of this change is not “fewer boxes in the diagram” but a smaller fault domain:
• No separate external compute cluster maintained for ingestion alone;
• No repeated task-state synchronization across multiple control planes;
• Table commits and consumption progress coordinated within one managed path;
• Schema, compaction, and Snapshot cleanup no longer reimplemented per job.
In short, native lake ingestion does not eliminate the processing; it changes who owns the processing.
Zoom in on the pipeline: a batch of Kafka messages typically passes through the following steps between being read and becoming part of an open table.
First, the system reads a definite offset range from one or more partitions, records the input boundary, and then maps message keys, values, headers, or timestamps to table fields per configuration.
Next comes schema awareness. Suppose the business widens amount from DECIMAL(10,2) to DECIMAL(12,2), relaxes country from required to optional, and adds an optional field channel STRING. These changes share one property: old data can still be validly interpreted under the new schema, so the table structure can be updated within supported evolution rules. Conversely, if a new type cannot accommodate old data, the system must not guess silently; it should block the commit and surface the error.
Then, the system generates candidate DataFiles and DeleteFiles per append or CDC semantics and builds new Manifest and Snapshot metadata; files may already be in storage, but they remain invisible to readers.
Finally comes the commit: the metadata location in the Catalog is conditionally updated, switching from metadata N to metadata N+1, which references the new Snapshot. Only after the update is confirmed does the whole batch become visible at once. If a conditional conflict occurs, the metadata location still points to the old version, and the commit must be rebuilt on the latest metadata. If the commit outcome is unknown, the system must first refresh current metadata to confirm whether the previous commit succeeded, then decide the next step. The two cases must not be conflated.
This is a crucial boundary in open tables: files are the physical carrier; the Snapshot is the logical version.
The CDC scenario further shows why lake ingestion cannot be mere format conversion.
Suppose the source table identifies rows by primary key id:
• An insert adds a record and can be written to a DataFile;
• A delete requires an equality delete against that key;
• An update can be read as “delete the row for the old key, then write the new version.”
The equality delete here means “a logical delete matched on one or more equality fields.” In the implementation discussed here, it does not modify old DataFiles in place but writes the invalidated primary-key values into a DeleteFile. At read time the engine merges DeleteFiles with DataFiles and filters out matched old records, so one update appears in the table as “old value invalidated plus new value written.” Readers see the current view, not a simple stack of change events.
At read time, the compute engine does not simply concatenate all DataFiles but combines DeleteFiles to filter invalidated records, yielding the logical view of the current Snapshot.
Two easily overlooked preconditions sit behind this. First, CDC requires a stable, identifiable primary key; otherwise the system cannot express precisely “which row was deleted.” Second, source event ordering and duplicate-delivery semantics must be clear; otherwise multiple updates to one key may be reconstructed incorrectly.
Hence a CDC ingestion configuration may expose only a few options on the surface, yet underneath it ties together message ordering, primary-key semantics, file organization, and table version management.
In distributed systems, failure can occur at any boundary. A typical scenario: files are fully written, the Snapshot is not yet committed, and the job crashes.
After recovery, if the system cannot confirm whether the old files took effect, the safest choice is often to re-read and rewrite the batch. Physical storage may then hold File-Set-A and File-Set-B in sequence.
The example in the figure starts at offsets 100–199 of orders-0. The initial state is Snapshot 41 with confirmed progress 99. The first attempt writes File-Set-A but fails before commit. After recovery, the same batch is replayed and produces File-Set-B. Only when Snapshot 42 and consumption progress 199 are confirmed consistently do those 100 events become visible downstream. The next batch starts at offset 200.
So the precise meaning of exactly-once is: even if a batch of input goes through retries, it takes effect in the table’s logical version exactly once.
It does not promise that underlying storage ever holds only one physical copy, since files unreferenced by any valid Snapshot can be reclaimed by background orphan-file cleanup.
Delivering these semantics takes more than Iceberg’s atomic Snapshots: input progress and table version activation must be coordinated. In the current message lake ingestion feature, consumption progress is stored in Kafka leader metadata, with the managed path handling post-failure resumption and deduplication. The exact capability boundary should follow the product documentation at release time.
A continuously written table can become harder to use over time even when its semantics are perfectly correct.
Suppose data comes from multiple Kafka partitions, each committing files on a very short cycle. A day later, the table may hold large numbers of files of only a few hundred KB or a few MB. Query engines open more files and read more metadata; background systems incur more request and scheduling cost.
Long-term operation therefore needs three classes of maintenance:
• Compaction: merge small files into read-friendly large files and optimize data layout;
• Snapshot expiration: drop historical version references beyond the retention policy;
• Orphan-file cleanup: reclaim stray files left unreferenced by any valid Snapshot after failed commits or retries.
Table maintenance and query pruning solve two related but different problems: the former controls long-term fragmentation and metadata size, while the latter uses the current table’s partitions and statistics to cut the files a given query must scan. The figure uses a standalone log query as an example: records from 10:00–11:00 on January 1, 2024 with level=‘ERROR’ and status_code >= 500. Candidate files pass partition, Manifest, and file-level statistics filters in turn, shrinking from 2,380 to 940, to 420, and finally to 189. These numbers are not a before/after comparison of compaction.
They illustrate the mechanism rather than a fixed performance gain for any real workload, since actual gains depend on data distribution, file sizes, partitioning strategy, statistics quality, and query predicates. What matters is that an open table does not “list every file and scan them all”; instead, multi-layer metadata lets engines read as few files and do as little I/O as possible.
Once data enters the lakehouse, it is not yet an open asset if only the writing system can read it.
Full openness includes at least three layers:
• Format openness: data files and table metadata follow open specifications such as Iceberg;
• Catalog openness: engines can discover namespaces, tables, and Snapshots through standardized entry points;
• Consumption openness: Spark, Flink, Trino, PyIceberg, and cloud analytics engines can reuse the same table data for their own scenarios.
OSS Tables provides an API endpoint compatible with the Apache Iceberg REST Catalog; this compatibility layer prioritizes behavioral consistency with AWS S3 Tables. To consuming engines, the Catalog first returns the table’s metadata location, and the engine then resolves Snapshot, Manifest List, and Manifest down to the DataFiles and DeleteFiles.
This layered design yields one long-term benefit: the write path need not decide every future compute style up front. Trino can serve interactive queries today, Spark batch processing tomorrow, and feature engineering later. Compute engines may change; the data asset need not move with them.
Of course, “compatible interface” does not mean every extension behaves identically. Support matrices across engine versions, Catalog interfaces, and Iceberg features should follow the current official compatibility documentation.
The TABLE path solves the long-term persistence problem, but many teams are unwilling to wait for data to become a table before computing.
For example: one-minute window aggregation over paid orders per user, with results written continuously back to a result topic; or real-time filtering of abnormal logs, field extraction, and conditional routing to different downstream systems. These needs focus on immediate processing at arrival, and suit the SQL path.
Conceptually, the logic looks like this:
SELECT
user_id,
TUMBLE_START(event_time, INTERVAL '1' MINUTE) AS window_start,
SUM(amount) AS paid_amount
FROM orders
WHERE status = 'PAID'
GROUP BY
user_id,The point of this SQL lies less in its exact syntax than in the responsibility boundary: Kafka provides continuous input and output, while the SQL runtime owns window state, compute resources, failure recovery, and observability. Each job deploys independently, so one job’s resource fluctuations do not directly affect others.
Table and SQL are therefore not substitutes:
• If you only need to reliably persist raw messages into open tables, prefer message lake ingestion;
• If you need arrival-time filtering, windowing, aggregation, and routing, use stream computing;
• If you need complex joins, custom operators, cross-system orchestration, or mature job governance, an external ecosystem compute platform such as Flink remains an option;
• If you need both real-time results and long-term assets from the same data, the two paths can run in parallel.
This also avoids a common misconception: forcing one runtime to carry every job so the architecture looks “unified.” True unification belongs at the data and product entry points, not at the cost of the isolation different workloads deserve.
When you face a new Kafka lake ingestion requirement, four questions can draw the boundary first.
First: does the data only need to land in tables continuously, or does it also need complex compute? If it is only field mapping, schema evolution, and CDC reconstruction, native capabilities reduce external runtimes. If it includes complex joins, custom UDFs, or multi-stream state, a professional compute engine fits better.
Second: who owns long-term table health? Confirming that the first batch wrote successfully is not enough. Clarify who runs small-file compaction, Snapshot expiration, and orphan-file cleanup, and how those are observed.
Third: what is the correctness boundary after failure? Clarify how input progress, file writes, and Snapshot commits correlate. If downstream performs second-stage writes, end-to-end exactly-once also depends on the target system’s transactional or idempotent capabilities.
Fourth: which engines will consume the data going forward? Catalog interfaces, Iceberg versions, and specific features must be aligned item by item against the support matrices of Spark, Flink, Trino, or other engines.
These four questions can turn a “product selection” into a verifiable responsibility split.
Back to the main thread: storage-compute separation gives Kafka a common data foundation for Table and SQL. What this article focuses on is how Kafka messages continuously become a trusted, open, healthy table.
The value of native lake ingestion is not deleting necessary steps from the architecture. It is pulling schema, CDC, atomic commits, failure recovery, and background maintenance inside a managed boundary, converging the external nodes users must watch over the long term from a complex pipeline into a steadier topic–table relationship.
At the same time, real-time SQL offers the same data a second value path: events feed into computation the moment they arrive, continuously producing business-facing real-time results.
This article only outlines real-time SQL’s overall role and boundaries; its runtime mechanics, time semantics, state management, and consistency practices will be unpacked in follow-up articles.
The whole architecture compresses into two sentences: TABLE turns messages into open assets; SQL turns events into real-time results.
Both share one Kafka data starting point, each keeping clear runtime and responsibility boundaries. Data creates value in the present, and different engines can reuse it again and again in the future.
One caveat: as of this writing (August 2026), the open regions, instance versions, supported syntax, and public-preview status of topic message lake ingestion and stream computing are still evolving. Current documentation marks message lake ingestion as a public-preview capability with requirements on serverless instance versions and regions; stream computing also carries explicit limits on regions, SQL capabilities, and end-to-end consistency. Before formal activation or release, follow the console and the latest official documentation.
• Message lake ingestion: continuously write Kafka topic data into OSS Table Bucket
• Iceberg REST API compatibility notes
• ApsaraMQ for Kafka stream computing capabilities
• Stream computing architecture and high availability
• Stream computing quick start and usage limits
The next hop for this real-time pipeline is often consumption by AI agents. Whether agents can survive in production depends largely on whether upstream data is real-time and trustworthy. If context is assembled at T+1, timing-sensitive tasks lose accuracy repeatedly.
On the afternoon of August 28, an in-person meetup at Shanghai’s Xuhui Riverside discusses exactly one thing: how to keep feeding models and agents with real-time data in a low-latency, highly reliable, and elastic way.
Three frontline engineers walk the data pipeline in order: how real-time streams become model-consumable context, which capabilities a data pipeline must add to be AI-ready, and whether stream ingestion, stream computing, and lake ingestion can converge into one platform.
If you are pushing AI applications, agents, or RAG into production, join us in person.
The Player Says 'My Top-Up Never Arrived.' The Log Only Says BAG_FULL.
795 posts | 60 followers
FollowAlibaba Cloud Native Community - September 20, 2026
Alibaba Cloud Native Community - May 25, 2026
Alibaba Cloud Native - July 18, 2024
Alibaba Cloud Native Community - June 4, 2026
Alibaba Cloud Native - June 7, 2024
Alibaba Cloud Serverless - July 5, 2023
795 posts | 60 followers
Follow
Message Queue for Apache Kafka
A fully-managed Apache Kafka service to help you quickly build data pipelines for your big data analytics.
Learn More
ApsaraMQ for RocketMQ
ApsaraMQ for RocketMQ is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.
Learn More
Cloud-Native Applications Management Solution
Accelerate and secure the development, deployment, and management of containerized applications cost-effectively.
Learn More
Realtime Compute for Apache Flink
Realtime Compute for Apache Flink offers a highly integrated platform for real-time data processing, which optimizes the computing of Apache Flink.
Learn MoreMore Posts by Alibaba Cloud Native Community