As AI moves from demo to production, a consensus is crystallizing: model choice is no longer the bottleneck. The other pivotal variable is the data pipeline that feeds business events into the model.
Three things have shifted. Workload: Agent tasks, inference calls, and tool execution run long and erratically — ill-suited to traditional messaging's short, fast, uniform cadence. Latency: RAG and real-time recommendation need context within seconds; offline batch can't keep pace with an Agent's decision point, so context latency becomes decision quality. Trust: data now flows to ever more models, Agents, and third-party tools, and once a sensitive field enters a model's context it can't be recalled. A network edge governs who may connect — not where a sensitive field travels once legitimately read.
On August 28, at the "AI Real-Time Data Salon · Shanghai" (co-hosted by InfoQ, Alibaba Cloud, and IBM), three engineers from Alibaba Cloud and IBM traced this pipeline top-down: Agents at the business-event front line, a redefined data streaming platform, and an architecture-level rebuild of the Kafka cluster. Each observation builds on the last, converging on one conclusion — the center of gravity for AI in production is shifting from the model to the real-time data pipeline.
Minda Wu (Senior Architect, Client Engineering Team, IBM China Technology Business Unit)
The pivotal leap for AI, said Minda Wu, is from "answering questions" to "achieving goals" — from a "user asks, model answers" exchange to an event-driven mode in which a business event occurs and an Agent perceives it in real time and takes governed action. He terms this Event-Driven AI: Agents are no longer summoned by humans but embedded in enterprise infrastructure, reacting the instant conditions change. AI's emphasis shifts from observing output to driving outcomes.
Wu distilled the technical foundation into a single diagram: Confluent Intelligence's three pillars, plus a supporting layer.
● Perceive (Confluent's AI/ML Functions) embeds anomaly detection, sentiment analysis, and PII masking as stream functions, isolating the signals that warrant an Agent's decision from vast event volumes.
● Serve (the Real-Time Context Engine) materializes Kafka topics into low-latency, read-only serving tables; Agents query business facts — orders, inventory, device status — over MCP, recasting "batch-fed context" as "queryable the instant an event arrives."
● Act (Streaming Agents) keeps Agents running continuously on Kafka topics — event-triggered, invoking external tools — with Agent and data-stream lifecycles governed on a single plane.
● Beneath sits the supporting layer: model inference, real-time embedding, external-table retrieval, secure connectivity, RBAC, and audit.
To illustrate the pillars closing the loop, Wu walked through a credit-card fraud investigation.
Day 1, real-time decisioning: card transactions stream into Kafka; stream jobs run anomaly detection and stateful enrichment (catching patterns like "two locations, spending spiking within 30 minutes"); an Agent weighs rules, merchant watchlists, and the user's flag history to assess risk on the spot and trigger remediation — push alerts, card freezes, tickets.
Day 2, retrospective: a second Agent reads Day 1's decision stream over MCP, reasons across systems using human handling outcomes and user disputes, and proposes rule changes that feed back into Day 1's execution path — an event-driven rule loop with no periodic offline batch. The takeaway: to own a business responsibility in production, an Agent must stand at the moment the event unfolds — responding to a changing world, not answering a question.
Event-driven puts the Agent on the business scene; the other half is making it governable. IBM watsonx Orchestrate (IBM's agent orchestration and governance platform) covers this — observe, evaluate, troubleshoot, and manage in one place, with built-in checks for effectiveness, hallucination, and toxicity, decomposing every run into traceable steps so "tool problem or model problem is obvious at a glance." Event-driven supplies context; the control plane governs it — together, the complete picture for scaling Event-Driven AI in the enterprise.
Here Confluent Connect serves as the real-time data bus and integration hub, giving Agents a stable, secure context stream. No-code prebuilt connectors and a distributed architecture link heterogeneous systems — legacy databases, cloud storage, vector databases — in real time, keep streams uninterrupted with automatic fault recovery at scale, and span hybrid clouds over private networking to carry enterprise data safely into AI systems.
In practice, Connect cleanses and protects data in-flight. Syncing a customer-transaction database to an Agent, for instance, it masks sensitive fields — ID and card numbers — automatically in the transport path via Single Message Transforms (SMT): no ancillary cleansing service, no sensitive data exposed to the model, and a faster path from experiment to production.
Yao Liu (Head of Kafka Product, Alibaba Cloud)
Each pillar — Perceive, Serve, Act — demands more of the data streaming platform than traditional Kafka delivers. Yao Liu's take: scaling AI is redefining what qualifies as a good-enough streaming platform, along three axes — consumption model, data freshness, trust boundary. He cast this ApsaraMQ for Confluent release as "a generational platform leap, not a patch upgrade" — the three landing at once in three headline features: a cross-generation kernel, Queues for Kafka GA, and CSFLE + Alibaba Cloud KMS.
"ApsaraMQ for Confluent headline features" · cross-generation kernel / Queues for Kafka GA / CSFLE + Alibaba Cloud KMS
Traditional consumer-group partition assignment suits "event-stream" workloads — high volume, fast per-message processing, order-sensitive. The Agent era is anything but: Agent tasks, inference calls, and tool execution are long-running, uneven, task-shaped, and one slow task blocks every message behind it in the partition. This release brought Queues for Kafka (KIP-932 share groups) to GA — multiple consumers read the same partition at once, with per-message acknowledgment and redelivery, so consumer count is no longer capped by partitions. "Want order, use a consumer group; want concurrency, use a share group — both coexist on the same cluster, the same topic." One Kafka now carries event streams and task queues at once, fusing "messaging + queuing" from two architectures into one for teams rolling out Agents and inference scheduling.
RAG and real-time recommendation need context within seconds — and "seconds" isn't merely fast stream processing; metadata, control plane, and fault recovery must all keep pace. This release completed the cross-generation kernel move: KRaft is now the sole metadata plane on an Apache Kafka 4.2 kernel, ZooKeeper fully retired. The payoff: faster recovery, smoother control-plane failover, a higher metadata ceiling. One fewer distributed coordination system means one fewer failure domain and one fewer ops-and-licensing cost — for freshness-sensitive AI, the precondition for serving data in seconds.
As data flows to more models, Agents, and third-party tools, a network edge settles only who may connect — not where a sensitive field ends up. This release brought CSFLE (Client-Side Field Level Encryption) to GA, moving the encryption boundary to the client: encrypt/decrypt runs inside the SDK, applied automatically by schema field tags; data remains ciphertext across server, disk, backups, and cross-region replication, with no server-side decryption; field-level granularity encrypts sensitive fields (ID, phone numbers) individually while others stay searchable and computable. "CSFLE makes 'data can flow' and 'sensitive fields don't leak' no longer mutually exclusive — encryption travels with the data, rather than locking it behind a perimeter."
A pivotal capability for mainland China: CSFLE integrates natively with Alibaba Cloud KMS — the KEK stays resident there, keys never leave the country or the cloud, and it binds into Alibaba Cloud RAM permissions and action audit. For finance, government, and healthcare — with hard requirements on key residency and autonomous control — this is the precondition for running Confluent in the cloud.
These three shifts aren't isolated upgrades — they're the AI era's redefinition of the enterprise data streaming platform. The same release pays dividends across three customer lenses: compliance-sensitive industries (auditable for MLPS classified-protection and industry audits), high-concurrency online businesses (long-tail tasks no longer slow the batch), and teams shipping AI (data masking and concurrency elasticity at once).
Meiping Zhang (Head of Kafka Engineering, Alibaba Cloud)
For those three redefinitions to reach production, one last challenge remains: the Kafka cluster's own architecture must be rebuilt. Meiping Zhang's framing: running a "real-time ingest → compute → lakehouse" pipeline once meant stitching together Kafka + a separate compute engine + a lakehouse tool + a metadata system — every added system brings another set of connections, another state store, another failure boundary. His answer: make Kafka, "the first hop of real-time data," shoulder more — compute on arrival, land in the lake on completion — one platform converging stream, compute, and lakehouse. "Integrated" doesn't mean cramming everything into one process; it means a shared entry point and foundation, with compute and lakehouse each an independent, composable closed loop.
"Overall architecture: one copy of data, unified across stream · compute · storage" · the complete path from source, through stream · compute · lakehouse, to the AI ecosystem
The "compute" layer splits into three capabilities: stream-processing SQL (filtering, windowed aggregation, multi-stream joins, multi-way output); real-time embedding (data vectorized as it flows through Kafka, via the Tongyi DashScope model series, written to DashVector / OpenSearch); and Streaming Agents (Agents on the data stream, invoking Qwen / Model Studio in real time for content understanding, classification, routing, and continuous decisioning). Stacked in parallel, they render Wu's three pillars into concrete product form. Nor must they chain in sequence: compliance-retention teams can turn a raw topic directly into a continuously updated table; real-time-risk teams can run SQL alone; a "cleanse-aggregate-then-persist" scenario carries the SQL result topic onward into a table.
Under traditional Kafka, a node failure means shuttling historical data between replicas — slow, bandwidth-hungry failover. The team re-architected the HA path around shared storage + IO Fence: partition data persists to shared storage, replicas retain only the state needed to take over; on failover or scaling, a new node reuses the shared data directly — seconds-level switchover, no full migration. Rebalancing and data shuttling under the old architecture are precisely the hidden ceiling on elasticity amid AI traffic peaks and troughs. The community is already moving from local replica replication to tiered / shared storage (Kafka 3.9 Tiered Storage production-ready, KIP-1150 Accepted); the shared-storage foundation engineers that direction into reality.
Traditionally, even without windows or joins, writing Kafka data continuously into an open table means running an ETL job that manages offsets, schemas, transaction commits, and recovery. "Table Topic" collapses that external runtime into a managed topic-to-table relationship: enable it on the original topic in one click, and the platform takes on message-to-Iceberg mapping, schema evolution, transaction commits, and recovery; downstream reads through the Iceberg Catalog, and Spark / Flink / Trino / MaxCompute / Hologres reuse the same asset. The key shift: real-time lakehouse ingest is no longer a separate pipeline to build and maintain, but a configurable, managed capability on the topic.
Lakehouse ingest carries an easily underestimated long-term cost: small files, snapshots, and metadata keep accruing. The chores — small-file compaction, snapshot expiry, unreferenced-file cleanup — run automatically in the service backend, with no standing job for the customer. Shifting governance cost from user jobs into the storage kernel is the hidden threshold for whether lakehouse ingest endures.
One unified entry point lets users develop, deploy, and operate real-time jobs in SQL. These jobs run on independent compute resources and runtime, decoupled from the Kafka broker's core messaging; compute-side pressure — complex joins, aggregations, large-state computation, backpressure — stays confined to the stream-processing domain, never touching messaging stability. Successive SQL-engine optimizations for single-node and scale efficiency surface where users feel them: higher real-time throughput, markedly lower scan latency at large state, steadier recovery — stateful operators and high-traffic scenarios leap from "state bounded by local memory" to "state decoupled from compute capacity." In Zhang's words: "The first two layers push single-node performance to the limit; the last two decide whether the system, under large state and high traffic, can store it, run it fast, scale it, and recover it steadily."
Taken together — foundation, lakehouse ingest, stream processing — the barrier isn't the feature mix but four engineering loops holding at once: HA correctness (failover loses nothing, reorders nothing, moves no data), Table assets (offsets and snapshots advance in lockstep), stream-processing isolation (compute faults never disturb the messaging path), and AI governance (model calls decoupled from the messaging hot path). For customers, one copy of real-time data no longer needs a pipeline per use, and integration doesn't come at the cost of clean runtime boundaries. "The feature list is copyable; correctness, isolation, and the long-term operational loop are not."
Across the three observations — Event-Driven AI's paradigm demands, the platform's three redefinitions, the cluster's architecture-level rebuild — a broader trend emerges: the distance from demo to production increasingly hinges on data-side engineering, not model-side selection. Judging an AI team's maturity now means looking past its grasp of models to its command of the data pipeline — context freshness, multi-source fusion, and a unified stream-and-batch architecture. These capabilities once lay scattered across data engineering, message middleware, and stream processing; scaling AI is now converging them onto one platform, toward one goal — keeping every Agent decision grounded in the latest business state. The engineering practice around this pipeline is taking shape fast.
Compiled from on-stage talks at the "AI Real-Time Data Salon · Shanghai" on August 28, 2026 (co-hosted by InfoQ (Geek Media), Alibaba Cloud, and IBM); views belong to the speakers. Speakers: Minda Wu (Senior Architect, Client Engineering Team, IBM China Technology Business Unit), Yao Liu (Head of Kafka Product, Alibaba Cloud), Meiping Zhang (Head of Kafka Engineering, Alibaba Cloud).
Takeover in Seconds: Rethinking Apache RocketMQ Cloud High Availability via Protocol-Level Fencing
HiRoute Smart Routing in Action: GPT-6 Astra × Qwen-3.8 Flash, Same Quality at 90% Lower Cost
784 posts | 60 followers
FollowAlibaba Cloud Native Community - November 27, 2025
ApsaraDB - December 29, 2025
Neel_Shah - May 22, 2026
Alibaba Cloud Native Community - September 8, 2025
Alibaba Cloud Big Data and AI - June 3, 2026
Amuthan Nallathambi - October 14, 2024
784 posts | 60 followers
Follow
Message Queue for Apache Kafka
A fully-managed Apache Kafka service to help you quickly build data pipelines for your big data analytics.
Learn More
Realtime Compute for Apache Flink
Realtime Compute for Apache Flink offers a highly integrated platform for real-time data processing, which optimizes the computing of Apache Flink.
Learn More
AgentCore
AgentCore unifies agent construction, governance, and execution capabilities so enterprises can deploy diverse agents with confidence.
Learn More
ApsaraMQ for RocketMQ
ApsaraMQ for RocketMQ is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.
Learn MoreMore Posts by Alibaba Cloud Native Community
