ApsaraMQ for Kafka is a fully managed, Apache Kafka-compatible event streaming service on Alibaba Cloud. It delivers write throughput up to 50 GB/s across tens of thousands of partitions with 99.99% availability -- without the operational burden of self-managed clusters.
Existing Apache Kafka producers, consumers, and Kafka Connect pipelines work without code changes. ApsaraMQ for Kafka supports Apache Kafka versions 0.10.x to 3.x with all standard clients.
At a glance
| Attribute | Details |
|---|---|
| Kafka compatibility | Apache Kafka 0.10.x to 3.x, all standard clients |
| Max write throughput | Up to 50 GB/s |
| Partition scalability | Tens of thousands of partitions with no performance degradation |
| Scaling speed | Seconds |
| Service availability | 99.99% |
| Data reliability | 99.9999999% (nine 9s) |
| Deployment options | Provisioned instances or Serverless (pay-as-you-go) |
Video introduction
Learn about Alibaba Cloud ApsaraMQ for Kafka in three minutes.
Why use a managed Kafka service?
Running Apache Kafka in production requires ongoing effort: broker provisioning, partition rebalancing, patching, monitoring, and capacity planning. As partition counts grow, open source Kafka performance degrades, adding tuning overhead.
ApsaraMQ for Kafka eliminates this work:
No cluster management -- No broker provisioning, patching, or rebalancing.
Stable performance at scale -- Full throughput across tens of thousands of partitions.
Scaling in seconds -- Add capacity instantly instead of waiting for manual broker rollouts.
99.99% availability, nine-9s data reliability -- Persistent message storage provides durability even under heavy backlog.
Pay for what you use -- Serverless instances bill based on actual usage with no cost for idle capacity.
Core concepts
| Concept | Description |
|---|---|
| Producer | A client application that writes messages to topics. |
| Topic | A named feed that organizes messages by category, similar to a folder where each message is a file. |
| Partition | A subdivision of a topic that enables parallel processing. More partitions mean higher throughput, similar to adding lanes on a highway. |
| Broker | A Kafka server that stores data and serves client requests. ApsaraMQ for Kafka manages brokers for you. |
| Consumer | A client application that reads messages from topics. |
| Consumer group | A set of consumers that cooperatively read from the same topic. Each partition is read by exactly one consumer in the group, distributing the load. |
| Offset | A position marker that tracks where a consumer has read to within a partition. |
| Connector | A component for streaming data between Kafka and external systems such as databases, storage services, and analytics platforms. |
| Instance | A managed Kafka deployment unit in Alibaba Cloud. Each instance runs an isolated Kafka cluster. |
How it works
Producers write messages to topics on your ApsaraMQ for Kafka instance.
Messages are distributed across partitions within each topic for parallel processing and persistent storage.
Consumers read messages by tracking their offset in each partition. Multiple consumer groups can read the same topic independently.
Connectors stream data between Kafka topics and external systems such as databases, object storage, and analytics platforms.
Use cases
| Scenario | Description |
|---|---|
| Log aggregation | Centralize logs from distributed applications into a single stream for monitoring, alerting, and troubleshooting. |
| Real-time analytics | Feed clickstream data, transaction events, or IoT telemetry into stream processing engines for dashboards and real-time insights. |
| Event-driven architecture | Decouple microservices with durable, ordered event streams. Each service publishes and subscribes to the events it needs. |
| Data integration | Move data between systems -- databases, data warehouses, search indexes -- using Kafka Connect pipelines. |
| IoT telemetry ingestion | Collect high-volume device data from sensors, vehicles, or industrial equipment for processing and storage. |
Capabilities
Apache Kafka compatibility
Connect with any standard Apache Kafka client (versions 0.10.x to 3.x) -- no proprietary SDK required. Existing producers, consumers, and Kafka Connect pipelines work as-is. Migrate from self-managed clusters without code changes.
High throughput at scale
An optimized storage kernel sustains full throughput across tens of thousands of partitions. This resolves a limitation of open source Apache Kafka where performance degrades at high partition counts.
| Metric | Capability |
|---|---|
| Write throughput | Up to 50 GB/s |
| Partition scale | Tens of thousands, no performance loss |
| Scaling speed | Seconds |
Availability and data durability
Messages are persistently stored with 99.99% service availability and 99.9999999% (nine 9s) data reliability.
Throughput remains stable even when large message backlogs accumulate.
Write performance is maintained during phased updates, service upgrades, and cold data reads.
Operations and observability
A dedicated operations team and automated tooling manage cluster health:
HouseKeeping diagnostics -- Scans clusters at one-minute intervals, alerts on unhealthy instances, and produces daily inspection reports.
Consumer group monitoring -- Tracks message accumulation and alerts on anomalies before they affect downstream systems.
API coverage -- Manage resources and automate operations programmatically.
Serverless deployment
Serverless instances scale automatically with pay-as-you-go billing -- no need to provision for peak capacity and no cost for idle resources.
Ecosystem integration
ApsaraMQ for Kafka connects to Alibaba Cloud services and third-party systems:
Connectors -- Stream data between Kafka topics and external systems for data integration and real-time computing.
Built-in ETL -- Fully managed extract, transform, and load with programmable functions. No separate ETL infrastructure required.
What's next
Get started -- Create an instance and publish your first message.
Benefits -- Compare ApsaraMQ for Kafka with open source Apache Kafka.
Billing -- Pricing for Provisioned and Serverless instances.