ApsaraMQ for Kafka:Service architecture

Cluster architecture

An ApsaraMQ for Kafka cluster consists of four components: producers, brokers, ZooKeeper, and consumer groups.

• Producer

  • A producer publishes messages to topics in the cluster. Messages typically carry application data such as page view events, server logs, and system metrics like CPU utilization and memory usage.

• Broker

  • A broker is a server that persists and serves messages, similar to a node in a distributed database. Brokers scale horizontally: adding brokers increases cluster throughput.

• Consumer group

  • A consumer group (shown as Group in the ApsaraMQ for Kafka console) pulls messages from brokers to consume data from subscribed topics.

• ZooKeeper

  • ZooKeeper coordinates the cluster: it manages cluster configuration, elects the leader partition, and rebalances load when consumer group membership changes.

Publish-subscribe model

ApsaraMQ for Kafka uses the publish-subscribe model. Producers publish messages to topics, and consumer groups subscribe to those topics to consume messages.

Topics and consumer groups have a many-to-many relationship:

• A consumer group can subscribe to multiple topics.

• A topic can be consumed by multiple consumer groups simultaneously.

Within a consumer group, each message from a subscribed topic is delivered to exactly one consumer in the group.