This topic describes the architecture of Message Queue for Apache Kafka and the publish-subscribe model.
Message Queue for Apache Kafka architecture
A Message Queue for Apache Kafka cluster consists of producers, brokers, ZooKeeper, and groups, as shown in Figure 1.
- A producer pushes messages to Message Queue for Apache Kafka brokers. The messages sent can be page view information, server logs, and information related to system resources such as CPU utilization and memory usage.
- Kafka Broker
- A broker is a server used to store messages. Brokers can be scaled out. The more the brokers, the higher the throughput of the Message Queue for Apache Kafka cluster.
- A group pulls messages from a Message Queue for Apache Kafka broker to subscribe to topics and consume messages.
- Zookeeper manages the cluster configuration, elects the leader partition, and balances the load when a group changes.
Publish-subscribe model of Message Queue for Apache Kafka
Message Queue for Apache Kafka uses the publish-subscribe model, as shown in Figure 2.
- Groups and topics are in the many-to-many relationship. One group can subscribe to multiple topics, and one topic can be subscribed to by multiple groups.
- However, the messages of a topic to which a group is subscribed can be consumed by only one consumer in the group.