Kafka message monitoring - Kafka Eagle - Alibaba Cloud Developer Forums: Cloud Discussion Forums

Adolph
Engineer
Engineer
  • UID623
  • Fans4
  • Follows1
  • Posts72
Reads:2263Replies:0

[Share]Kafka message monitoring - Kafka Eagle

Created#
More Posted time:Nov 1, 2016 9:28 AM
1. Overview
During development, the data changes because of the consumption of messages in the Kafka cluster are what we care about. Given that the business is not complicated, we can use the command tools provided by Kafka, in combination with the ZooKeeper Client tool, to accomplish our work conveniently. With the increasing complexity of the business, the growing number of groups and topics, the command tools provided by Kafka gradually exhibit their incompetency. At this time, the Kafka monitoring system becomes especially important when we need to observe the details of the consumption application. The monitoring system industry has a lot of outstanding open-source monitoring systems. In the early stages, we used KafkaMonitor and Kafka Manager. But with the rapid development of the business and the emerging special demands of internet companies, the existing open-source monitoring system is far from enough to satisfy the demand in terms of performance, scalability, and DEVS use efficiency. Therefore, based on the needs of internet companies, user experiences and feedback from using DEVS, we combined some ideas about open-source Kafka message monitoring in the industry, and designed and developed the Kafka cluster message monitoring system: Kafka Eagle.
Kafka Eagle is used to monitor the consumption status of the topics in the Kafka cluster. It includes the generation of logs, changes in the offset, distribution of the partitions, owners, as well as the time when the topics are created and modified among other information. Kafka Eagle can be downloaded from the link below:
• [Download Kafka Eagle]
2. Content
Kafka Eagle involves the following content modules:
• Dashboard
• Topic(Create & List)
• Consumers
• Cluster Info
2.1 Dashboard
Enter http://host:port/ke in the browser to visit the Dashboard of Kafka Eagle. The page contains the following content:
• Brokers
• Topics
• Zookeepers
• Consumers
• Kafka Brokers Graph
The number of topics of the Kafka cluster, the number of consumers, the number of brokers of Kafka and the ZooKeeper cluster to which the Kafka cluster belongs to. A screenshot of the Dashboard can be found below:


2.2 Topic
The Topic module displays detailed information about the topic creation and the topic.
2.2.1 Create
Through the Create module, you can create a topic with custom partitions and number of backups, as shown in the figure below:


2.2.2 List
This module lists all the topics in the Kafka cluster, including the number of partitions of the topic, the topic creation time and modification time, as shown in the figure below:


In the figure above, every topic name corresponds to a hyperlink to the topic details. Click the hyperlink and you can view details about the topic, such as the partition index number, leader, replicas and isr, as shown in the figure below:


2.3 Consumers
This module displays information of the topic with consumption records, including the following:
• Running
• Pending
• Active Topic Graph
• Offsets Rate Graph
2.4 Cluster Info
This module displays the Kafka cluster information and the ZooKeeper cluster information, including the following:
• Kafka Broker Host & IP
• Kafka Broker Created & Modify Date
• Zookeeper Host & IP
3. Data Collection
The data source of Kafka Eagle message monitoring is the ZooKeeper. Creation, modification or consumption of Kafka messages will all be registered in ZooKeeper, so we can obtain the changes to the data, such as topics, brokers, partitions and groups among others. The structured storage of Kafka in ZooKeeper is shown in the figure below:


4. Conclusion
   Kafka Eagle is easy to install and use. You just need to download and install it, and configure the address of the ZooKeeper cluster to which the Kafka cluster belongs to. The installation and deployment documents can be found below:
• Kafka Eagle User Manual
Guest