Alibaba Cloud Managed Service for Prometheus allows you to install and configure Kafka exporters for self-managed Kafka instances. Managed Service for Prometheus also provides out-of-the-box dashboards that you can use to monitor your applications.
Background information
Alibaba Cloud Managed Service for Prometheus allows you to install and configure Kafka exporters for self-managed Kafka clusters deployed in Kubernetes clusters or on Elastic Compute Service (ECS) instances. The following types of Kubernetes clusters are supported: Alibaba Cloud Container Service for Kubernetes (ACK), serverless Kubernetes (ASK), and registered Kubernetes clusters. Managed Service for Prometheus provides Kafka exporters of the following editions:
- Kafka (Basic Edition): Kafka (Basic Edition) exporters collect basic metrics such as the number of brokers, partitions of topics, and message group latency. You do not need to configure or restart the Kafka server. For more information about how to connect to Kafka (Basic Edition) exporters, see the Add a Kafka (Basic Edition) exporter section of this topic.
- Kafka (Advanced Edition): Kafka (Advanced Edition) exporters have the capabilities of Kafka (Basic Edition) exporters and can collect important metrics of producers, brokers, consumers, and their internal modules based on a JMX agent. You can monitor the entire process of Kafka messages from the perspective of an expert by using the metrics. You need to manually deploy the JMX agent and restart the Kafka broker process. For more information about how to connect to Kafka (Advanced Edition) exporters, see the Add a Kafka (Advanced Edition) exporter section of this topic.
Go to the Integration Center page of Managed Service for Prometheus
- Log on to the ARMS console.
- In the left-side navigation pane, choose .
- Click the name of the Prometheus instance instance that you want to manage to go to the Integration Center page.
Go to the Integration Center page of ARMS
- Log on to the Application Real-Time Monitoring Service (ARMS) console.
- In the left-side navigation pane, click Integration Center. On the Integration Center page, click Add on the Kafka(Basic Edition) or Kafka(Advanced Version) card in the Application Components section. In the panel that appears, configure the required parameters to add a Kafka exporter.
Add a Kafka (Basic Edition) exporter
In this example, a Kafka (Basic Edition) exporter is added on the Integration Center page of the ARMS console.
Step 1: Add a Kafka (Basic Edition) exporter
- Log on to the ARMS console.
- In the left-side navigation pane, click Integration Center. On the Integration Center page, click Add on the Kafka(Basic Edition) card in the Application Components section. In the panel that appears, perform the following steps to add a Kafka (Basic Edition) exporter:
Step 2: View the status of the Kafka (Basic Edition) exporter
- Log on to the ARMS console.
- In the left-side navigation pane, choose .
- Click the name of the Prometheus instance instance that you want to manage to go to the Integration Center page.
- Click the Kafka(Basic Edition) card in the Installed section. In the panel that appears, you can view information on the following tabs:
- Targets: You can view the list of targets discovered by default jobs on the Targets tab.
- Metrics: You can view information about metrics and discard metrics on the Metrics tab. For more information, see Configure metrics.
- Dashboards: You can click a dashboard thumbnail to view the Grafana dashboard on the Dashboards tab. The following figure shows the information that is displayed on the Kafka (Basic Edition) dashboard.
- The number of Kafka brokers.
- The number of partitions in each topic.
- The numbers of inbound messages, outbound messages, and accumulated messages in each topic.
- The number of in-sync replicas (ISRs) in each topic.
- Alerts: You can view Prometheus alerts on the Alerts tab. The following figure shows the key metrics of Prometheus alerts. You can add alert rules based on your business requirements. For more information about how to create a Prometheus alert rule, see Create an alert rule for a Prometheus instance.
- Service Discovery Configurations: You can manage the default basic monitoring jobs on the Service Discovery Configurations tab. For example, you can modify data extraction intervals or disable collection jobs.
- Exporter: You can view the version number of
node-exporter
that is installed by default on the Exporter tab.
Add a Kafka (Advanced Edition) exporter
In this example, a Kafka (Advanced Edition) exporter is added on the Integration Center page of the ARMS console.
Step 1: Add a Kafka (Advanced Edition) exporter
- Log on to the ARMS console.
- In the left-side navigation pane, click Integration Center. On the Integration Center page, click Add on the Kafka(Advanced Version) card in the Application Components section. In the panel that appears, perform the following steps to add a Kafka (Advanced Version) exporter:
Step 2: View the status of the Kafka (Advanced Version) exporter
- Log on to the ARMS console.
- In the left-side navigation pane, choose .
- Click the name of the Prometheus instance instance that you want to manage to go to the Integration Center page.
- Click the Kafka(Advanced Version) card in the Installed section. In the panel that appears, you can view information on the following tabs:
- Targets: You can view the list of targets discovered by default jobs on the Targets tab.
- Metrics: You can view information about metrics and discard metrics on the Metrics tab. For more information, see Configure metrics.
- Dashboards: You can click a dashboard thumbnail to view the Grafana dashboard on the Dashboards tab. The Kafka (Advanced Edition) exporter provides dashboards based on instances and topics.
- Instance dashboardThe following metrics of Kafka brokers are displayed on the dashboard:
- Core metrics: the numbers of brokers, offline partitions, under-replicated partitions, and controllers, and information about CPUs and networks
- Java Virtual Machine (JVM) metrics: the key information about the JVM memory and garbage collection (GC)
- Partition metrics: the partition information, such as the partition quantity, ISR, unclean leader election, replica lag, offline partitions, and under-replicated partitions
- Time metrics: the time metrics in the Produce, Request, and Fetch phases
- Cluster traffic metrics: the overall traffic metrics of the cluster
- Broker traffic metrics: broker-specific traffic details
- Topic dashboardThe following end-to-end metrics of each Kafka topic are displayed on the dashboard:
- Producer: the key metrics of the producer, including the message sending rate, message compression ratio, and message sending latency
- Server (Kafka broker): the number of partitions in a topic, and the rates and traffic volumes of inbound messages and outbound messages
- Consumer: the message consumption rate, message consumption latency, and rebalance information
- Instance dashboard
- Alerts: You can view Prometheus alerts on the Alerts tab. Managed Service for Prometheus provides alert metrics for self-managed Kafka producers, instances, and consumers.
- Producer: Three alert metrics are provided, including the message sending failure rate, message sending duration, and message sending retry rate. You can use the metrics to identify exceptions on the producer.
- Instance: Thirteen alert metrics are provided, including topics with excessive partitions, offline partitions, unclean leader election, under-replicated partitions, a decrease in effective brokers, the number of effective controllers, the number of rejected messages, numbers of inbound messages and outbound messages of the instance, and numbers of inbound messages and outbound messages of topics. You can use the metrics to identify exceptions on the broker.
- Consumer: The alert metric for message accumulation is provided. You can use this metric to identify exceptions on consumption.
You can also create alert rules based on your business requirements. For more information about how to create a Prometheus alert rule, see Create an alert rule for a Prometheus instance.
- Service Discovery Configurations: You can manage the default basic monitoring jobs on the Service Discovery Configurations tab. For example, you can change data extraction intervals or disable collection jobs.
- Exporter: You can view the version number of
node-exporter
that is installed by default on the Exporter tab.