Alibaba Cloud Managed Service for Prometheus allows you to install and configure Kafka exporters for self-managed Kafka instances. Managed Service for Prometheus also provides out-of-the-box dashboards that you can use to monitor your applications.

Background information

Alibaba Cloud Managed Service for Prometheus allows you to install and configure Kafka exporters for self-managed Kafka clusters deployed in Kubernetes clusters or on Elastic Compute Service (ECS) instances. The following types of Kubernetes clusters are supported: Alibaba Cloud Container Service for Kubernetes (ACK), serverless Kubernetes (ASK), and registered Kubernetes clusters. Managed Service for Prometheus provides Kafka exporters of the following editions:
  • Kafka (Basic Edition): Kafka (Basic Edition) exporters collect basic metrics such as the number of brokers, partitions of topics, and message group latency. You do not need to configure or restart the Kafka server. For more information about how to connect to Kafka (Basic Edition) exporters, see the Add a Kafka (Basic Edition) exporter section of this topic.
  • Kafka (Advanced Edition): Kafka (Advanced Edition) exporters have the capabilities of Kafka (Basic Edition) exporters and can collect important metrics of producers, brokers, consumers, and their internal modules based on a JMX agent. You can monitor the entire process of Kafka messages from the perspective of an expert by using the metrics. You need to manually deploy the JMX agent and restart the Kafka broker process. For more information about how to connect to Kafka (Advanced Edition) exporters, see the Add a Kafka (Advanced Edition) exporter section of this topic.

Go to the Integration Center page of Managed Service for Prometheus

  1. Log on to the ARMS console.
  2. In the left-side navigation pane, choose Prometheus Service > Prometheus Instances.
  3. Click the name of the Prometheus instance instance that you want to manage to go to the Integration Center page.

Go to the Integration Center page of ARMS

  1. Log on to the Application Real-Time Monitoring Service (ARMS) console.
  2. In the left-side navigation pane, click Integration Center. On the Integration Center page, click Add on the Kafka(Basic Edition) or Kafka(Advanced Version) card in the Application Components section. In the panel that appears, configure the required parameters to add a Kafka exporter.

Add a Kafka (Basic Edition) exporter

In this example, a Kafka (Basic Edition) exporter is added on the Integration Center page of the ARMS console.

Step 1: Add a Kafka (Basic Edition) exporter

  1. Log on to the ARMS console.
  2. In the left-side navigation pane, click Integration Center. On the Integration Center page, click Add on the Kafka(Basic Edition) card in the Application Components section. In the panel that appears, perform the following steps to add a Kafka (Basic Edition) exporter:
    1. In the STEP1 section, select the environment in which you want to deploy the Kafka exporter.
    2. In the STEP2 section, select the Prometheus instance on which you want the Kafka exporter to reside.
    3. On the configuration tab in the STEP3 section, configure the parameters and click OK.
      ParameterDescription
      Exporter NameThe unique name of the Kafka exporter.
      kafka addressThe endpoint of the self-managed Kafka broker. Separate multiple broker addresses with commas (,) or semicolons (;).
      • If your Kafka instance is deployed in a container service environment, you can enter the IP address or service address of the Kafka broker.
      • If your Kafka instance is deployed in an ECS environment, you can enter the IP address or domain name system (DNS) address of the Kafka broker.
      Metrics space interval (seconds)The interval at which you want the service to collect monitoring data.
      kafka versionThe version number of the Kafka broker. The latest version is V3.2.0.
      SASL enabledSpecifies whether to enable the Simple Authentication and Security Layer (SASL) feature on the Kafka broker.
      SASL usernameThe username of the SASL account. If you enable the SASL feature, you must specify the parameter.
      SASL passwordThe password of the SASL account. If you enable the SASL feature, you must specify the parameter.
      SASL mechanismThe SASL mechanism. The following authentication mechanisms are supported: PLAIN, SCRAM-SHA-512, and SCRAM-SHA-256.
      TLS enabledSpecifies whether to enable the Transport Layer Security (TLS) feature on the Kafka broker.
      insecure skip TLS verifySpecifies whether to skip TLS security verification. If TLS is enabled on the Kafka broker and a self-signed certificate is used, you can skip the TLS security verification.

Step 2: View the status of the Kafka (Basic Edition) exporter

  1. Log on to the ARMS console.
  2. In the left-side navigation pane, choose Prometheus Service > Prometheus Instances.
  3. Click the name of the Prometheus instance instance that you want to manage to go to the Integration Center page.
  4. Click the Kafka(Basic Edition) card in the Installed section. In the panel that appears, you can view information on the following tabs:
    • Targets: You can view the list of targets discovered by default jobs on the Targets tab.
    • Metrics: You can view information about metrics and discard metrics on the Metrics tab. For more information, see Configure metrics.
    • Dashboards: You can click a dashboard thumbnail to view the Grafana dashboard on the Dashboards tab.
      The following figure shows the information that is displayed on the Kafka (Basic Edition) dashboard.vry
      • The number of Kafka brokers.
      • The number of partitions in each topic.
      • The numbers of inbound messages, outbound messages, and accumulated messages in each topic.
      • The number of in-sync replicas (ISRs) in each topic.
    • Alerts: You can view Prometheus alerts on the Alerts tab. The following figure shows the key metrics of Prometheus alerts. You can add alert rules based on your business requirements. For more information about how to create a Prometheus alert rule, see Create an alert rule for a Prometheus instance.
    • Service Discovery Configurations: You can manage the default basic monitoring jobs on the Service Discovery Configurations tab. For example, you can modify data extraction intervals or disable collection jobs.
    • Exporter: You can view the version number of node-exporter that is installed by default on the Exporter tab.

Add a Kafka (Advanced Edition) exporter

In this example, a Kafka (Advanced Edition) exporter is added on the Integration Center page of the ARMS console.

Step 1: Add a Kafka (Advanced Edition) exporter

  1. Log on to the ARMS console.
  2. In the left-side navigation pane, click Integration Center. On the Integration Center page, click Add on the Kafka(Advanced Version) card in the Application Components section. In the panel that appears, perform the following steps to add a Kafka (Advanced Version) exporter:
    1. In the STEP1 section, select the environment in which you want to deploy the Kafka exporter cluster.
    2. In the STEP2 section, select the Prometheus instance on which you want the Kafka exporter to reside.
    3. On the configuration tab in the STEP3 section, configure the parameters and click OK.
      ParameterDescription
      Instance nameThe unique name of the Kafka exporter.
      Kafka instance nameThe name of the Kafka instance that you want to monitor. You can specify an instance name on the dashboard to view the producer, broker, and consumer of a Kafka cluster.
      Pod/ECS Label Key (service discovery)The key and value that are specified for the pod or ECS instance when the JMX agent is deployed. Managed Service for Prometheus uses this key-value pair for service discovery.
      Pod/ECS Label value
      JMX Agent listening portThe listening port that is specified when the JMX agent is deployed.
      Metrics pathThe HTTP path that is used by Prometheus to collect monitoring data from the JMX agent. Default value: /metrics.
      Metrics space interval (seconds)The interval at which you want the service to collect monitoring data.

Step 2: View the status of the Kafka (Advanced Version) exporter

  1. Log on to the ARMS console.
  2. In the left-side navigation pane, choose Prometheus Service > Prometheus Instances.
  3. Click the name of the Prometheus instance instance that you want to manage to go to the Integration Center page.
  4. Click the Kafka(Advanced Version) card in the Installed section. In the panel that appears, you can view information on the following tabs:
    • Targets: You can view the list of targets discovered by default jobs on the Targets tab.
    • Metrics: You can view information about metrics and discard metrics on the Metrics tab. For more information, see Configure metrics.
    • Dashboards: You can click a dashboard thumbnail to view the Grafana dashboard on the Dashboards tab. The Kafka (Advanced Edition) exporter provides dashboards based on instances and topics.
      • Instance dashboard
        The following metrics of Kafka brokers are displayed on the dashboard:
        • Core metrics: the numbers of brokers, offline partitions, under-replicated partitions, and controllers, and information about CPUs and networks
        • Java Virtual Machine (JVM) metrics: the key information about the JVM memory and garbage collection (GC)
        • Partition metrics: the partition information, such as the partition quantity, ISR, unclean leader election, replica lag, offline partitions, and under-replicated partitions
        • Time metrics: the time metrics in the Produce, Request, and Fetch phases
        • Cluster traffic metrics: the overall traffic metrics of the cluster
        • Broker traffic metrics: broker-specific traffic details
      • Topic dashboard
        The following end-to-end metrics of each Kafka topic are displayed on the dashboard:
        • Producer: the key metrics of the producer, including the message sending rate, message compression ratio, and message sending latency
        • Server (Kafka broker): the number of partitions in a topic, and the rates and traffic volumes of inbound messages and outbound messages
        • Consumer: the message consumption rate, message consumption latency, and rebalance information
    • Alerts: You can view Prometheus alerts on the Alerts tab. Managed Service for Prometheus provides alert metrics for self-managed Kafka producers, instances, and consumers.
      • Producer: Three alert metrics are provided, including the message sending failure rate, message sending duration, and message sending retry rate. You can use the metrics to identify exceptions on the producer.
      • Instance: Thirteen alert metrics are provided, including topics with excessive partitions, offline partitions, unclean leader election, under-replicated partitions, a decrease in effective brokers, the number of effective controllers, the number of rejected messages, numbers of inbound messages and outbound messages of the instance, and numbers of inbound messages and outbound messages of topics. You can use the metrics to identify exceptions on the broker.
      • Consumer: The alert metric for message accumulation is provided. You can use this metric to identify exceptions on consumption.

      You can also create alert rules based on your business requirements. For more information about how to create a Prometheus alert rule, see Create an alert rule for a Prometheus instance.

    • Service Discovery Configurations: You can manage the default basic monitoring jobs on the Service Discovery Configurations tab. For example, you can change data extraction intervals or disable collection jobs.
    • Exporter: You can view the version number of node-exporter that is installed by default on the Exporter tab.