This topic describes how to expose data by using instrumentation points in Kafka applications, capture data by using Prometheus Monitoring of Application Real-Time Monitoring Service (ARMS), and display data on the Prometheus Grafana dashboard to monitor Kafka applications.

Background information

The following figure shows the procedure.

How It Works

Step 1: Start the JMX service

Enable the Java Management Extensions (JMX) service in Kafka applications to obtain resource information.

  1. Add export JMX_PORT="9999" to the first line in the /opt/kafka/kafka_2.11- file.
  2. Restart Kafka.

Step 2: Start jmx_exporter

Start jmx_exporter to allow access to JMX information by using HTTP so that ARMS Prometheus Monitoring can capture data.

  1. Download the kafka.yml file to the /opt/exporter_kafka/ directory.
  2. Modify the downloaded file /opt/exporter_kafka/kafka-0.8.2.yml
    by adding hostPort: localhost:9999 to the first line in the file to expose the running port of the JMX service to jmx_exporter.
  3. Download the executable file of jmx_exporter to the /opt/exporter_kafka/ directory.
  4. Start jmx_exporter.
    java - - -cp 
    io.prometheus.jmx.WebServer 9997 /opt/exporter_kafka/kafka-0-8-2.yml &
    After you perform the preceding operations, the application configuration is complete. You can run the following command to check whether jmx_exporter is running properly:
    curl http://<IP address of the server where jmx_exporter is deployed>:9997/metrics

    For more information about Kafka metrics, see Monitoring Kafka.

Step 3: Configure ARMS Prometheus Monitoring to capture the data of Kafka applications

  1. Log on to the ARMS console.
  2. In the left-side navigation pane, click Prometheus Monitoring.
  3. In the upper-left corner of the Prometheus Monitoring page, select the region where your Container Service for Kubernetes (ACK) clusters are deployed. Click Settings in the Actions column corresponding to a cluster.
  4. On the page that appears, click the Prometheus Settings tab.
  5. On the Prometheus Settings tab, enter the following code and click Save:
      scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
      evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
      - job_name: 'kafka'
        - targets: ['']

Step 4: Display the data of Kafka applications on the Grafana dashboard

Import the Grafana dashboard template on the Prometheus Monitoring page and specify the cluster where the Prometheus data source is deployed.

  1. Go to Host Dashboard.
  2. In the left-side navigation pane, choose + > Import. On the Import page, enter 10973 in the Dashboard field and click Load.
    Import Grafana Dashboard
  3. On the Import page, enter the following information and click Import.
    Import Grafana Dashboard with Options
    1. Enter a custom dashboard name in the Name field.
    2. Select the cluster from the Folder drop-down list.
    3. Select the cluster from the Select a Prometheus data source drop-down list.
    After the configuration is complete, the Prometheus Grafana Kafka dashboard appears, as shown in the following figure.prometheus_kafka_grafana

Step 5: Create an ARMS Prometheus Monitoring alert

  1. Log on to the ARMS console.
  2. In the left-side navigation pane, click Prometheus Monitoring.
  3. In the top navigation bar, select a region. Then, click the name of the required Kubernetes cluster.
  4. In the left-side navigation pane, choose Alarm configuration beta. Then, click Create Alert in the upper-right corner.
  5. In the Create Alert dialog box, configure the following parameters, and then click OK.
    Note The Time parameter is not supported.
    1. Enter a name in the Rule Name field. Example: alerts for inbound traffic.
    2. Enter an expression that uses a PromQL statement. Example: (sum(rate(kube_state_metrics_list_total{job="kube-state-metrics",result="error"}[5m])) / sum(rate(kube_state_metrics_list_total{job="kube-state-metrics"}[5m]))) > 0.01.
      Notice An error may be reported if a PromQL statement contains a dollar sign ($). You must remove the equal sign (=) and the parameters on both sides of the equal sign (=) from the statement that contains the dollar sign ($). For example, change sum (rate (container_network_receive_bytes_total{instance=~"^$HostIp.*"}[1m])) to sum (rate (container_network_receive_bytes_total[1m])).
    3. In the Labels section, click Create Tag to specify alert tags. The specified tags can be used as options for a dispatch rule.
    4. In the Annotations section, specify a template for alert messages. Click Create Annotation. Set Key to message and Value to {{variable name}} alert message. The specified annotation is in the format of message:{{variable name}} alert notification. Example: message:{{$labels.pod_name}} restart.

      You can customize a variable name or select an existing tag as the variable name. Existing tags:

      • The tags that are carried in the metrics of an alert rule expression.
      • The tags that are created when you create an alert rule. For more information, see Create an alert.
      • The default tags provided by ARMS. The following table describes the default tags.
        Tag Description
        alertname The name of the alert. The format is <Alert name>_<Cluster name>.
        _aliyun_arms_alert_level The level of the alert.
        _aliyun_arms_alert_type The type of the alert.
        _aliyun_arms_alert_rule_id The ID of the alert rule.
        _aliyun_arms_region_id The ID of the region.
        _aliyun_arms_userid The ID of the user.
        _aliyun_arms_involvedObject_type The subtype of the associated object, for example, ManagedKubernetes or ServerlessKubernetes.
        _aliyun_arms_involvedObject_kind The type of the associated object, for example, app or cluster.
        _aliyun_arms_involvedObject_id The ID of the associated object.
        _aliyun_arms_involvedObject_name The name of the associated object.
    Prometheus-Create alarm

What to do next

After the Prometheus Grafana Kafka dashboard is configured, you can view ARMS Prometheus Monitoring metrics and customize the dashboard. For more information, see the following references:

View Prometheus Monitoring metrics

Use ARMS Prometheus Monitoring to customize Grafana dashboards

Create an alert