Application Real-Time Monitoring Service:Deploy and configure a Kafka JMX agent

Limitations

The Kafka JMX agent integrates only with a Prometheus instance for ACK or a Prometheus instance for ECS.

Step 1: Deploy the Kafka JMX agent

1. Download the Kafka JMX agent package to each pod or ECS instance where Kafka is deployed.

2. In the Java Virtual Machine (JVM) startup parameters for your Kafka Producer, Broker, and Consumer, add the following argument: -javaagent:/<path_to_agent>/kafka-jmx_prometheus_javaagent-1.18.1.jar=<jmx_listening_port>.

   • <path_to_agent>: Replace this placeholder with the directory where you saved the agent.

   • <jmx_listening_port>: Replace this placeholder with the actual JMX listening port.

   After restarting Kafka, run the ps -ef|grep javaagent command to verify that the JMX agent is loaded. If the process parameters include -javaagent:/kafka-jmx_prometheus_javaagent-1_18_1.jar=5556, the agent is deployed successfully.

   
   #ps -ef|grep javaagent
   admin    30038  30036 63 10:44 ?    00:00:14 /opt/taobao/java/bin/java -javaagent:/kafka-jmx_prometheus_javaagent-1.18.1.jar=5556 -Xmx4g -Xms4g -Xmn2g -Xss1024k -server -XX:+UseG1GC -XX:G1HeapRegionSize=16m -XX:MaxGCPauseMillis=20 ...
   

3. Restart the Kafka Producer, Broker, and Consumer.

4. To verify that the Kafka JMX agent is running, run the curl localhost:<jmx_listening_port>/metrics command on each pod or ECS instance. The agent is running if the command returns metric data.

   <jmx_listening_port>: Replace this placeholder with the actual JMX listening port.

   
   [root@kaikaxxx ~]# curl localhost:5556/metrics
   # HELP jvm_memory_pool_allocated_bytes_total Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
   # TYPE jvm_memory_pool_allocated_bytes_total counter
   jvm_memory_pool_allocated_bytes_total{pool="G1 Old Gen",} 1.7518xxx
   jvm_memory_pool_allocated_bytes_total{pool="Code Cache",} 1.4520xxx
   jvm_memory_pool_allocated_bytes_total{pool="G1 Eden Space",} 1.4xxx
   jvm_memory_pool_allocated_bytes_total{pool="G1 Survivor Space",}
   jvm_memory_pool_allocated_bytes_total{pool="Compressed Class Space",} 5352720.0
   jvm_memory_pool_allocated_bytes_total{pool="Metaspace",} 4.xxx
   # HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
   # TYPE jvm_gc_collection_seconds summary
   jvm_gc_collection_seconds_count{gc="G1 Young Generation",} 21.0
   jvm_gc_collection_seconds_sum{gc="G1 Young Generation",} 0.346
   jvm_gc_collection_seconds_count{gc="G1 Old Generation",} 0.0
   jvm_gc_collection_seconds_sum{gc="G1 Old Generation",} 0.0
   # HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
   # TYPE process_cpu_seconds_total counter
   process_cpu_seconds_total 22.85
   # HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
   # TYPE process_start_time_seconds gauge
   process_start_time_seconds 1.6654xxx
   # HELP process_open_fds Number of open file descriptors.
   # TYPE process_open_fds gauge
   process_open_fds 204.0
   # HELP process_max_fds Maximum number of open file descriptors.
   # TYPE process_max_fds gauge
   process_max_fds 65535.0
   # HELP process_virtual_memory_bytes Virtual memory size in bytes.
   # TYPE process_virtual_memory_bytes gauge
   process_virtual_memory_bytes 4.428xxx
   # HELP process_resident_memory_bytes Resident memory size in bytes.
   # TYPE process_resident_memory_bytes gauge
   process_resident_memory_bytes 9.36xxx
   # HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded.
   # TYPE jmx_config_reload_failure_total counter
   jmx_config_reload_failure_total 0.0
   # HELP jmx_exporter_build_info A metric with a constant '1' value labeled with the version of the JMX exporter.
   # TYPE jmx_exporter_build_info gauge
   jmx_exporter_build_info{version="0.17.1-SNAPSHOT",name="jmx_prometheus_javaagent",} 1.0
   # HELP jvm_memory_objects_pending_finalization The number of objects waiting in the finalizer queue.
   # TYPE jvm_memory_objects_pending_finalization gauge
   jvm_memory_objects_pending_finalization 0.0
   # HELP jvm_memory_bytes_used Used bytes of a given JVM memory area.
   # TYPE jvm_memory_bytes_used gauge
   jvm_memory_bytes_used{area="heap",} 4.0462xxx
   jvm_memory_bytes_used{area="nonheap",} 6.57xxx
   # HELP jvm_memory_bytes_committed Committed (bytes) of a given JVM memory area.
   # TYPE jvm_memory_bytes_committed gauge
   jvm_memory_bytes_committed{area="heap",} 1.07374xxx
   jvm_memory_bytes_committed{area="nonheap",} 7.0xxx
   # HELP jvm_memory_bytes_max Max (bytes) of a given JVM memory area.
   # TYPE jvm_memory_bytes_max gauge
   jvm_memory_bytes_max{area="heap",} 1.0737xxx
   jvm_memory_bytes_max{area="nonheap",} -1.0
   # HELP jvm_memory_bytes_init Initial bytes of a given JVM memory area.
   # TYPE jvm_memory_bytes_init gauge
   jvm_memory_bytes_init{area="heap",} 1.0737xxx
   jvm_memory_bytes_init{area="nonheap",} 255xxx
   # HELP jvm_memory_pool_bytes_used Used bytes of a given JVM memory pool.
   # TYPE jvm_memory_pool_bytes_used gauge
                 

(Optional) Step 2: Configure container port

If you use a Prometheus instance for ACK, you must also configure the container port to allow Managed Service for Prometheus to scrape data from the Kafka JMX agent. If you use a Prometheus instance for ECS, skip this step.

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Workloads > Deployments.

3. In the Actions column for the target workload, click Edit. On the Edit page, configure the port Name, Container Port, and Protocol, and then click Update.

   • Name: The port name of the Kafka JMX agent, for example, app1-kafka-jmx.

   • Container Port: The JMX listening port that you specified in Step 1: Deploy the Kafka JMX agent.

   • Protocol: Select TCP.

Step 3: Configure pod labels or ECS tags

Managed Service for Prometheus uses service discovery to identify scrape targets based on pod labels or ECS instance tags. You must add a label or tag to enable Managed Service for Prometheus to discover the targets. You can skip this step if your services already have suitable labels or tags.

Configure labels for Prometheus instance for ACK

On the pods for the Kafka Producer, Broker, and Consumer, add a label in the format <custom_label_key>:<custom_label_value>, for example, arms-kafka-exporter:my-kafka1.

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Workloads > Pods.

3. On the Container Group page, click Edit in the Actions column of the target pod, add the label, and then click Update.

   
   apiVersion: v1
   kind: Pod
   metadata:
     annotations:
       k8s.aliyun.com/pod-ips: 10.119.xxx.xxx
     creationTimestamp: '2025-01-10T05:58:06Z'
     generateName: nginx-xxx-deployment-basic-8f9xxx t-
     labels:
       app: nginx
       pod-template-hash: 8f95d5bd4
     managedFields:
     - apiVersion: v1
       fieldsType: FieldsV1
       fieldsV1:
         'f:metadata':
   

Configure tags for Prometheus instance for ECS

Next steps

After you configure the Kafka JMX-Agent, you can complete the integration configuration and view its monitoring data in the Managed Service for Prometheus console. For more information, see How to Use Prometheus to Monitor Kafka.