Managed Service for Prometheus collects Kafka performance metrics through a Java Management Extensions (JMX) agent. Deploy the agent on your Kafka nodes to expose metrics over HTTP, then configure service discovery so that Prometheus scrapes the data automatically.
This guide covers both Container Service for Kubernetes (ACK) and Elastic Compute Service (ECS) deployment environments.
Prerequisites
Before you begin, make sure that you have:
A Kafka cluster with Producer, Broker, and Consumer components running on ACK or ECS
A Prometheus instance for ACK or a Prometheus instance for ECS (other instance types are not supported)
Access to the pods or ECS instances where Kafka is deployed
Access to the ACK console or the ECS console, depending on your deployment type
Step 1: Deploy the Kafka JMX agent
Download the Kafka JMX agent package (
kafka-jmx_prometheus_javaagent-1.18.1.jar) to each pod or ECS instance where Kafka runs.Add the following flag to the JVM startup parameters for each Kafka component (Producer, Broker, and Consumer):
Placeholder Description Example <jmx-agent-directory>Directory where you saved the JMX agent JAR file /opt/kafka/libs<jmx-listening-port>Port on which the JMX agent exposes metrics over HTTP 9404-javaagent:/<jmx-agent-directory>/kafka-jmx_prometheus_javaagent-1.18.1.jar=<jmx-listening-port>
Restart Kafka Producer, Broker, and Consumer for the changes to take effect.
Verify that the agent is running. On each pod or ECS instance, run: Replace
<jmx-listening-port>with the port you specified in sub-step 2. If the agent is running correctly, the command returns metric data in Prometheus exposition format.curl localhost:<jmx-listening-port>/metrics
Step 2: Configure service discovery
Managed Service for Prometheus uses tags to discover scrape targets. The configuration differs depending on whether Kafka runs on ACK or ECS.
ACK deployments
For ACK deployments, optionally expose the JMX agent port and add labels to your pods.
(Optional) Expose the container port
If Managed Service for Prometheus cannot scrape the JMX metrics endpoint, expose the agent port in your Kubernetes deployment. If you deploy a Kafka JMX agent in a Prometheus instance for ECS, skip this step.
Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, find your cluster and click its name. In the left-side pane, choose Workloads > Deployments.
In the Actions column of the target deployment, click Edit.
In the panel that appears, configure the following fields:
Field Description Name A descriptive name for the JMX port. Example: app1-kafka-jmx.Container Port The <jmx-listening-port>you specified in Step 1.Protocol Select TCP. 
Add labels to pods
Add a label in the format <custom-tag-key>:<custom-tag-value> to each Kafka Producer, Broker, and Consumer pod. Example: arms-kafka-exporter:my-kafka1.
In the ACK console, navigate to your cluster. In the left-side pane, choose Workloads > Pods.
On the Pods page, click Edit in the Actions column of the target pod.
Add the labels and click Update.
Note: If your pods already have labels configured, skip this step.
ECS deployments
Add tags in the format <custom-tag-key>:<custom-tag-value> to each ECS instance that runs Kafka Producer, Broker, or Consumer. Example: arms-kafka-exporter:my-kafka1. For instructions, see Tags.
Note: If your ECS instances already have tags configured, skip this step.
What to do next
After you complete the deployment and service discovery configuration, configure the Kafka JMX agent in Managed Service for Prometheus to start viewing monitoring data. For instructions, see Monitor self-managed Kafka clusters and ApsaraMQ for Kafka instances.