ApsaraMQ for Kafka provides a dashboard feature that is powered by the metric storage and display capabilities of Alibaba Cloud ARMS Prometheus Service and Grafana. This feature offers a comprehensive and centralized way to monitor metrics and collect metric data, which helps you quickly understand your business status.
Prerequisites
A service-linked role is created.
Role Name: AliyunServiceRoleForAlikafka
Policy Name: AliyunServiceRolePolicyForAlikafka
Permissions: Allows ApsaraMQ for Kafka to use this role to access other Alibaba Cloud services, such as CloudMonitor and ARMS. This access is required to enable the CloudMonitor and dashboard features.
For more information, see Service-linked Role.
Billing
Metrics for ApsaraMQ for Kafka are considered basic metrics in Managed Service for Prometheus and are free of charge. You are not charged for using the dashboard feature.
For more information, see Metric description and Pay-as-you-go.
Metric details
The following table describes the fields in the metrics.
Field | Value |
Metric type |
|
Label |
|
Instance metrics
Metric type | Metric name | Description | Labels |
Gauge | kafka_disk_log_size | The storage size of the instance. Unit: bytes. |
|
Gauge | kafka_server_cloudenhancedreplicamanager_allreplicascount | The total number of partitions for all replicas. |
|
Gauge | kafka_server_socket_server_metrics_connection_count | The number of connections. |
|
Gauge | kafka_instance_io_spec_write | The reserved specification for instance sending. |
|
Gauge | kafka_instance_io_spec_read | The reserved specification for instance consumption. |
|
Counter | kafka_server_brokertopicmetrics_bytesin_total | Production traffic (jmx-exporter metric). |
|
Counter | kafka_server_brokertopicmetrics_bytesout_total | Consumption traffic (jmx-exporter metric). |
|
Counter | kafka_server_brokertopicmetrics_failedproducerequests_total | The number of failed producer requests. |
|
Counter | kafka_server_brokertopicmetrics_totalproducerequests_total | The total number of producer requests. |
|
Counter | kafka_server_brokertopicmetrics_failedfetchrequests_total | The number of failed consumer requests. |
|
Counter | kafka_server_brokertopicmetrics_totalfetchrequests_total | The total number of consumer requests. |
|
Gauge | kafka_network_socketserver_expiredconnectionskilledcount | The number of expired connections. |
|
Summary | kafka_network_requestmetrics_requestbytes | The size of the request body. |
|
Counter | kafka_server_brokertopicmetrics_bytesrejected_total | The traffic rejected by each topic when the record batch size is larger than the max.message.bytes configuration. Unit: bytes. |
|
Counter | kafka_server_brokertopicmetrics_nokeycompactedtopicrecords_total | The number of records in a compacted topic that do not have a key. |
|
Counter | kafka_server_brokertopicmetrics_invalidmessagecrcrecords_total | The number of cyclic redundancy check (CRC) failures. |
|
Counter | kafka_server_brokertopicmetrics_invalidmagicnumberrecords_total | The number of message version verification failures. |
|
Counter | kafka_server_brokertopicmetrics_invalidoffsetorsequencerecords_total | The number of message authentication failures caused by discontinuous offsets or sequence numbers. |
|
Topic metrics
Metric type | Metric name | Description | Labels |
Gauge | kafka_log_log_size | The storage size of the topic partition. |
|
Gauge | kafka_topic_partition_current_offset | The maximum offset of the partition. |
|
Gauge | kafka_topic_partition_oldest_offset | The minimum offset of the partition. |
|
Gauge | kafka_consumergroup_lag | The amount of message accumulation. |
|
Counter | kafka_server_brokertopicmetrics_totalfetchrequests_total | The number of requests. |
|
Counter | kafka_server_brokertopicmetrics_bytesin_total | Production traffic (jmx-exporter metric). |
|
Counter | kafka_server_brokertopicmetrics_bytesout_total | Outbound traffic. Unit: bytes. |
|
Counter | kafka_server_brokertopicmetrics_messagesin_total | The number of inbound messages. Unit: messages. |
|
Group metrics
Metric type | Metric name | Description | Labels |
Gauge | kafka_consumergroup_lag | The total amount of message accumulation. |
|
Counter | kafka_consumergroup_current_offset | The consumer offset of the group. |
|
View dashboards in the ApsaraMQ for Kafka console
Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.
On the Instances page, click the name of the instance that you want to manage.
In the navigation pane on the left, click .
On the Dashboard page, set a time range in the upper-right corner to view metrics such as instance storage size, number of partitions, number of connections, and inbound and outbound traffic.
To view dashboard monitoring data for topics and groups, navigate to the details page of the specific topic or group and click the Dashboard tab.
On the Topic Details page, you can view production-related metrics and producer client metrics for a specific topic.
On the Group Details page, you can view consumption-related metrics and consumer client metrics for a specific group.
View dashboards in the Managed Service for Grafana console
Log on to the Managed Service for Grafana console. In the navigation pane on the left, click Workspace Management.
On the Workspace Management page, click the URL in the Endpoint column of the target workspace to open Grafana.
View dashboards based on your Grafana version.
Grafana 9.x
In the navigation pane on the left of Grafana, click the
icon and then click Browse.On the Browse tab, click a dashboard in the target folder to view all monitoring charts on the dashboard.
For example, to view the dashboard data of an instance in the China (Hangzhou) region, click kafka-dashboard-v3 in the folder that starts with
cloud-product-prometheus_cn-hangzhou. Then, enter the instance ID on the dashboard page to view the monitoring data of the instance.
Grafana 10.x
On the Grafana home page, click the
icon in the upper-left corner.In the navigation pane on the left of Grafana, select Dashboards. Click a dashboard in the target folder to view all monitoring charts on the dashboard.
For example, to view the dashboard data of an instance in the China (Hangzhou) region, click kafka-dashboard-v3 in the folder that starts with
cloud-product-prometheus_cn-hangzhou. Then, enter the instance ID on the dashboard page to view the monitoring data of the instance.
FAQ
How do I obtain dashboard metric data?
Log on to the ARMS console with your Alibaba Cloud account.
In the navigation pane on the left, click Integration Center.
On the Integration Center page, enter
Kafkain the search box and click the search icon.In the search results, select the Alibaba Cloud service that you want to integrate, such as Alibaba Cloud Kafka Message Queue Service. For more information, see Step 1: Integrate monitoring data of an Alibaba Cloud service.
NoteTo obtain metric data for ApsaraMQ for Kafka Serverless instances, turn on the Advanced Monitoring Metrics switch in the Configuration Information section during integration. Otherwise, you can obtain metric data only for non-Serverless instances.
After the integration is successful, click Integration Management in the navigation pane on the left.
On the Integration Management page, click the Cloud Service Region Environment tab.
In the Cloud Service Region Environment list, click the name of the target environment to open the environment details page.
On the Component Management tab, in the Basic Information section, click the region of the Prometheus Instance.
On the Settings tab, you can view the available data access methods.
How do I connect dashboard metric data to a self-managed Grafana instance?
All metric data for ApsaraMQ for Kafka is stored in your Managed Service for Prometheus instance. You can use the APIs provided by Managed Service for Prometheus to connect the dashboard metric data from ApsaraMQ for Kafka to your self-managed Grafana instance.
For more information, see Use an HTTP API endpoint to connect Prometheus data to a Grafana instance or a self-managed application.
Before you connect the data, make sure that Managed Service for Prometheus in the region where your ApsaraMQ for Kafka instance resides is integrated with Alibaba Cloud Kafka Message Queue Service.