Monitor instance risks with real-time health check - ApsaraMQ for Kafka

Message Queue for Apache Kafka periodically runs health checks on your instances, reports diagnosed issues with fix suggestions, and sends alerts for detected risks to specified contacts.

Implementation plan

Alert notifications

You receive alert notifications only for urgent and unhealthy alerts.
If you do not add an alert contact, notifications are sent by default to the instance owner's Alibaba Cloud account contact.
If you add alert contacts, they receive notifications only during the time range specified for each contact. For more information, see Manage alert contacts.

View Kafka risk events in Cloud Monitor Event Center

Instance risks are synchronized to Cloud Monitor Event Center, where you can subscribe to risk events and configure advanced notification policies such as denoising, notification methods, channels, and templates. For details, see Manage event subscriptions (Recommended). To subscribe to Kafka risk events: Cloud service: Kafka; Event type: Abnormal; Event name: Instance Risk Alert.

Checks

Note

To resolve a threat on an instance, follow the suggestions on the console.

Risk type	Severity	Finding	Recommendation	Verification
CPU usage	Moderate Major	CPU usage: xx%	High CPU usage is often caused by issues such as fragmented message sending. Resolving these issues can reduce CPU consumption.	An alert is triggered in Prometheus if CPU usage exceeds 70%.
Disk usage	Major Critical	Disk usage: xx%	To ensure data security and cluster stability: When the disk watermark reaches 85% or higher, unexpired messages are periodically deleted. When the disk watermark reaches 90% or higher, the cluster enables write protection.	An alert is triggered in Prometheus if the instance disk usage exceeds 80%.
Disk skew	Major Critical	None	Disk skew prevents the cluster from fully utilizing its disk performance and capacity. Add topic partitions based on the suggested value or trigger a cluster rebalance. For more information, see What do I do if topic partitions are skewed?.	An alert is triggered if the difference between the maximum and minimum disk usage exceeds 50%. The console displays only the maximum disk usage.
Message production format conversion latency	Major Critical	TP98 message production format conversion latency: xx ms	Format conversion on the affected topics degrades overall sending performance. To fix this, ensure the producer client and server versions match.	Check whether the producer client version for the affected topics is inconsistent with the server version.
Message consumption format conversion latency	Major Critical	TP98 message consumption format conversion latency: xx ms	Format conversion degrades overall consumption performance. To fix this, ensure the consumer client and server versions match.	Check whether the consumer client version is inconsistent with the server version.
Topic format conversion	Moderate Major	xx topics are undergoing format conversion	Format conversion on the affected topics may degrade overall sending performance. Align the producer client and server versions to reduce performance loss.	Check whether the producer client version for the affected topics is inconsistent with the server version.
Consumer group subscribes to too many topics	Moderate Major	xx consumer groups are subscribed to too many topics	A consumer group subscribed to too many topics is prone to rebalances, which degrades overall consumption performance. Unless your business requires otherwise, maintain a one-to-one relationship between consumer groups and topics. For more information, see Subscriber best practices.	An alert is triggered if a consumer group is subscribed to more than one topic.
Use of Sarama Go client	Moderate Major	xx consumer groups use the Sarama Go client for consumption	The affected consumer groups use the Sarama Go client. This client is not recommended because of its many known issues. For more information, see Why is it not recommended to use the Sarama Go client to send and receive messages?.	An alert is triggered if a consumer client uses Sarama Go to commit consumer offsets.
Rebalance timeout	Moderate Major	xx consumer groups are experiencing rebalance timeouts	The affected consumer groups have long rebalance timeouts. Do not set the `max.poll.interval.ms` parameter to a large value, as this can prolong rebalances.	Go to the details page of the affected consumer group to view rebalance details.
Consumer client actively leaving group	Major Critical	xx consumer groups have clients that are actively leaving and triggering rebalances	In the affected consumer groups, clients are actively leaving and triggering rebalances: Check if the consumer client is stuck. Check if the consumer client configuration is appropriate. For more information, see Why does my consumer client frequently rebalance?.	Go to the details page of the affected consumer group to view rebalance details.
High consumption latency in consumer groups	Moderate Major	xx consumer groups are experiencing high data reception latency	High consumption latency in the affected consumer groups may be caused by the following: The consumer client parameter `fetch.max.bytes` is set too high. The client is in a poor network environment. For optimization solutions, see Subscriber best practices.	Check the consumption latency from the consumer client.
Consumer group quota	Moderate Major	Remaining consumer group quota: xx	The consumer group quota is almost exhausted.	View the total number of consumer groups on the instance details page.
Topic quota	Major Critical	Remaining topic quota: xx	The topic quota is almost exhausted.	View the total number of topics on the instance details page.
Partition quota	Major Critical	Remaining partition quota: xx	The partition quota is almost exhausted.	View the total number of partitions on the instance details page.
Server minor version update	Major Critical	The current server minor version is xx versions behind the latest release	The latest minor version fixes several known open-source bugs and improves cluster performance and stability. For service stability, update the server to the latest minor version as soon as possible.	View details about the minor version on the instance details page.
TCP connections per node	Major Critical	Number of TCP connections on a single node: xx	An excessive number of TCP connections can affect overall cluster stability. Check your connection method to see if connection objects are being repeatedly instantiated. A large number of clients can cause severe message fragmentation. Reduce the number of clients and adjust the `batch.size` and `linger.ms` parameters to aggregate data into batches before sending. A continuous increase in connections may cause some connection attempts to fail.	View the maximum number of TCP connections for an instance node on the dashboard or in Prometheus. An alert is triggered if the number exceeds the specified limit. For more information about limits, see Usage limits.
Public TCP connections per node	Major Critical	Number of public TCP connections on a single node: xx	An excessive number of public TCP connections can affect overall cluster stability. Check your connection method to see if connection objects are being repeatedly instantiated. A large number of clients can cause severe message fragmentation. Reduce the number of clients and adjust the `batch.size` and `linger.ms` parameters to aggregate data into batches before sending. A continuous increase in connections may cause some connection attempts to fail. A public network connection is heavyweight and performs poorly. Use it for development and testing only. Use VPC connections in production environments.	View the maximum number of public TCP connections for an instance node on the dashboard or in Prometheus. An alert is triggered if the number exceeds the specified limit. For more information about limits, see Usage limits.
Synchronous sending issue	Major Critical	xx topics have issues with synchronous sending	The affected topics use a synchronous disk flushing mechanism with `acks=all`, which results in poor sending performance and affects the cluster's processing efficiency. Unless your business requires it, set `acks=1` to significantly improve sending efficiency. For more information, see Publisher best practices.	Check if the producer client for the affected topics has `acks` configured as `all` or `-1`.
Fragmented message sending	Major Critical	xx topics have issues with fragmented message sending	The affected topics are experiencing fragmented sending, which can cause send queue timeouts and affect the cluster's overall sending throughput and stability. To improve sending performance: Set the `batch.size` and `linger.ms` parameters based on your business requirements. If your topics have many partitions, we recommend using the sticky partitioning strategy. For more information, see Publisher best practices.	An alert is triggered if the sent `batch.size` is less than 4 KB and the node CPU usage is greater than 60%. Configure the producer client parameters for the affected topics appropriately. We recommend using client version 2.4.0 or later.
Shared whitelist security group	Major Critical	The default endpoint whitelist shares security group ID: xx	Specifying a whitelist security group ID during instance deployment can cause multiple instances to share the same whitelist configuration. Modifying the whitelist for one instance will affect all other instances using the same security group. This increases the blast radius of misconfigurations. Be aware of the associated risks.	Check whether the security group used by the instance is also used by other resources.
Single-partition topic risk	Major Critical	xx single-partition topics are using cloud storage	A single-partition topic on cloud storage may become unavailable during downtime or an upgrade. Add partitions. If you have a strong dependency on a single partition, use local storage instead.	Check the number of partitions for the affected topics.
Topic partition skew	Major Critical	xx topics have partition skew	Topic partition skew poses the following risks: Prevents full utilization of disk performance and capacity. May trigger single-node throttling. Add topic partitions based on the suggested value. For optimization solutions, see What do I do if topic partitions are skewed?.	On the details page of the affected topic, check if the number of partitions matches the recommended number.
Node production traffic	Major Critical	Node production traffic has exceeded the limit by xx%	The node's production traffic has reached its limit. Scale up the instance as soon as possible to ensure instance stability.	In Prometheus, check the node's maximum production traffic (bytes/s) and the instance's production throttling queue length (requests/s) to see if throttling is occurring.
Node consumption traffic	Major Critical	Node consumption traffic has exceeded the limit by xx%	The node's consumption traffic has reached its limit. Scale up the instance as soon as possible to ensure instance stability.	In Prometheus, check the node's maximum consumption traffic (bytes/s) and the instance's consumption throttling queue length (requests/s) to see if throttling is occurring.
Production traffic	Major Critical	Production traffic has exceeded the limit by xx%	The cluster's production traffic has reached its limit. Some traffic may be throttled, causing production timeouts. Scale up the instance as soon as possible to avoid business impact and ensure cluster stability.	In Prometheus, check the instance's message production traffic (bytes/s) and its production throttling queue length (requests/s) to see if throttling is occurring.
Consumption traffic	Major Critical	Consumption traffic has exceeded the limit by xx%	The cluster's consumption traffic has reached its limit. Some traffic may be throttled, preventing messages from being consumed promptly and causing backlogs. Scale up the instance as soon as possible to avoid business impact and ensure cluster stability.	In Prometheus, check the instance's message consumption traffic (bytes/s) and its consumption throttling queue length (requests/s) to see if throttling is occurring.
Partition assignment strategy issue	Major Critical	In xx consumer groups, the same partition is assigned to multiple consumer threads	The same partition is being assigned to multiple consumer threads. Check for issues with the consumer assignment strategy. For more information, see Why is the same partition consumed by multiple consumer threads?.	Check the consumer clients to see if multiple consumers are processing the same partition.
Consumer offset commit frequency	Major Critical	A consumer client is committing consumer offsets xx times per second	Consumer clients are committing consumer offsets too frequently, which affects cluster performance and stability. Switch to automatic offset commits or reduce the commit frequency. The top 10 groups with the highest commit frequency are shown below. For optimization solutions, see Subscriber best practices.	None.
Daily rebalances in consumer group	Moderate Major	xx consumer groups triggered rebalances within the last day	The affected consumer groups have experienced rebalances within the last day. See the consumer group details page for specific times. Check if the rebalances were caused by improper consumer client configuration. For more information, see Why does my consumer client frequently rebalance?.	Check the rebalance details for the affected consumer groups.
Disk cold read	Major Critical	Cold read rate: xx%	Disk cold reads are occurring. Consumers are reading a large amount of data from disk, which affects cluster performance and stability. This may be caused by high consumption latency leading to message backlogs. Increase the consumption rate or reset the consumer offset.	An alert is triggered if disk IOPS or bandwidth usage exceeds 70%.
Disk write protection	Major Critical	Disk write protection has been triggered	Disk usage is too high and has triggered write protection. Optimize promptly.	In Prometheus, check if instance disk usage has exceeded 90%, which triggers disk write protection.
Consumer offset rollback	Major Critical	xx consumer groups have experienced consumer offset rollbacks	Consumer offset rollbacks have occurred in the affected consumer groups, possibly due to an offset reset.	Check if the consumer client committed a historical offset, which caused the consumer offset to roll back.
Use of GZIP compression	Major Critical	xx topics are using GZIP compression	The affected topics use GZIP compression, which increases the cluster load. Optimize promptly.	Check if the producer client is configured to use GZIP compression.
Early message deletion risk	Major Critical	Messages may be deleted before their retention period expires	The single-disk capacity is small and the disk watermark is high. Messages may be deleted prematurely. Monitor and address this issue promptly.	Check the disk capacity and the current maximum disk watermark.
Expired server major version	Major Critical	The server's major version has expired	Your server is running an old major version that has known open-source bugs. To ensure service stability, upgrade the server major version as soon as possible by following the documentation.	The server major version is 0.10.x.
Consumer offset includes leader_epoch	Major Critical	xx consumer groups are including the leader_epoch record when committing offsets	The consumer clients for the affected groups include the leader_epoch record when committing offsets, which can lead to consumption errors or failures. Upgrade the client to version 2.6 or later as soon as possible. For more information, see the open-source issue.	This occurs when the server major version is 2.2.0 and the consumer client (versions 2.3 to 2.5) includes the leader_epoch field when committing consumer offsets.
Local storage risk	Major Critical	xx topics are using local storage	Local storage has many known open-source issues. We are not liable for business losses caused by defects in open-source Kafka or its third-party components, or by improper configuration and use.	The related topic uses local storage.
ECS scheduled reboot	Important Fix as soon as possible	There are scheduled ECS restart events for xx nodes in the cluster.	A scheduled ECS restart event is planned for the cluster and may cause minor fluctuations in service traffic. Please monitor your services accordingly.	If a node in the cluster has a maintenance event, a risk notification is sent to the customer.
A Connector service has been idle for a long time.	General Important	The Connector service will soon start charging. If you do not need to use the service, please release your resources as soon as possible.	The Connector service will soon be a paid service. If you no longer need to use it, please go to Connector Ecosystem Integration/Task List to release your resources as soon as possible.	Go to the address from the prompt to check the connector service.
A message retrieval service has been unused for a long time.	General Important	The Message Retrieval Service will soon be billed. If you do not need this service, please release the resources as soon as possible.	The Message Retrieval service will soon start charging. If you do not need to use this service, go to Message Retrieval to release the resources as soon as possible.	Go to the specified address to view the message retrieval service.
You have a migration to the cloud service that has been inactive for an extended period	General Important	The Cloud Migration service will soon start to incur charges. If you no longer need the service, please release your resources as soon as possible.	The Cloud Migration Service will soon be subject to fees. If you no longer need the service, please go to Migration Tasks to release the resources as soon as possible.	Go to the provided URL to view the cloud migration service.
Message batch too large	Important Fix as soon as possible	xx topics have an issue with oversized message batches, which may cause send requests to be rejected. Please address this issue promptly.	An excessively large message batch will prevent messages from reaching the server. To resolve this, increase the maximum message size on the server or decrease the `max.request.size` parameter on the client. For more information, see Usage Limits.	Check if the `batch.size` configuration of the production client is too large.
Too many messages in a batch	Important Fix as soon as possible	xx topics contain too many messages per Batch, which may cause send requests to be rejected. Please address this issue promptly.	An excessive number of messages in a batch can cause send failures. We recommend that you decrease the `batch.size` value to prevent an overflow. For more information, see Usage limits.	Check if the `batch.size` configuration for the production client is too large, and if individual messages are small.
The server network architecture version is outdated.	Important Fix as soon as possible	The server-side network architecture must be retired because it is outdated and has poor security.	You can create a new serverless, subscription, or pay-as-you-go instance and go to the Message Queue for Apache Kafka console to migrate your old instance to the new one. After the migration, you can unsubscribe from the old instance.	The instance's network architecture is outdated and is scheduled for retirement.

Procedure

Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.
On the Instances page, click the name of the target instance.

On the Instance Details page, click the Instance Risks tab.

On the Instance Risks tab, view the risks for the instance.

Parameter	Description	Example
Risk Type	The type of risk that affects the instance.	Group with Long Consumption Time
Metric Level	The severity level of the metric. Valid values: Repair Required Important General	Important
Risk Status	The status of the risk. Valid values: To Be Fixed Fixed	To Be Fixed
Time of Last Alert	The time when this risk was last detected.	March 31, 2022
Actions	The available actions for a risk. Details: View the details and suggested fixes for the instance risk. In the Actions column of the target risk, click Details. Modify Alert Status: After you fix a risk, you can set Risk Status to Fixed, or ignore an unfixed risk for one month. In the Actions column of the target risk, click Modify Alert Status. Note After you fix a risk, the system no longer sends alert notifications. If the same risk occurs again, the system sends another alert notification after 7 days. Delete: After a risk is fixed and its status is changed to Fixed, you can delete the risk. In the Actions column of the target risk, click Delete. Suggestion: After you set the Risk Status to Fixed, we recommend that you wait for a period before you delete the risk. This prevents new alerts caused by delays in data cleanup. The recommended waiting period is 7 days.	None

References

For other common instance issues and solutions, see FAQ.