How to configure alert rules for ApsaraDB for ClickHouse - ApsaraDB for ClickHouse

ApsaraDB for ClickHouse supports setting up alerts for key cluster metrics. When the value of a metric is outside the configured threshold, the system sends an alert notification to the contacts in an alert group to help you quickly identify problems. This topic describes how to set up alert rules for an ApsaraDB for ClickHouse cluster.

Prerequisites

The cluster is a Community Edition or Enterprise Edition cluster.
Alerting for ApsaraDB for ClickHouse depends on Application Real-Time Monitoring Service (ARMS). If you are a RAM user, you must have the AliyunARMSFullAccess permission to manage ARMS. For more information about how to grant this permission, see Manage permissions for a RAM user.

Usage notes

You cannot view monitoring information for Enterprise Edition clusters in the CloudMonitor console.

Procedure

ApsaraDB for ClickHouse console

Note

If your cluster meets both of the following conditions, see Procedure for the old console.

The cluster was created before December 1, 2021.
The cluster resides in the China (Qingdao) or China (Hohhot) region.

Log on to the ApsaraDB for ClickHouse console.
In the upper-left corner of the page, select the region where the target cluster is located.
On the Clusters page, select Clusters of Community-compatible Edition or Enterprise Edition Clusters, and then click the ID of the target cluster.
In the left-side navigation pane, click Monitoring and Alerting.
On the Monitoring Details page, click the Cluster Alerting tab.
Click the button to create a cluster alert rule.

Note
For Community Edition clusters, the button is Create ClickHouse Alert Rule.

For Enterprise Edition clusters, the button is Create Enterprise Edition ClickHouse Alert Rule.

On the Create ClickHouse Alert Rule or Create Enterprise Edition ClickHouse Alert Rule page, configure the following parameters.

Note

ApsaraDB for ClickHouse provides two check types for alert rules: static threshold and custom PromQL. Parameters vary depending on the selected check type.

The static threshold check type lets you quickly create an alert rule by selecting from preset metrics.
The custom PromQL check type is for creating alert rules for metrics that are not preset by the system.

Static threshold

Parameter	Description		Example
Alert Name	The name of the alert rule.		CPU utilization alert
Check Type	Select Static Threshold.		Static Threshold
Cluster	The target cluster for the alert rule.		cc-bp1lxbo89u95****
Alert group	Select an alert group. The available alert groups vary based on the Prometheus instance type.		ClickHouse Open Source Edition
Alert Metric	The metric to monitor. The available metrics depend on the selected Alert group.		CPU utilization
Alert Condition	Configure the trigger condition for the selected metric.		The alert is triggered when the CPU utilization is `greater than` 80%.
Filter Conditions	No filter		None
Preview Data	The Preview Data section displays the PromQL statement that corresponds to the alert condition and plots the values of the monitoring metrics configured for the current alert rule as a time series curve. By default, only the real-time value of one resource is displayed. You can select a target resource and a time range to view values for different resources over different periods. Note The alert threshold is shown as a red horizontal line in the chart. Time series data that meets the threshold is displayed in dark red, and data that does not meet the threshold is displayed in blue. Hover over the time series chart to view resource details at a specific point in time. On the time series chart, select a time range to view the time series for the specified period.		None
Duration	An alert is triggered immediately when the condition is met: An alert is triggered if any data point reaches the threshold. An alert is triggered only when the condition persists for N minutes.		1
Alert Level	The severity level of the alert. The default level isDefault. The severity levels, from lowest to highest, are Default, P4, P3, P2, and P1.		P2
Alert Content	The content of the alert notification. You can use Go template syntax to include variables.		`Node: {{$labels.pod_name}} CPU utilization {{$labels.metrics_params_opt_label_value}} {{$labels.metrics_params_value}}%, current value {{ printf "%.2f" $value }}%`
Alert Notification	Simple mode	Alert Contact: the object to be notified when an alert is triggered. Notification period: the time period during which alert notifications are sent. Alerts that are triggered outside this period are ignored. Repetition policy: the policy for sending repeated notifications when an alert is triggered.	Alert Contact: Select contacts and specify notification methods (Email, Phone, SMS, or Group @). Click + Create Notification Object to add a new contact. Notification Period: Set the start and end time (for example, 00:00~23:59). Cross-day configuration within 24 hours is supported. Whether to Resend Notifications: Select No escalation policy required. Send only once when the alert is not recovered, or select When the alert is not recovered, send repeated notifications at specified intervals.
	Standard Mode	Specify a policy by using Quickly Specify Notification Policy. Important The Quickly Specify Notification Policy option ensures that events from the current alert rule are matched by the selected notification policy. However, these events might also be matched by other notification policies that use fuzzy matching. The relationship between alert rule events and notification policies is many-to-many. Do not specify a notification rule. Select an existing notification policy. Note You can click Create Notification Policy to create a notification policy. For more information, see Notification policies.	Do not specify a notification policy
Advanced Settings
Alert check period	The frequency for checking the alert condition. The minimum and default value is 1 minute.		1 minute
Data complete check	Wait until data is collected, transmitted, and stored before an alert check is performed. The default value is Yes. In most cases, you do not need to change this value.		Yes
Tag	Set tags for the alert rule. The tags can be used as matching conditions in notification policies.		None
Comment	Add annotations for the alert rule.		None

Custom PromQL

Parameter	Description		Example
Alert Name	The name of the alert rule.		Pod CPU utilization is greater than 8%
Check Type	Set to custom PromQL.		custom PromQL
Cluster	The target cluster for the alert rule.		cc-bp1lxbo89u95****
Reference Alert Contact Group	Select an alert group. The available alert groups vary based on the Prometheus instance type.		ClickHouse Open Source Edition
Reference Metrics	Optional. Provides PromQL templates for common metrics. You can select a template and then modify it. The available Reference alert metric options are automatically filtered based on the selected Prometheus instance type.		Number of HTTP connections
Custom PromQL statement	The PromQL expression that defines the alert condition.		`clickhouse_http_conn_usage_count{} > 1000`
Preview Data
Duration	An alert is triggered immediately when the condition is met: An alert is triggered if any data point reaches the threshold. An alert is triggered only when the condition persists for N minutes.		1
Alert Level	The severity level of the alert. The default level isDefault. The severity levels, from lowest to highest, are Default, P4, P3, P2, and P1.		Default
Alert Content	The content of the alert notification. You can use Go template syntax to include variables.		`Namespace: {{$labels.namespace}}/Pod: {{$labels.pod_name}}/Disk device: {{$labels.device}} utilization exceeds 90%, current value {{ printf "%.2f" $value }}%`
Alert Notification	Simple mode	Alert Contact: the object to be notified when an alert is triggered. Notification period: the time period during which alert notifications are sent. Alerts that are triggered outside this period are ignored. Whether to Resend Notifications: the policy for sending repeated notifications when an alert is triggered.
	Standard Mode	Specify a policy by using Quickly Specify Notification Policy. Important The Quickly Specify Notification Policy option ensures that events from the current alert rule are matched by the selected notification policy. However, these events might also be matched by other notification policies that use fuzzy matching. The relationship between alert rule events and notification policies is many-to-many. Do not specify a notification rule. Select an existing notification policy. Note You can click Create Notification Policy to create a notification policy. For more information, see Notification policies.	Do not specify a notification policy
Advanced Settings
Alert check period	The frequency for checking the alert condition. The minimum and default value is 1 minute.		1 minute
Check When Data Is Complete	Wait until data is collected, transmitted, and stored before an alert check is performed. The default value is Yes. In most cases, you do not need to change this value.		Yes
Tag	Set tags for the alert rule. The tags can be used as matching conditions in notification policies.		None
Comment	Add annotations for the alert rule.		None

After you configure the parameters, click Completed.

CloudMonitor console

Log on to the CloudMonitor console.
In the left-side navigation pane, choose Alerts > Alert Rules.
On the Alert Rules page, click Create Alert Rule.

In the Create Alert Rule panel, configure the following parameters.

Parameter		Description
Products		Select the cluster type for ApsaraDB for ClickHouse. Valid values: ApsaraDB for ClickHouse: Community Edition clusters purchased before December 1, 2021. ApsaraDB for ClickHouse Community Edition: Community Edition clusters purchased on or after December 1, 2021.
Resource Range		The scope of resources to which the alert rule applies. Valid values: All Resources: The alert rule applies to all resources of the specified cloud service, including newly added resources. Application Group: The alert rule applies to all resources within a specified application group of the cloud service, including newly added resources. Instance: The alert rule applies to specific resources of the cloud service.
Description		Defines the trigger conditions for the alert. To configure the rule description, follow these steps: Click Add Rule. In the Set Rule Description panel, enter a Rule Name and then set the rule conditions. Single Metric: Select a metric and then set a threshold and an alert level for it. Multiple Metrics: Select an alert level and then set alert conditions for two or more metrics. Dynamic Threshold: For more information about intelligent thresholds, see Overview and Create an alert rule for an intelligent threshold. Note The intelligent threshold feature is in beta testing. You must submit a ticket to apply for access. Click OK. Note For information about how to set complex alert conditions, see Expressions for alert rules.
Mute Period		The delay before CloudMonitor resends a notification for an unresolved alert. Valid values: 5 minutes, 15 minutes, 30 minutes, 60 minutes, 3 hours, 6 hours, 12 hours, and 24 hours. When a monitored metric reaches its threshold, an alert is sent. If the metric continues to exceed the threshold during the mute period, no new alerts are sent. If the metric is still in an alert state after the mute period, CloudMonitor sends another alert notification. For example, if you set Mute Period to 12 Hours, CloudMonitor resends the notification after 12 hours if the alert is not resolved.
Effective Time		The time period during which the alert rule is active. Notifications are sent only for alerts triggered during this period. Note If an alert is triggered outside the effective period, no notification is sent, but the alert is still recorded in the Alert History list.
Alert Contact Group		The contact group to which alert notifications are sent. Alert notifications for an application group are sent to the contacts in the specified contact group. A contact group can contain one or more contacts. For information about how to create contacts and contact groups, see Create an alert contact or a contact group.
Tag		The tags of the alert rule, which consist of a tag key and a tag value. Note You can add up to six tags.
Advanced Settings	Alert Callback	A publicly accessible URL that receives alert information from CloudMonitor through POST requests. Only the HTTP protocol is supported. For information about how to configure an alert callback, see Use a webhook to send threshold-triggered alerts. To test the connectivity of the alert callback URL, perform the following steps: Click Test next to the callback URL. In the Webhook Test panel, you can check the webhook status code and test result details to diagnose connectivity issues. Note You can also set the Callback Template Type and Language for the webhook, and then click Test again to view the corresponding test results. Click Disable.
	Auto Scaling	If you enable Auto Scaling, the corresponding scaling rule is triggered when an alert occurs. You must configure the Region, ESS Group, and ESS Rule for Auto Scaling. For information about how to create a scaling group, see Manage scaling groups. For information about how to create a scaling rule, see Manage scaling rules.
	Log Service	If you enable Log Service, alert information is sent to a Logstore in Log Service when an alert occurs. You must configure the Region, ProjectName, and Logstore for Log Service. For information about how to create a Project and a Logstore, see Use iLogtail to collect and analyze text logs from ECS instances.
	Simple Message Queue (formerly MNS) - Topic	If you enable Message Service MNS-Topic, alert information is sent to a topic in Message Service (MNS) when an alert occurs. To enable this feature, you must configure the region and topic for Message Service (MNS). For information about how to create a topic, see Create a topic.
	Function Compute	If you enable Function Compute, alert notifications are sent to Function Compute for formatting when an alert occurs. You must configure the region, service, and function for Function Compute. For information about how to create a service and a function, see Create a function.
	NoDataPolicy	The method for handling alerts when no monitoring data is available. Valid values: Do not do anything (Default) Send alert notifications Treat as recovered

(Optional) Old console

Log on to the ApsaraDB for ClickHouse console.
In the top-left corner of the page, select the target cluster's region.
On the Clusters page, select Default Instance List, and click the target cluster ID.
Click Monitoring Details in the left navigation bar.
In the upper-right navigation bar, click Alert Monitoring.
In the CloudMonitor console, click Alerts > Alert Rules in the left navigation bar.
On the Threshold Value Alert tab, click Create Alert Rule.

On the Create Alert Rule page, configure the following parameters.

Configure Associate with Resources. The following table describes the parameters.

Parameter	Description
Products	Select ApsaraDB for ClickHouse from the drop-down list.
Resource Range	All Resources: An alert is sent if any cluster of the selected product meets the alert condition. Cluster: An alert is sent only if a specified cluster meets the alert condition.
Region	This parameter is required when Resource Range is set to Cluster. Select the region where the cluster is located.
Cluster	This parameter is required when Resource Range is set to Cluster. Select one or more cluster IDs.

Configure Set Alert Rules. For more information, see Create an alert rule.
Set the Notification Method.

Note
You must create a contact group in advance. For more information, see Create an alert contact or a contact group.

After you configure the preceding parameters, click Confirm. The alert rule then automatically takes effect.

Next steps

To manage alert rules created in the ApsaraDB for ClickHouse console, see Manage alert rules.

ApsaraDB for ClickHouse:Configure alert rules