Group evaluation is a parameter in an alert rule. When the alerting system computes the results of a query and analysis, you can group the results by a specific field. The alerting system then evaluates each group against the trigger condition independently, generating a separate alert. This lets you use a single alert rule to monitor multiple targets and manage alerts and events independently for each group.
-
After you configure group evaluation, a single evaluation can generate up to 100 alert groups. If more than 100 groups are generated, the system randomly selects 100 of them and sends them to the alert policy.
-
When you select a field for group evaluation, choose one that uniquely identifies the monitored entity and has a limited number of possible values. Do not select fields that fail to distinguish between entities. Using such fields can create numerous groups, each triggering a separate alert. This can cause an alert storm, leading you to miss critical alert notifications.
For example, for Nginx logs, use fields like host and method. For OSS access logs, use the bucket field. Do not use fields like request_time or body_size from Nginx logs, or the err_cnt field from error logs.
Example 1: Group monitoring for time series data
For example, you store metric data from multiple servers in a single time series database. You want Log Service to send a separate alert for each server when its CPU utilization (cpu_util) exceeds 95%. To achieve this, you can configure group evaluation when you create an alert rule.
The configuration is as follows:
-
Query and statistics:
* | select promql_query_range('cpu_util') from metrics limit 1000This query calculates CPU utilization.
-
Group evaluation: Auto label
Log Service automatically groups query and analysis results for time series data.
-
Trigger Condition: Data Matching, value > 95, Severity: High
The system triggers a high-severity alert when the value in the query and analysis results exceeds 95.
-
Add Annotation: Configure annotations such as the alert title and description. You can reference field variables, such as ${host}, in the annotations. For more information, see Add labels and annotations.
Set the title to CPU utilization for host ${host} has spiked and the desc to CPU utilization for host ${host} has reached ${value}.
Example 2: Group monitoring for logs
For example, when you monitor OSS access logs, you want to receive a separate alert for each bucket that experiences more than 1,000 HTTP 500 errors per minute. To achieve this, you can configure group evaluation when you create an alert rule.
The configuration is as follows:
-
Query and statistics:
http_status=500 | select bucket,count(1) as pv group by bucket having pv >1000 order by pv descThis query finds buckets with more than 1,000 HTTP 500 errors.
-
Group evaluation: Custom label, bucket
The system groups the query and analysis results by bucket.
-
Trigger Condition:
-
Condition 1: Data Matching, pv > 3000, Severity: High
The system triggers a high-severity alert when the pv value in the query and analysis results exceeds 3,000.
-
Condition 2: Data Exists, Severity: Medium
The system triggers a medium-severity alert if the query and analysis returns any results.
-
-
Add Annotation: Configure annotations such as the alert title and description. You can reference field variables, such as ${pv}, in the annotations. For more information, see Add labels and annotations.
Set the title to ${bucket} bucket is experiencing 500 errors and the desc to Error count: ${pv}.