Cloud Monitor:Why is a threshold-triggered alert rule triggered with a delay?

You must be familiar with the following two concepts:

• Period of a metric

  The period of a metric indicates the interval between two consecutive data points and the aggregation period of metric values. For example, if the period of a metric is 60 seconds, a data point is generated every 60 seconds. A metric value is the last value, or the aggregate value that is calculated based on the metric data of the previous 60 seconds.

• Alert detection period

  The detection period of an alert rule is determined by the period of a metric. If a multi-metric expression contains multiple periods, the shortest period prevails. For example, if the detection period is 60 seconds, Alert Service queries the latest metric data every 60 seconds.

If the period of a metric in an alert rule is long (for example, 300 or 900 seconds), writing the metric data to a disk may be delayed or the data point may be slightly earlier than the point in time at which an alert is detected. As a result, the alert may be triggered with a delay of one detection period. If the period of a metric is short, no significant delay occurs.

The availability monitoring alerts and group process monitoring alerts of application groups, metric-based alerts of cloud services, site monitoring alerts, and alerts created by using PromQL statements for Hybrid Cloud Monitoring have expressions with fixed thresholds, regardless of single metrics, multiple metrics, complex expressions, and period-based comparison. Simple expressions for single metrics are used in the following two issues:

Issue 1: The metric data is reported periodically. The data meets the alert condition for N consecutive times, but an alert is triggered with a delay of nearly one period.

Cause

The point in time at which an alert is detected is slightly earlier than the data point. As a result, the alert is triggered with a delay of nearly one period.

Example

Assume that the period of a metric is 300 seconds, the detection period of the alert is also 300 seconds. The alert condition is that an alert is triggered if the average metric value is greater than the threshold for three consecutive periods. The following table lists the data points.

Assume that the alert is detected for the first time at 10:06:00, which is slightly earlier than the data point 10:06:30. The metric data queried at 10:06:00 is value 5 of the last period, which does not reach the metric threshold.

• The alert is detected for the second time at 10:11:00. The metric data queried is value 15, which reaches the metric threshold for the first time.

• The alert is detected for the third time at 10:16:00. The metric data queried is value 14, which reaches the metric threshold for the second time.

• The alert is detected for the fourth time at 10:21:00. The metric data queried is value 13, which reaches the metric threshold for the third time. An alert notification is sent.

From the above data, the customer considers that the metric data that meets the threshold for the third time is supposed to appear at 10:16:30. However, the alert is triggered with a delay of 4.5 minutes, which is close to one period.

Issue 2: The metric data is reported sparsely and aperiodically. An alert is triggered with a delay of one period.

Cause

If metric data is reported aperiodicically or sparsely, writing the data to a disk is delayed.

Example

• Example 1: Assume that the period of a metric is 300 or 900 seconds. If the data is reported sparsely or aperiodicically, the data point may have a delay of up to 5 minutes. For example, if you query the data of data point 10:00:00 at 10:01:00, no result is returned. If you query the data of data point 10:00:00 at 10:05:00, a result is returned. Therefore, the current data may be unavailable at the point in time when the alert is detected, and the alert is triggered with a delay of one period.

• Example 2: The alert condition is that an alert is triggered if the threshold is greater than 10 for once. The following table lists the alert detection process.