Application Real-Time Monitoring Service:Dynamic threshold

Scenarios

• Application performance monitoring: monitors the key metrics of a website or service, such as the response time and request speed. If the response time of a service suddenly exceeds the dynamic thresholds, the system immediately issues an exception warning. This enables website administrators to quickly locate and solve the problem.

• Server resource optimization: monitors the CPU utilization and memory usage of a server. If the resource usage of a server continuously exceeds the dynamic thresholds, the system automatically generates an exception event. This helps you adjust resource allocation in a timely manner to prevent system crashes.

• Application connection pool analysis: monitors key metrics, such as the query speed and the number of concurrent connections. If some metrics of a thread exceed the dynamic thresholds, the system automatically triggers an exception event to optimize program performance in a timely manner.

• Microservice model monitoring: monitors resource usage and response performance of each microservice. The interactions and dependencies among microservices are complex. With dynamic thresholds, if an exception occurs in a microservice, you can quickly locate the problem to ensure the stability of the entire microservice.

Example:

Assume that the normal page view of a website from 10:00 to 18:00 is greater than 1,000. If the page view is still greater than 1,000 from 22:00 to 06:00, the website is likely to be attacked. In this case, the expected data range of the page view changes over time. If you configure a static threshold value 1000, you can receive alert notifications when the page view is less than 1000 during the day. However, if the website is attacked at night, alerts are not triggered. In this case, you can use dynamic thresholds to intelligently update the data range and detect anomalies.

Prerequisites

Your application is monitored in Application Monitoring eBPF Edition. For more information, see Connect an application to Application Monitoring eBPF Edition and Manually connect an application to Application Monitoring eBPF Edition.

Configure interval detection

1. Log on to the ARMS console. In the left-side navigation pane, choose Application Monitoring eBPF > Application Monitoring eBPF Edition Alert Rules.

2. On the Alert Rules page, click Create Alert Rule for Application Real-time Monitoring (eBPF).

3. On the Create Alert Rule for Application Real-time Monitoring eBPF page, set Alert Name and Alert Detection Type to Interval Detection.

   Note

   For more information about how to configure threshold detection, see Threshold detection.

4. In the Alert Object section, select an alert application, metric type, and filter conditions.

   Parameter	Description
   Select Applications	Select the application that you want to detect. Currently, interval detection only supports configuring alerts for a single application.
   Metric type	Select the type of the metric that you want to detect. For more information, see Alert metrics. After you select a metric, the system automatically calculates the upper and lower boundaries and renders the metric in real time. You can preview the metric trend in the Alert Condition section. Note The values of the Alert Condition and Filter Condition parameters vary based on the value of the Metric Type parameter. The initial rendering takes a long time, about 2 to 4 seconds. For more information about how to calculate the upper and lower boundary values, see How to calculate threshold intervals.
   Filter Conditions	The metrics are further filtered to shorten the monitoring scope. The method that is used to filter the metrics for which alerts are generated. Valid values: Traversal: traverses all values of the metric type that you specify. No: calculates the sum of all values of the metric type that you specify. =: The alert content shows only the specified values of the dimension. !=: filters the values of the metric type that you specify. The filtered values must be unequal to the value that you specify in the text box on the right. Contain: filters the values of the metric type that you specify. The filtered values must contain the value that you specify in the text box on the right. Do Not Contain: filters the values of the metric type that you specify. The filtered values cannot contain the value that you specify in the text box on the right. Match Regular Expression: filters the values of the metric type that you specify. The filtered values must conform to the regular expression that you specify in the text box on the right.

   

5. In the Alert Rules section, configure Alert Conditions.

   Parameter	Description
   Alert Trigger Mode	Interval detection only supports single-condition triggers, and does not support the combination of multiple trigger modes.
   Alert Conditions	Configure specific alert conditions, including the following factors: Last X Minutes: the time range to monitor. You can select a maximum of 60 minutes. Metric Measures: indicates the data or values that can be quantified for a metric. The metrics can be measured based on different metric types, such as the number of calls and the call response time. Aggregation method: The calculation of metric data, including the average value, maximum value, and minimum value, depends on the metric and measure. Comparison method: compares calculated data to find abnormal points. Interval detection includes three comparison methods: Outside the upper and lower bounds of the dynamic threshold: The system automatically calculates the upper and lower bounds of the current time. If a data point is found to be outside the upper or lower bounds, the data is abnormal and an alert is triggered. Dynamic Threshold Outside Upper Bound: The system automatically calculates the upper and lower bounds of the current time. If a data point is found to be outside the upper bound, the data is abnormal and an alert is triggered. Outside the lower bound of the dynamic threshold: The system automatically calculates the upper and lower bounds of the current time. If a data point is found to be outside the lower bound, the data is abnormal and an alert is triggered. Alert Level: Set the severity level from P1 to P4. In the data preview area, the blue line represents the actual data points, and the green area is the upper and lower boundary range.
   Tolerance	The boundary is stretched or shrunk based on the upper and lower boundaries that are automatically calculated by the system. If the tolerance is higher (sliding to the right), the upper and lower boundaries are wider, and the higher the threshold for the data to be diagnosed as abnormal, the lower the threshold for the data to be diagnosed as abnormal, and the lower the threshold for the data to be diagnosed as abnormal, and the lower the threshold for the data to be diagnosed as abnormal.
   Alert count prediction	View the number of times that the metric is expected to exceed the threshold within the selected time period. Click a specific alert value to query the metric value that triggers an alert at a historical point in time. Each time you create or modify an alert rule, you recommend use the Alert count prediction feature. This feature uses algorithms to analyze historical data and predict the number of alerts within the specified time range. This allows you to adjust the threshold. For more information, see Alert quantity prediction.

6. Set Notification Policy and Advanced Alert Settings.

   Parameter	Description
   create a notification policy	If you do not specify a notification rule, no alert is sent when the alert is triggered. Alerts are sent only when the matching rule of the notification policy is triggered. If you specify a notification rule, ARMS sends alert notifications by using the notification method specified in the notification policy. You can select an existing notification policy or create a notification policy. For more information, see Create and manage a notification policy.
   Advanced Alert Settings
   No data	This parameter is used to fix data anomalies, such as no data, abnormal composite metrics, and abnormal period-over-period comparison results. If the metric data does not meet the specified conditions, the metric data is automatically changed to 0 or 1, or the alert is not triggered. For more information, see Terminologies.

7. Click Save.

Threshold calculation

The dynamic thresholds of ARMS are mainly developed based on the Prophet algorithm. After dynamic thresholds are enabled, ARMS analyzes historical data of last 7 days every 24 hours, extracts the tendency and seasonality, and then draws a trend chart for the predicted data in the next 24 hours. At the same time, an expected data range is calculated based on the fluctuations of the metric. When you configure dynamic thresholds, you can preview the upper and lower boundaries calculated by the algorithm. In the following figure, the color blue represents data points, and the color green specifies an allowed data range.

Different from static thresholds, dynamic thresholds do not need to be updated by manually editing alert rules even if the expected data range of a metric changes over time. This is because ARMS analyzes metric trends once a day and predicts the upper and lower boundaries only of the next day.

Alert quantity prediction

The alert quantity prediction feature uses an algorithm to analyze historical data, display the time when historical alerts occur, and then predicts the number of alerts within a specified period of time. The feature helps you configure static thresholds or improve alert sensitivity for dynamic thresholds.

Implementation

Based on metric data in the last 24 hours, ARMS calculates the number of times that each threshold of a metric is exceeded to predict the quantity of alerts in the future. In addition, ARMS provides the metric details, including the specific time when each threshold is exceeded. You can adjust thresholds based on your business requirements.