Simple Log Service intelligent inspection uses three algorithms to detect anomalies in metrics and business logs: stream graph, stream decomposition, and supervised anomaly detection. Each algorithm suits different scenarios based on data volume, noise level, and cyclic patterns.
Stream graph algorithm
The stream graph algorithm is based on Time-Series Event Prediction with Evolutionary State Graph (Time2Graph). It reduces data noise and calculates the offset of abnormal samples. This algorithm suits scenarios with a large number of noisy time series and minimal cyclic changes.
Scenarios
The stream graph algorithm analyzes each sample in real time using online machine learning. It identifies anomalies in the following time series types:
-
Machine-level metrics, such as CPU utilization, memory usage, and disk read and write speeds
-
Performance metrics, such as queries per second (QPS), traffic volume, success rates, and latency
-
Golden metrics
Parameters
Configure stream graph parameters in the Algorithm Configurations section of the Create Intelligent Inspection Job wizard. Use SQL statements to aggregate metrics for real-time inspection.
|
Parameter |
Subparameter |
Description |
|
(Required) Advanced Parameters |
Time Series Segments |
The number of segments into which the time series is discretized. Discretization helps construct metric charts.
|
|
Observation Length |
The number of historical samples used for anomaly detection.
|
|
|
Period-over-period Comparison Length |
The period (in days) for period-over-period comparison during anomaly detection. Set to 0 to disable period-over-period analysis. |
|
|
Major Capture Type |
The anomaly types to prioritize. Valid values:
|
|
|
Trees |
The number of decision trees used for auxiliary anomaly inspection. |
|
|
Sample Size per Tree |
The number of samples collected per detection tree during construction. |
|
|
Overall Anomaly Rate |
The estimated anomaly rate in the time series. Valid values: 0.001 to 0.01. |
|
|
Minimum Window of Anomaly Type Check |
The minimum time series length referenced during anomaly capture. |
|
|
Maximum Window of Anomaly Type Check |
The maximum time series length referenced during anomaly capture. |
|
|
Minimum Window for Anomaly Confirmation |
The minimum time series length inspected during anomaly confirmation. |
|
|
Maximum Window for Anomaly Confirmation |
The maximum time series length inspected during anomaly confirmation. |
|
|
Single-dimension Feature Configuration |
- |
The time series features to inspect. Configure each feature separately:
|
|
Notification Sensitivity Configuration |
- |
The alert notification threshold. Configure different thresholds for different time periods. For example, suppress alerts during scheduled maintenance windows. |
Stream decomposition algorithm
The stream decomposition algorithm is based on RobustSTL: A Robust Seasonal-Trend Decomposition Algorithm for Long Time Series. It supports batch processing at higher cost than the stream graph algorithm and suits precise inspection of a small number of performance metrics. For large datasets, split data into batches or use the stream graph algorithm.
Scenarios
Use the stream decomposition algorithm for data with major cyclic changes, such as performance metrics with strong periodic patterns.
Examples of cyclic data: game visit counts and customer order volumes.
Parameters
Configure stream decomposition parameters in the Algorithm Configurations section of the Create Intelligent Inspection Job wizard. Use SQL statements to aggregate metrics for real-time inspection.
-
Configure the algorithm.
Parameter
Subparameter
Description
Automatic periodic detection
-
Specifies whether to automatically detect the period of the time series. This is suitable for time series data with seasonality. If the seasonality of the time series is consistent, we recommend that you disable this feature and manually set the period length.
Periodic detection frequency
-
This parameter takes effect when Automatic periodic detection is enabled. The algorithm periodically updates the period of the time series based on the specified frequency. For example, if the frequency is 12 hours, the algorithm automatically detects and updates the period of the sequence every 12 hours.
Period length
-
This parameter takes effect when Automatic periodic detection is disabled. Set the duration of the sequence period. If the sequence has no period, set this to 0.
Observation length
-
The length of historical data used for anomaly detection. If the sequence contains seasonality, we recommend that the observation length be three times the sequence period length. For example, if the sequence period is 1 day, set the observation length to 3 days.
Sensitivity
-
The higher the sensitivity, the more anomalies are detected and the higher the anomaly scores. This increases the recall rate but may lower the precision.
Advanced parameters
Trend component sensitivity
The algorithm decomposes the sequence into trend, seasonal, and noise components. The higher the sensitivity for the trend component, the more anomalies are detected and the higher the anomaly scores during trend component anomaly detection. This increases the recall rate but may lower the precision.
Noise component sensitivity
The algorithm decomposes the sequence into trend, seasonal, and noise components. The higher the sensitivity for the noise component, the more anomalies are detected and the higher the anomaly scores during noise component anomaly detection. This increases the recall rate but may lower the precision.
Trend component sampling step
The algorithm decomposes the sequence into trend, seasonal, and noise components. If the sequence observation length is too long, the trend component analysis can be slow. A larger sampling step for the trend component speeds up the analysis but may reduce the detection accuracy. For example, if you set the trend component sampling step to 8, one data point is sampled for every eight points in the original sequence for trend analysis.
Seasonal component sampling step
The algorithm decomposes the sequence into trend, seasonal, and noise components. If the sequence observation length is too long, the seasonal component analysis can be slow. A larger sampling step for the seasonal component speeds up the analysis but may reduce the detection accuracy. For example, if you set the seasonal component sampling step to 8, one data point is sampled for every eight points in the original sequence for seasonal analysis. We recommend that you set this to a value of 5 or less.
Window length
If the sequence observation length is too long, anomaly detection can be slow. Set a window length to make the detection algorithm detect the data sequence in segments by using a sliding window, which improves detection speed. We recommend that you set this to a value of 5,000 or less. If you do not want to use a sliding window for detection, set this to 0.
-
In the preview section, test the algorithm and view the results based on the current parameter settings.
-
Set a time range to define the start and end times for the time series to be detected. Click Data Query. The system processes data within this range by using the query statement from the Data Feature Settings step to generate the time series data.
-
Select an Entity and a Feature to define the feature series to be detected. Click Preview. The detection algorithm processes the specified feature series and displays the results below. Click Display Parameters to view the current algorithm configuration.
-
The detection results include Trend Component Preview, Seasonal Component Preview, and Noise Component Preview. You can adjust the anomaly thresholds in the Trend Component Preview and Noise Component Preview sections. An alert is triggered only when an anomaly's score exceeds its corresponding threshold.
-
Supervised anomaly detection algorithm
Supervised anomaly detection algorithm
The supervised anomaly detection algorithm constructs features from time series data and uses these features along with anomaly labels to train classification models such as decision trees and random forests. The trained models then perform anomaly detection.
Scenarios
The supervised anomaly detection algorithm suits labeled time series data that the stream graph or stream decomposition algorithms cannot handle effectively.
Parameters
Configure supervised anomaly detection parameters in the Algorithm Configurations section of the Create Model Training Job wizard. Use SQL statements to aggregate metrics for model training.