SQL aggregation for intelligent inspection - Simple Log Service

The intelligent inspection feature automatically and adaptively inspects service logs to identify anomalies. You can use SQL statements to aggregate metric data for real-time intelligent inspection.

Prerequisites

Collect logs to the source Logstore. For more information, see Data collection overview.
Configure indexes for the source Logstore. For more information, see Create indexes.
Create an Intelligent Anomaly Analysis instance. For more information, see Create an instance.

Create an intelligent inspection job

Go to the job creation page

Log on to the Simple Log Service console.
1. In the Log Application section, click Intelligent Anomaly Analysis.
2. In the instance list, click the target instance.
3. In the left-side navigation pane, click Intelligent Inspection.
4. Click Real-time Inspection.
5. In the Inspection Job section, click Create Now.

Basic information

In the Basic Information section of the Create Intelligent Inspection Job configuration wizard, configure the following parameters, and then click Next.

Parameter	Description
Task Name	The name of the intelligent inspection job.
Project	The project where the source Logstore or Metricstore resides.
Region	The region where the selected project resides.
Logstore Type	The storage type, determined by where your data resides. If your data is stored in a Logstore, select Logstore. If your data is stored in a Metricstore, select Metricstores.
Source Logstore	If you set Logstore Type to Logstore, you must set Source Logstore to the Logstore that contains your source data.
Metricstores	If you set Logstore Type to Metricstores, you must set Metricstores to the Metricstore that contains your source data.
Role	If you granted permissions when creating the instance, the Alibaba Cloud Resource Name (ARN) of the `AliyunLogETLRole` role is automatically displayed.
Target Store	The destination Logstore. Fixed to `internal-ml-log`.

Data feature configuration

In the Data feature configuration section, set Data Type to SQL Aggregation, enter a query and analysis statement, and then configure the related parameters. For more information, see Query overview and Query and analysis overview.

Query statement

* | select __time__ - __time__ % 60 as time, domain, sum(request_size) as request_size from log group by time, domain limit 100000

Entity: domain
Feature: request_size

Parameter	Description
Time	The time field from your source data. By default, Log Service uses the `__time__` field in the Logstore.
Granularity	The observation interval of the data, in seconds. Valid values: 5 to 3600. An interval of 60 seconds or greater is recommended.
Entity	The field used to aggregate data into a time series.
Feature	The field that identifies the feature in your source data.

Algorithm configuration

In the Algorithm configuration section, select an Algorithm. You can select Stream Graph Algorithm or Stream Decomposition Algorithm. The required parameters depend on the algorithm you select.

Stream Graph Algorithm

Parameter	Subparameter	Description
Advanced parameters (required)	Time series segments	The number of segments used to discretize time series values and construct a time-series evolution graph. Default value: 8. Recommended range: [5, 20]. Fewer segments make anomaly detection less sensitive.
	observation length	The number of historical data points to reference during anomaly detection. Default value: 2880. Recommended range: [200, 4000]. If the time series is periodic, set the observation length to cover at least two periods. For example, if the data point interval is 1 minute and the period is 1 day, two periods contain 2,880 data points. Set the observation length to 2880 or greater.
	Period-over-period comparison length	The time range for period-over-period analysis, in days. During anomaly detection, the algorithm analyzes the period-over-period features of the metric. Setting this parameter to 0 disables period-over-period analysis.
	Major capture type	The time-series anomaly type to focus on. Valid values: Upward Spike: The metric value suddenly increases at a single point. Downward Spike: The metric value suddenly decreases at a single point. Upward Shift: The metric value continuously increases to a higher level over several points. Downward Shift: The metric value continuously decreases to a lower level over several points. Upward Trend: The metric value continuously increases over several points. Downward Trend: The metric value continuously decreases over several points.
	Trees	The number of decision trees used by the anomaly detection algorithm.
	Sample size per tree	The number of data samples from the observed data used to construct each decision tree.
	Overall anomaly rate	The estimated proportion of anomalous data in the time series. Recommended range: [0.001, 0.01].
	Minimum window of anomaly type check	The minimum observation sequence length to reference when identifying an anomaly type.
	Maximum window of anomaly type check	The maximum observation sequence length to reference when identifying an anomaly type.
	Minimum window for anomaly confirmation	The minimum sequence length to inspect when confirming an anomaly type.
	Maximum window for anomaly confirmation	The maximum sequence length to inspect when confirming an anomaly type.
Single-dimension feature configuration	-	Configure each feature of the time series to inspect. Settings include: Maximum Value: The upper bound for this feature. Minimum Value: The lower bound for this feature. Normalization: The normalization method for inspecting this feature series. Anomaly focus type: The anomaly pattern to focus on during detection for the feature sequence.
Notification Sensitivity Configuration	-	Configure different notification thresholds for abnormal events detected during different time periods. For example, you can ignore abnormal events during weekly scheduled maintenance.

Stream decomposition algorithm

Configure the algorithm.

Parameter	Subparameter	Description
Automatic period detection	-	Enable automatic period detection for time series data that has a periodic pattern. If the time series has a constant period, disable this feature and set the period length manually.
Period detection frequency	-	This parameter takes effect when automatic period detection is enabled. The algorithm updates the period of the time series at the specified frequency. For example, if you set the period detection frequency to 12 hours, the algorithm detects and updates the series period every 12 hours.
Period length	-	This parameter takes effect when automatic period detection is disabled. Set the period length of the series. If the series has no period, set this value to 0.
Observation length	-	The length of historical data to reference for anomaly detection. If the series is periodic, set the observation length to three times the period length. For example, if the series has a period length of 1 day, set the observation length to 3 days.
Sensitivity	-	A higher sensitivity value results in more detected anomalies and higher anomaly scores. This increases the recall rate for anomalies but decreases the precision rate.
Advanced parameters	Trend component sensitivity	The algorithm decomposes a series into trend, periodicity, and noise components. This parameter adjusts the sensitivity for anomaly detection on the trend component. A higher value results in more detected anomalies and higher anomaly scores, which increases the recall rate but decreases the precision rate.
	Noise component sensitivity	The algorithm decomposes a series into trend, periodicity, and noise components. This parameter adjusts the sensitivity for anomaly detection on the noise component. A higher value results in more detected anomalies and higher anomaly scores, which increases the recall rate but decreases the precision rate.
	Trend component sample length	The algorithm decomposes a series into trend, periodicity, and noise components. If the observation length is too long, trend analysis can be slow. A longer sample length speeds up the analysis but may reduce accuracy. For example, if the sample length is 8, the algorithm analyzes one data point for every 8 points in the original series.
	Periodicity component sample length	The algorithm decomposes a series into trend, periodicity, and noise components. If the observation length is too long, periodicity analysis can be slow. A longer sample length speeds up the analysis but may reduce accuracy. For example, if the sample length is 8, the algorithm analyzes one data point for every 8 points in the original series. Set this value to 5 or less.
	Window length	If the series observation length is too long, anomaly detection can be slow. Set a window length to make the detection algorithm use a sliding window. The algorithm detects anomalies in segments of the data series, which improves detection speed. Set this value to 5000 or less. If you do not need to use a sliding window, set this value to 0.

Preview the algorithm's performance with the current parameter settings.
1. Set a time range to specify the start and end times for the time series. Click Data Query. The query and analysis statements that you set in Data Feature Settings are used to process the data in the specified time range and generate time series data.
2. Select Entity Information and Feature to specify the feature series for detection. Click Preview. The detection algorithm processes the specified feature series and displays the results below. Click Show Parameters to view the current parameter settings for the algorithm.
3. The detection results show the Trend Component Preview, Periodicity Component Preview, and Noise Component Preview. Adjust the anomaly threshold in the Trend Component Preview and Noise Component Preview. Only anomalous events with a score greater than the threshold can generate alerts.

In the Scheduling Settings area, complete the following settings.

Parameter	Description
Start time	The start time for the time series that the algorithm processes. The task reads time-series data and performs detection from this point.
Data latency duration	The maximum delay for writing time-series data to Log Service. The task waits for this duration before reading data, ensuring that the complete time-series data is available.
Model training start time	Optional. If set, the background task begins model training after this time. Defaults to the scheduled start time of the task.
Model training end time	Optional. If not set, the model learns continuously and starts detection after the model training start time. If set, the background task stops updating the model at this time and immediately starts detection.

Alert configuration

In the Alert Configuration section of the Create Intelligent Inspection Job wizard, configure the following parameters, and then click Complete.

Parameter

Description

Alert policy

Merges, mutes, and suppresses generated alerts.

If you select Simple Mode or Standard mode, you do not need to configure an alert policy. Log Service uses the built-in SLS dynamic alert policy (sls.builtin.dynamic) for alert management by default.
If you select Advanced Mode mode, you can select a built-in or custom alert policy for alert management. For more information, see Create an alert policy.

Action policy

Controls alert notification channels and frequency.

When Alert Policy is set to Simple Mode, simply configure an action group.

After you configure an action group, Log Service automatically creates an action policy named Rule Name-Action Policy. This action policy sends notifications for all alerts that the alert monitoring rule triggers. For more information, see Notification channels.

Important
You can modify this action policy on the Action Policy management page. For more information, see Action policies. If you add a condition when modifying the action policy, the Alert Policy automatically changes to Standard.
When Alert Policy is set to Standard or Advanced Mode, you can select a built-in or custom action policy for alert notifications. For more information, see Action policies.

When you set Alert Policy to Advanced Mode, you can also enable or disable Custom Action Policy. For more information, see Dynamic action policy mechanism.

Manage intelligent inspection jobs

View a job: In the job list, click the ID of the target job to view its details.
Edit a job: In the job list, find the target job and click the Edit icon in the Actions column to modify the job configuration.
Delete a job: In the job list, find the target job and click the Delete icon in the Actions column.

Important
Deleted intelligent inspection jobs cannot be recovered. Proceed with caution.

Next steps

Alerts and tag feedback