如何通过SQL聚合指标数据进行智能巡检 - Simple Log Service

The intelligent inspection feature automatically detects anomalies in your service logs. You can create an intelligent inspection job that uses SQL to aggregate metric data for real-time detection.

Prerequisites

Collect and store logs in the source Logstore. For more information, see Data collection.
Create an index for the source Logstore. For more information, see Create indexes.
Create an Intelligent Anomaly Analysis instance. For more information, see Create an instance.

Create an intelligent inspection job

Navigate to the job creation page

Log on to the Simple Log Service console.
Navigate to the job creation page.
1. In the Log Application section, click Intelligent Anomaly Analysis.
2. In the instance list, click the target instance.
3. In the left-side navigation pane, click Intelligent Inspection.
4. Click Real-time Detection.
5. In the Inspection Job section, click Create Now.

Basic information

In the Basic Information section of the Create Intelligent Inspection Job wizard, complete the following configurations, and then click Next.

Parameter	Description
Task Name	Enter a name for the intelligent inspection job.
Project	Select the Project that contains the source Logstore or Metricstore.
Region	Displays the region of the selected Project.
Logstore Type	Select the type of data store. If your data is stored in a Logstore, select Logstore. If your data is stored in a Metricstore, select Metricstores.
Source Logstore	When Logstore Type is set to Logstore, you must set Source Logstore, which is the Logstore where your source data is located.
Metricstores	When Logstore Type is set to Metricstores, you must set Metricstores to the Metricstore that contains your source data.
Role	If you completed authorization when you created the instance, the ARN of the `AliyunLogETLRole` role is automatically displayed.
Target Store	The destination Logstore. This is fixed as `internal-ml-log`.

Data feature configuration

In the Data feature configuration step, for Data Type, select Format Data with SQL. Enter a query and analysis statement and configure the parameters as described in the following table. For more information, see Query overview and Query and analysis overview.

Sample query statement

* | select __time__ - __time__ % 60 as time, domain, sum(request_size) as request_size from log group by time, domain limit 100000

Entity: domain
Feature: request_size

Parameter	Description
Time	The field that represents time in the source data. By default, Log Service uses the `__time__` field in the Logstore.
Granularity	The interval at which data is observed, in seconds. The value must be between 5 and 3600. A value of 60 or greater is recommended.
Entity	The field that identifies a specific entity in the source data. Data is aggregated by this entity to generate a time series.
Feature	The field that identifies feature data in the source data.

Algorithm configuration

In the Algorithm configuration step, select an Algorithm. The available options are the Stream Graph Algorithm and Stream Decomposition Algorithm. The parameters vary depending on the selected algorithm.

Stream graph algorithm

Parameter	Subparameter	Description
Advanced parameters (Required)	Time series segments	The number of segments that time series values are divided into based on their magnitude. This is used to discretize the time series and build a time-series evolution graph. Default value: 8. Recommended range: [5, 20]. A smaller number of segments makes anomaly detection less sensitive.
	Observation length	The number of historical data points to observe for anomaly detection. Default value: 2880. Recommended range: [200, 4000]. If the time series to be detected has seasonality, we recommend that the observation length covers at least two full periods of data points. For example, if a time series has a 1-minute granularity and a 1-day period, two periods contain 2,880 data points. In this case, we recommend that you set the observation length to 2880 or greater.
	Period-over-period comparison length	The duration for period-over-period analysis, in days. Anomaly detection focuses on analyzing the period-over-period features of the metric being detected. If you set this parameter to 0, the algorithm does not perform a period-over-period analysis.
	Major capture type	The types of time series anomalies to focus on. Valid values: Upward Spike: A single-point surge in the metric value. Downward Spike: A single-point drop in the metric value. Upward Shift: The metric value rises and stays on a higher plateau across multiple points. Downward Shift: The metric value falls and stays on a lower plateau across multiple points. Upward Trend: The metric value continuously increases over several consecutive points. Downward Trend: The metric value continuously decreases over several consecutive points.
	Trees	The number of decision trees used for auxiliary detection.
	Sample size per tree	The number of data samples to use from the observed data when constructing a decision tree for anomaly detection.
	Overall anomaly rate	An estimate of the proportion of anomalous data in the time series. A reasonable range is [0.001, 0.01].
	Minimum window of anomaly type check	The minimum length of the observed sequence used to capture the anomaly pattern of the time series.
	Maximum window of anomaly type check	The maximum length of the observed sequence used to capture the anomaly pattern of the time series.
	Minimum window for anomaly confirmation	The minimum length of the sequence to be detected when capturing the anomaly pattern of the time series.
	Maximum window for anomaly confirmation	The maximum length of the sequence to be detected when capturing the anomaly pattern of the time series.
Single-dimension feature configuration	-	Configure each feature of the time series to be detected. This includes the following: Maximum value: The maximum value of the feature metric. Minimum value: The minimum value of the feature metric. Normalization: The method used to normalize the sequence when detecting this feature. Anomaly attention type: The type of anomaly pattern to focus on when detecting this feature.
Notification sensitivity configuration	-	Configure different notification thresholds for anomalies detected in different time periods. For example, you can ignore anomalies that occur during scheduled weekly maintenance.

Stream decomposition algorithm

Configure the algorithm.

Parameter	Subparameter	Description
Automatic periodic detection	-	Enables automatic detection of the time series period. This is suitable for time series data with seasonality. If the seasonality is consistent, we recommend that you disable this feature and manually set the period length.
Periodic detection frequency	-	Takes effect when Automatic periodic detection is enabled. The algorithm periodically updates the period of the time series at the specified frequency. For example, if the frequency is 12 hours, the algorithm detects and updates the period every 12 hours.
Period length	-	Takes effect when Automatic periodic detection is disabled. Set the duration of the sequence period. If the sequence has no period, set this to 0.
Observation length	-	The length of historical data used for anomaly detection. If the sequence contains seasonality, we recommend that the observation length be three times the sequence period length. For example, if the sequence period is 1 day, set the observation length to 3 days.
Sensitivity	-	The higher the sensitivity, the more anomalies are detected and the higher the anomaly scores. This increases the recall rate but may lower the precision.
Advanced parameters	Trend component sensitivity	The algorithm decomposes the sequence into trend, seasonal, and noise components. The higher the sensitivity for the trend component, the more anomalies are detected and the higher the anomaly scores during trend component anomaly detection. This increases the recall rate but may lower the precision.
	Noise component sensitivity	The algorithm decomposes the sequence into trend, seasonal, and noise components. The higher the sensitivity for the noise component, the more anomalies are detected and the higher the anomaly scores during noise component anomaly detection. This increases the recall rate but may lower the precision.
	Trend component sampling step	The algorithm decomposes the sequence into trend, seasonal, and noise components. If the sequence observation length is too long, the trend component analysis can be slow. A larger sampling step for the trend component speeds up the analysis but may reduce the detection accuracy. For example, if you set the trend component sampling step to 8, one data point is sampled for every eight points in the original sequence for trend analysis.
	Seasonal component sampling step	The algorithm decomposes the sequence into trend, seasonal, and noise components. If the sequence observation length is too long, the seasonal component analysis can be slow. A larger sampling step for the seasonal component speeds up the analysis but may reduce the detection accuracy. For example, if you set the seasonal component sampling step to 8, one data point is sampled for every eight points in the original sequence for seasonal analysis. We recommend that you set this to a value of 5 or less.
	Window length	If the sequence observation length is too long, anomaly detection can be slow. Set a window length to make the detection algorithm detect the data sequence in segments by using a sliding window, which improves detection speed. We recommend that you set this to a value of 5,000 or less. If you do not want to use a sliding window for detection, set this to 0.

In the preview section, test the algorithm and view the results based on the current parameter settings.
1. Set a time range to define the start and end times for the time series to be detected. Click Data Query. The system processes data within this range by using the query statement from the Data Feature Settings step to generate the time series data.
2. Select an Entity and a Feature to define the feature series to be detected. Click Preview. The detection algorithm processes the specified feature series and displays the results below. Click Display Parameters to view the current algorithm configuration.
3. The detection results include Trend Component Preview, Seasonal Component Preview, and Noise Component Preview. You can adjust the anomaly thresholds in the Trend Component Preview and Noise Component Preview sections. An alert is triggered only when an anomaly's score exceeds its corresponding threshold.

In the Scheduling Settings step, configure the following parameters.

Parameter	Description
Start at	The time from which the job starts reading and detecting data.
Data latency	The maximum expected delay for data to be written to Log Service. The job waits for this duration to ensure complete time series data is available.
Model start learning time	Optional. If set, the background task begins building the model at this time, which should be consistent with the job's scheduled start time.
Model end learning time	Optional. If not set, the model learns continuously and starts detection after the job's start time. If set, the background task stops updating the model at this time and immediately starts detection.

Alert configuration

In the Alert Configuration area of the Create Intelligent Inspection Job configuration wizard, complete the following configurations, and then click Complete.

Parameter

Description

Alert Policy

An Alert Policy groups, silences, and suppresses generated alerts.

If you select Simple Mode or Standard Mode, you do not need to configure an Alert Policy. Log Service uses the default dynamic alert policy (sls.builtin.dynamic) to manage alerts.
If you select Advanced Mode, you can select a built-in or custom Alert Policy to manage alerts. For more information about how to create an Alert Policy, see Create an Alert Policy.

Action Policy

An Action Policy controls how and when alert notifications are sent, including the notification channel and frequency.

If Alert Policy is set to Simple Mode, you only need to configure an action group.

After you configure an action group, Log Service automatically creates an Action Policy named Rule Name-Action Policy. All alerts triggered by this alert rule are sent according to this Action Policy. For more information, see Notification channels.

Important
You can modify this Action Policy on the Action policies page. For more information, see Action policies. If you add conditions when modifying the Action Policy, the Alert Policy setting automatically changes to Standard Mode.
If Alert Policy is set to Standard Mode or Advanced Mode, you can select a built-in or custom Action Policy to manage alert notifications. For more information about how to create an Action Policy, see Action policies.

If you set Alert Policy to Advanced Mode, you can also enable or disable Custom Action Policy. For more information, see Dynamic action policy mechanism.

Manage intelligent inspection jobs

View a job: Find the job in the job list and click its ID to view details.
Edit a job: To edit a job, find it in the list, and click Edit in the Actions column.
Delete a job: To delete a job, find it in the list, and click Delete in the Actions column.

Important
A deleted intelligent inspection job cannot be recovered. Proceed with caution.

Next steps

Alerts and feedback