The intelligent anomaly detection feature of Simple Log Service provides model training and real-time health check capabilities. It supports automated, intelligent, and adaptive model training and anomaly detection for data such as logs and metrics.
Starting July 15, 2025 (UTC+8), the intelligent anomaly analysis feature will no longer be available to new users. Existing users can continue to use it.
-
Scope of impact
The following core features will be unpublished: intelligent health check, text analytics, and time series forecasting.
-
Feature Migration Solutions
The machine learning syntax, scheduled query and analysis (scheduled SQL), and dashboard features of SLS can fully replace the unpublished features.
Background information
Time-based data, such as logs and metrics, accumulates over time. For example, if a service generates 10 million data entries per day, a total of 3.6 billion data entries are accumulated per year. If you use fixed rules to check this data, you may encounter the following issues:
-
Low efficiency: You must manually configure various rules to identify and catch anomalies.
-
Poor timeliness: Most time series data is time-sensitive. Faults and changes affect metric patterns. An anomaly identified by a specific rule may be considered normal at a later time.
-
Difficult configuration: Time series data comes in various forms, such as spikes, turning points, and periodic changes. Threshold ranges also vary. For complex patterns, configuring rules is often difficult.
-
Poor performance: Data streams change dynamically and business models evolve rapidly, which makes fixed rules ineffective in new business scenarios. This can lead to many false positives or false negatives. The tolerance for anomalies varies across scenarios and users. When you troubleshoot issues, catching more valid anomaly points helps the investigation. When you handle alerts, fewer but more important anomaly points improve alert processing efficiency.
To resolve these issues, Simple Log Service provides the intelligent health check feature. This feature uses proprietary AI algorithms to aggregate, check, and create alerts for streaming data such as logs and metrics. You only need to specify the monitored items. The algorithm model automatically detects anomalies, adapts to business changes, and refines alerts without manual rule configuration.
How it works
Simple Log Service uses SQL statements to construct and aggregate monitoring metrics. It pulls data into a model at regular intervals based on scheduling rules, writes health check results to a destination Logstore (internal-ml-log) based on event standards, and sends alert notifications for anomalies.
Features
Intelligent health check provides the following features.
|
Feature |
Description |
|
Configure monitored objects |
Use SQL or query and analysis statements to convert log data into monitoring metrics and initiate tasks. |
|
Analyze data at regular intervals |
Configure data features, entity fields, and metric fields as needed. The health check instance automatically discovers new monitored entities, pulls data at regular intervals, and performs automatic modeling and intelligent analysis. The model supports pull intervals as short as one second. |
|
Parameter settings and model performance preview |
Preview the effects of different model parameters, and visualize the metric time series curve and anomaly score curve. This helps you configure the model parameters that best suit your data features. |
|
Multi-channel result output |
Health check results are stored in the destination Logstore. Anomaly information is sent to you through alert notifications. |
Terms
The following table describes the key terms used in intelligent health check.
|
Term |
Description |
|
Task |
A health check task includes information such as data features, model parameters, and alert policies. |
|
Instance |
A health check task generates an execution instance based on the task configuration. Each instance pulls data, runs the algorithm model, and distributes health check results at regular intervals based on the task configuration.
|
|
Instance ID |
The unique identifier of an execution instance. |
|
Creation time |
The time when the instance is created. An instance is typically created based on the task rules you configure. An instance is created immediately for backfill runs or to catch up on delays. |
|
Runtime |
The time when the instance starts to run. If a task is retried, this is the start time of the last run. |
|
End time |
The time when the instance finishes running. If a task is retried, this is the end time of the last run. |
|
Execution status |
The execution status of the instance. Valid values:
|
|
Data features |
Data features include the following configurations:
|
|
Algorithm configuration |
Different algorithms have different configuration items. For more information about the configuration items of each algorithm, see Use SQL to aggregate metric data for real-time detection. |
|
Inspection event |
A health check event includes the following information:
|
Scheduling and execution scenarios
The following table describes common scheduling and execution scenarios for health check tasks.
|
Scenario |
Description |
|
Start a health check task from a historical point in time |
After you create a health check task, it processes historical data based on the task rules. The algorithm model quickly consumes historical data, performs model training, and gradually catches up to the current time. After the task creation time or the model learning end time is reached, health check events are generated. |
|
Modify the scheduling configuration |
After you modify the scheduling configuration, the next instance is generated based on the new configuration. The algorithm model remembers the current consumption time and continues to check new data. |
|
Retry a failed instance |
If an instance fails due to issues such as insufficient permissions, a non-existent source or destination database, or an invalid configuration, the system retries automatically. If the status remains Starting, the configuration may be incorrect. The error log is written to the internal-etl-log Logstore. Check the configuration and restart the task. After successful scheduling and execution, the system updates the instance status to Succeeded or Failed based on the result. |
Usage recommendations
To get the most out of intelligent health check, follow these recommendations:
-
Before configuring a health check task, understand the format of data in the Logstore, clarify the meaning of fields, and determine the observation interval.
-
To configure algorithm parameters effectively, understand the time series data changes of your monitored objects, including stability and periodicity, and have a preliminary expectation of anomaly patterns.
-
Align the health check task time window to whole time units, such as whole seconds, minutes, or hours. This ensures the timeliness of anomaly event alerts and the accuracy of multi-event correlation.
Model training
You can use model training to enhance anomaly learning and improve the accuracy of future anomaly alerts. Model training provides the following advantages.
-
If anomaly detection accuracy does not meet your expectations with the real-time health check feature alone, use model training tasks to improve it.
-
If the anomalies detected by a real-time health check task differ from what you consider anomalous, run a model training task first to adaptively detect the required anomaly types.
Basic process
-
Data input: Provide the data required for the model training service. This includes labeled and unlabeled metric data. This data is stored in Simple Log Service and must be retrieved using SQL queries. Labeled metric data can be sent directly to the algorithm service. Unlabeled metric data must be labeled using anomaly injection simulation before it can be sent to the algorithm service.
-
Algorithm service: This service consists of two main parts: feature engineering and a supervised model. In the algorithm service, a model is trained for each entity. The entity ID is used to identify the corresponding model.
-
Result storage and visualization: After a model training task is complete, the system stores the trained model on the cloud. The system also saves the validation results of the dataset and events from the task run as logs in a Logstore named internal-ml-log. You can also view the visualization results in the task details.
-
Create a prediction task: After a model training task is complete, you obtain the model trained for each entity in the task. You can then create a prediction task to perform real-time anomaly detection on future metric data. Use the Simple Log Service labeling tool to label the results, obtain more labeled data, and repeatedly train the model to improve its accuracy.
Algorithm service overview
The algorithm service consists of the following three main parts.
-
Dataset: A dataset is constructed based on a specified time range and is divided into a training set and a validation set.
The training set must cover a period of more than 12 days because the model training task requires one week of historical data as a prerequisite for feature engineering. The validation set must cover a period of more than 3 days because three days of data are required to generate a validation report that better explains the model's fit, robustness, and performance.
-
Feature engineering: This includes periodic comparison features, shift features, trend features, window features, and time features.
-
Ensemble model: An ensemble model is built by integrating multiple tree-based models.