Use Lindorm ML for time series anomaly detection - - Alibaba Cloud Documentation Center

This topic describes how to use Lindorm machine learning (ML) to perform time series anomaly detection.

Background information

Time series anomaly detection is a common feature for time series data analysis and is widely used in various scenarios, such as network security check and the daily O&M of large industrial equipment. Time series anomaly detection is also commonly used in AIOps scenarios. For example, you may need to monitor the resource usage and workload within specific time periods of multiple systems with different performance and requirements at the same time. In this case, if you use traditional monitoring and alerting features based on static thresholds, you must configure different policies for different businesses. This leads to problems such as low O&M efficiency, false alerts, missing alerts, and excessive alerts. Time series anomaly detection can automatically detect abnormal data points in continuous time series data based on AI algorithms, learn the characteristics of each time series based on machine learning algorithms, and then use the learned results to detect abnormal time series data. You can configure flexible policies to obtain accurate detection results in a time manner. This helps you improve the O&M efficiency.

Prerequisites

Lindorm ML is enabled for LindormTSDB. For more information about how to enable Lindorm ML, see Enable Lindorm ML.
The version of LindormTSDB is 3.4.21 or later. For more information about how to view or upgrade the version of LindormTSDB, see Release notes of LindormTSDB and Upgrade the minor engine version of a Lindorm instance.
You are connected to LindormTSDB by using lindorm-cli. A table is created in your database, and data is imported to the table. For more information, see Use Lindorm-cli to connect to and use LindormTSDB.

Data preparation

In this topic, a table named service_monitor is used as an example. The following example shows the schema of the service_monitor table:

+--------------+-----------+------------+------------+
| columnName   | typeName  | columnKind | primaryKey |
+--------------+-----------+------------+------------+
| time         | TIMESTAMP | TIMESTAMP  | false      |
| service_name | VARCHAR   | TAG        | false      |
| op_name      | VARCHAR   | TAG        | false      |
| host_ip      | VARCHAR   | TAG        | false      |
| qps          | DOUBLE    | FIELD      | false      |
| rt           | DOUBLE    | FIELD      | false      |
+--------------+-----------+------------+------------+

In this topic, the service_monitor table contains the following data:

+---------------------------+---------------+-----------+----------+-----+----+
|           time            |  service_name |  op_name  | host_ip  | qps | rt |
+---------------------------+----------------------------------+--------------+
| 2021-01-01T00:00:00+08:00 |   service_1   |    put    | 10.0.0.1 | 500 | 10 |
| 2021-01-01T00:00:05+08:00 |   service_1   |    put    | 10.0.0.1 | 600 | 8  |
| 2021-01-01T00:00:10+08:00 |   service_1   |    put    | 10.0.0.1 | 400 | 12 |
| 2021-01-01T00:00:15+08:00 |   service_1   |    put    | 10.0.0.1 | 700 | 7  |
| 2021-01-01T00:00:20+08:00 |   service_1   |    put    | 10.0.0.1 | 900 | 5  |
+---------------------------+---------------+-----------+----------+-----+----+

Procedure

The following example describes how to use Lindorm ML to detect time series anomalies in a business monitoring system.

Use the CREATE MODEL statement to train the model. The following statements provide an example on how to train the model:
```
CREATE MODEL esd_model
FROM (SELECT * FROM service_monitor)
TARGET qps
PROBLEM time_series_anomaly_detection
ALGORITHM esd
SETTINGS
(
);
```
Note If you set PROBLEM to TIME_SERIES_ANOMALY_DETECTION, you can use only the anomaly_detect function in model inference.

Manage the model. You can use the SHOW MODEL model_name statement to view the information about the model. The following statement provides an example on how to view the information about the model:

SHOW MODEL esd_model;

The following information is returned:

+-----------+--------+----------------+-------------------------------+-----------+---------------------------------+---------------+-----------------------+---------+-------------------------------+-------------------------------+
|   name    | status |  sql_function  |           task_type           | algorithm |             query               | preprocessors |       settings        | metrics |         created_time          |          update_time          |
+-----------+--------+----------------+-------------------------------+-----------+---------------------------------+---------------+-----------------------+---------+-------------------------------+-------------------------------+
| esd_model | Ready  | anomaly_detect | TIME_SERIES_ANOMALY_DETECTION | ESD       | SELECT * FROM `service_monitor` | []            | {train_mode=INENGINE} | {}      | 2022-11-02T18:48:28.717+08:00 | 2022-11-02T18:48:35.085+08:00 |
+-----------+--------+----------------+-------------------------------+-----------+---------------------------------+---------------+-----------------------+---------+-------------------------------+-------------------------------+

Perform inference based on the trained model. Use the anomaly_detect function to detect time series anomalies in real time. The following statement provides an example on how to perform time series anomaly detection:

SELECT `time`, service_name, op_name, host_ip, anomaly_detect(qps, 'esd_model') AS qps_detect_result FROM service_monitor WHERE `time` >= '2022-01-01T01:00:00+08:00' sample BY 0;

The following information is returned:

+---------------------------+--------------+---------+-----------+--------------------+
|           time            | servict_name | op_name |  host_ip  | qps_detect_result  |
+---------------------------+--------------+---------+-----------+--------------------+
| 2022-01-01T01:00:00+08:00 |  service_1   |   put   | 10.0.0.1  |    false           |
| 2022-01-01T01:00:05+08:00 |  service_1   |   put   | 10.0.0.1  |    true            |
| 2022-01-01T01:00:10+08:00 |  service_1   |   put   | 10.0.0.1  |    false           |
| 2022-01-01T01:00:15+08:00 |  service_1   |   put   | 10.0.0.1  |    false           |
| 2022-01-01T01:00:20+08:00 |  service_1   |   put   | 10.0.0.1  |    false           |
| 2022-01-01T01:00:25+08:00 |  service_1   |   put   | 10.0.0.1  |    false           |
+---------------------------+--------------+---------+-----------+--------------------+

Note You can use continuous queries together with time series anomaly detection to continuously detect time series anomalies. For more information, see Continuous queries and Detect time series data exceptions continuously.