The results of Intelligent Anomaly Analysis are stored in a Logstore named internal-ml-log. This topic describes the fields in the results.
Common tag fields
__tag__:__apply_time__:1638414250
__tag__:__batch_id__:a8343****5b0fd
__tag__:__data_type__:anomaly_detect
__tag__:__instance_name__:29030-****7bcdd
__tag__:__job_name__:etl-1637****3966-398245
__tag__:__model_name__:d52b5****c45397
__tag__:__region__:chengdu
__tag__:__schedule_id__:2457f****ebcdd| Field | Description |
|---|---|
| __tag__:__apply_time__ | The time that is required by a model to inspect a batch of data. Unit: seconds. |
| __tag__:__batch_id__ | The ID of a batch. Data in the same batch is identified by the same batch ID. |
| __tag__:__data_type__ | The type of data.
|
| __tag__:__instance_name__ | The name of the instance that is created for a task. The name consists of a project ID and a schedule ID. Each task is associated with an instance name on the backend server. |
| __tag__:__job_name__ | The name of a task. The name of each task in a project must be unique. |
| __tag__:__model_name__ | The name of a model. A model is created for each entity in a task. Each model is associated with a time series entity. |
| __tag__:__region__ | The region of a task. |
| __tag__:__schedule_id__ | The ID of the instance that is created for a task. Each task is associated with an instance ID on the backend server. |
Intelligent inspection (model training)
The type of a log varies based on the value of the __tag__:__data_type__ field.
Statistical runtime data of a task
If the value of the __tag__:__data_type__ field in the result data of your model training task is job_statistic, the data is the statistical runtime data of the task.
| Field | Description |
|---|---|
| meta | The project and Logstore to which the data source of the model training task belongs. The value is JSON-formatted data. |
| project_name | The project to which the data source of the model training task belongs. |
| logstore_name | The Logstore to which the data source of the model training task belongs. |
| result | The result content. The value is JSON-formatted data. |
| event_msg | The progress of the model training task at the specified timestamp. |
| occ_time | The timestamp that corresponds to the progress of the model training task. |
| tips | The overview of the progress for the model training task. For example, the model is stored. |
Detection result data of a model training task
If the value of the __tag__:__data_type__ field in the result data of your model training task is detection_process, the data is the detection result data of the task.
| Field | Description |
|---|---|
| meta | The project and Logstore to which the data source of the model training task belongs. The value is JSON-formatted data. |
| project_name | The project to which the data source of the model training task belongs. |
| logstore_name | The Logstore to which the data source of the model training task belongs. |
| result | The result content. The value is JSON-formatted data. |
| dim_name | A feature of an entity. |
| score | The anomaly score for the feature of an entity at a specific point in time. |
| value | The value size for the feature of an entity at a specific point in time. |
| is_train_step | Indicates whether the point belongs to the training set. |
Result data of a validation set
If the value of the __tag__:__data_type__ field in the result data of your model training task is eval_report, the data is the result data of each entity validation set after the task is complete.
| Field | Description |
|---|---|
| entity | The entity for which the model is created. The value is a key-value pair. |
| meta | The project and Logstore to which the data source of the model training task belongs. The value is JSON-formatted data. |
| project_name | The project to which the data source of the model training task belongs. |
| logstore_name | The Logstore to which the data source of the model training task belongs. |
| result | The result content. The value is JSON-formatted data. |
| evaluation_metrics.auc | The AUC of the validation set. The AUC is calculated by the supervision model that is trained for the entity. |
| evaluation_metrics.macro_f1 | The macro-averaged F1 score of the validation set. The macro-averaged F1 scoreis calculated by the supervision model that is trained for the entity. |
| evaluation_metrics.precision | The precision of the validation set. The precision is calculated by the supervision model that is trained for the entity. |
| evaluation_metrics.recall | The recall of the validation set. The recall is calculated by the supervision model that is trained for the entity. |
| time_config.training_start_time | The start time of model training for the entity. Unit: seconds. |
| time_config.training_stop_time | The end time of model training for the entity. Unit: seconds. |
| time_config.validation_end_time | The end time of model validation for the entity. Unit: seconds. |
| time_config.predict_time | The duration of model verification for the entity. Unit: seconds. |
| time_config.train_time | The duration of model training for the entity. Unit: seconds. |
| statistic.train_data_meta.train_anomaly_num | The number of anomaly points in the training set for the entity. |
| statistic.train_data_meta.train_data_length | The length of the training set for the entity. |
| statistic.evaluation_data_meta.evaluation_anomaly_num | The number of anomalies of the validation set for the entity. |
| statistic.evaluation_data_meta.evaluation_data_length | The length of the validation set for the entity. |
Intelligent inspection (real-time inspection)
The type of a log varies based on the value of the __tag__:__data_type__ field.
Statistical runtime data of a task
If the value of the __tag__:__data_type__ field in the result data of your real-time inspection task is job_statistic, the data is the statistical runtime data of the task.
{
"__tag__:__job_name__": "etl-1637133966-398245",
"__tag__:__region__": "chengdu",
"__tag__:__data_type__": "job_statistic",
"__tag__:__apply_time__": "1638415928",
"__tag__:__instance_name__": "29030-2457fbbd724de9421da8c73d37debcdd",
"result": {
"maxEntity": {
"host": "machine_001",
"ip": "192.0.2.1"
},
"maxTime": 1638415994,
"minEntity": {
"host": "machine_001",
"ip": "192.0.2.1"
},
"minTime": 1638415994,
"nTotalEntity": 1
}
}| Field | Description |
|---|---|
| result | The result item. The value is JSON-formatted data. |
| maxEntity | The information about the entity at the point in time that is the closest to the point in time of the current data consumption. |
| maxTime | The point in time of the entity that is the closest to the current data consumption. |
| nTotalEntity | The number of entities that are detected in the current task. |
Output data of entity inspection progress
If the value of the __tag__:__data_type__ field in the result data of your real-time inspection task is job_progress, the data is the output data of entity inspection progress. If a log contains the output data of entity inspection progress, you can determine whether errors occur. For example, you can determine whether a new entity appears or whether an existing entity does not have data.
{
"__tag__:__job_name__": "etl-1637133966-398245",
"__tag__:__region__": "chengdu",
"__tag__:__data_type__": "job_progress",
"__tag__:__apply_time__": "1638415883",
"__tag__:__instance_name__": "29030-2457fbbd724de9421da8c73d37debcdd",
"result": {
"new_entity": false,
"recently_arrived_time": 1638415994
},
"meta": {
"logstore_name": "machine_monitor",
"project_name": "sls-ml-demo"
},
"entity": {
"host": "machine_001",
"ip": "192.0.2.1"
}
}| Field | Description |
|---|---|
| meta | The project and Logstore of the current task. The value is JSON-formatted data. |
| project_name | The project to which the data source of the real-time inspection task belongs. |
| logstore_name | The Logstore to which the data source of the real-time inspection task belongs. |
| result | The result item. The value is JSON-formatted data. |
| new_entity | Indicates whether a new entity appears. |
| recently_arrived_time | The timestamp of the last valid data record in the current entity, which is specified by the entity field. |
| entity | The information about an entity. The information is of the dictionary data type. |
Result data of anomalies
If the value of the __tag__:__data_type__ field in the result data of your real-time inspection task is anomaly_detect, the data is the result data of anomalies.
{
"__time__": 1638416474,
"__tag__:__batch_id__": "a5870979816fc507cbeebc6b1133af0a",
"__tag__:__schedule_id__": "2457fbbd724de9421da8c73d37debcdd",
"__tag__:__apply_time__": "1638416291",
"__tag__:__job_name__": "etl-1637133966-398245",
"__tag__:__model_name__": "d52b59a6bfb3adcf2ee62a5064c45397",
"__tag__:__data_type__": "anomaly_detect",
"__tag__:__region__": "chengdu",
"__tag__:__instance_name__": "29030-2457fbbd724de9421da8c73d37debcdd",
"result": {
"anomaly_type": "None",
"dim_name": "value",
"is_anomaly": false,
"score": 0,
"value": "0.780000"
},
"meta": {
"logstore_name": "machine_monitor",
"project_name": "sls-ml-demo"
},
"entity": {
"host": "machine_001",
"ip": "192.0.2.1"
}
}| Field | Description |
|---|---|
| entity | The entity item. The value is JSON-formatted data and is obtained from the source data. The value is used to identify an entity. |
| meta | The configuration item. The value is JSON-formatted data and is obtained from the configuration information about an intelligent inspection task. |
| project_name | The project to which the Logstore belongs. |
| logstore_name | The Logstore to which the data source belongs. |
| result | The result item. The value indicates the inspection result of data at each point in time. |
| dim_name | The name of the dimension in which the generated inspection result is presented. The name is obtained from the source data. The value of the result field is presented only in a single dimension regardless of whether one or more dimensions are specified. |
| value | The value of the generated inspection result in the specified dimension. The value is obtained from the source data. The dimension is specified by the result.dim_name parameter. |
| score | The anomaly score. Valid values: [0,1]. A higher score indicates a higher degree of anomaly. |
| is_anomaly | Indicates whether an anomaly is considered true.
|
| anomaly_type | The anomaly type. A model preliminarily classifies an anomaly into the following types: Stab, Shrift, Variance, Lack, and OverThreshold. For more information, see Anomaly types. |
Text analysis
The results of a text analysis task include the common tag fields and the following common fields.
| Field | Description |
|---|---|
| algo_type | The algorithm type. |
| result_type | The result type, which is of the JSON data type. |
| result | The result content, which is of the JSON data type. The value of the result field varies based on the value of the result_type field. |
| meta | The metadata. The value is JSON-formatted data. |
| project_name | The project to which the Logstore belongs. |
| logstore_name | The Logstore to which the data source belongs. |
| topic | The log topic of the data source. |
| query | The method that is used to pull data. For example, a consumer group can be used to pull data. |
| win_size | The length of a time window. |
| version | The algorithm version. |
The value of the result field varies based on the value of the result_type field. The following sections describe the result fields.
cluster_info specified for the result_type field
If the value of the result_type field is cluster_info, the value of the result field includes information about a log category. The following example shows the structure of the result field in this scenario:
"result": {
"cluster_id": "xxxx",
"cluster_pattern": "xxxx",
"cluster_active_age": 120,
"cluster_alive_age": 150,
"anomaly_score": 0.1,
"count": 2,
"source": []
}| Field | Description |
|---|---|
| result.cluster_id | The ID of the log category. |
| result.cluster_pattern | The log template of the log category. |
| result.cluster_active_age | The number of time windows in which the log category is active. If the logs of a log category are detected in a time window, the log category is considered active in the time window. |
| result.cluster_alive_age | The number of time windows that are counted from the first time the log category appears to the current time. |
| result.anomaly_score | The anomaly score of the log category. |
| result.count | The number of logs that are included in the log category. |
| result.source | The possible values of variables in the log template. |
group_info specified for the result_type field
If the value of the result_type field is group_info, the value of the result field includes information about a log category group. The following example shows the structure of the result field in this scenario:
"result": {
"group_anomaly_score": 0.1,
"group_age": 10,
"group_n_event": 190,
"group_n_cluster": 10
}| Field | Description |
|---|---|
| result.group_anomaly_score | The anomaly score of the log category group. |
| result.group_age | The sequential number of the current time window. |
| result.group_n_event | The total number of logs in the group in the current time window. |
| result.group_n_cluster | The total number of log categories in the group in the current time window. |
anomaly_info specified for the result_type field
If the value of the result_type field is anomaly_info, the value of the result field includes information about an anomaly event. The following example shows the structure of the result field in this scenario:
"result": {
"anomaly_id": "xxxx",
"anomaly_type": "xxxx",
"value": 0,
"anomaly_score": 0.0,
"expect_lower": 0.0,
"expect_upper": 0.0
}| Field | Description |
|---|---|
| result.anomaly_id | The log category ID for the anomaly. |
| result.anomaly_type | The anomaly type. |
| result.value | The event value. The meaning of the result.value field varies based on the value of the result.anomaly_type field. |
| result.anomaly_score | The anomaly score. |
| result.expect_lower | The lower limit of the expected event value, which is specified by the result.value field. |
| result.expect_upper | The upper limit of the expected event value, which is specified by the result.value field. |
Time series forecasting
The results of a time series forecasting task include the common tag fields and the following common fields.
| Field | Description |
|---|---|
| algo_type | The algorithm type. The value is fixed as series_prediction. |
| result_type | The result type. The value is JSON-formatted data. If a forecasting operation is successful, the value is prediction_ok. If a forecasting operation fails, the value is prediction_error. |
| result | The result content. The value is JSON-formatted data. The value of the result field varies based on the value of the result_type field. |
| meta | The metadata. The value is JSON-formatted data. |
| project_name | The project to which the Logstore belongs. |
| logstore_name | The Logstore to which the data source belongs. |
| topic | The log topic of the data source. |
| version | The algorithm version. |
The value of the result field varies based on the value of the result_type field. The following sections describe the result fields.
prediction_ok specified for the result_type field
If the value of the result_type field is prediction_ok, the forecasting operation is successful, and each log includes the forecasting result of a point in the time series. The following example shows the structure of the result field in this scenario:
{
"entity": "xxxx",
"metric": "xxxx",
"time": xxxx,
"value": "xxxx",
"expect_value": "xxxx",
"expect_lower": "xxxx",
"expect_upper": "xxxx"
}| Field | Description |
|---|---|
| result.entity | The entity ID of the forecasted time series. |
| result.metric | The metric in the forecasted time series. |
| result.time | The timestamp of the current point in the forecasted time series. |
| result.value | The actual value of the current point in the forecasted time series. |
| result.expect_value | The forecast value of the current point in the forecasted time series. |
| result.expect_lower | The forecast lower limit of the current point in the forecasted time series. |
| result.expect_upper | The forecast upper limit of the current point in the forecasted time series. |
prediction_error specified for the result_type field
If the value of the result_type field is prediction_error and the value of the __tag__:__data_type__ field is job_error_message, an error occurs in the forecasting operation. The following example shows the structure of the result field in this scenario:
{
"entity": "xxxx",
"metric": "xxxx",
"error_type": "xxxx",
"error_msg": "xxxx"
}| Field | Description |
|---|---|
| result.entity | The entity ID of the forecasted time series. An error occurs in the forecasting operation. |
| result.metric | The metric of the forecasted time series. An error occurs in the forecasting operation. |
| result.error_type | The error type. |
| result.error_msg | The error details. |
Drill-down analysis
| Field | Description |
|---|---|
| result | The result content. The value is JSON-formatted data. The value of the result field varies based on the value of the __tag__:__data_type__ field. |
The type of a log varies based on the value of the __tag__:__data_type__ field.
Progress information about a drill-down analysis task
If the value of the __tag__:__data_type__ field is job_progress, the value of the result field includes progress information about a drill-down analysis task.
| Field | Description |
|---|---|
| result.from_ts | The start time of the task. |
| result.to_ts | The end time of the task. The value inf indicates that the task is ongoing. |
| result.progress | The current progress of the task. |
| result.message | The status information about the current progress of the task. |
Status information about a drill-down analysis task
If the value of the __tag__:__data_type__ field is job_status, the value of the result field includes status information about a drill-down analysis task.
| Field | Description |
|---|---|
| result.from_ts | The start time of the task. |
| result.to_ts | The end time of the task. The value inf indicates that the task is ongoing. |
| result.status | The status of the task. |
| result.message | The status details of the task. |
Root causes detected by a drill-down analysis task
If the value of the __tag__:__data_type__ field is root_cause, the value of the result field includes the root causes that are detected by a drill-down analysis task.
| Field | Description |
|---|---|
| result.status | Indicates whether root causes are detected. Valid values:
|
| result.snapshot_time | The point in time of the multi-dimensional time series data that is used for drill-down analysis. |
| result.elapsed_time | The duration of troubleshooting that is performed on the event to detect root causes. |
| result.event_info | The event that triggers root cause analysis. |
| result.root_cause | If the value of the result.status field is success, the value of this field indicates the result of the root cause analysis. |
| result.reason | If the value of the result.status field is fail, the value of this field indicates the reason why no causes are detected. |