This topic describes how to use the LogReduce feature of Log Service. You can enable the feature, view log clustering results and raw logs, and compare the number of clustered logs in different time periods.
Prerequisites
- A Standard Logstore is created. For more information, see Create a Logstore.
- Logs are collected. For more information, see Data collection overview.
- Indexes are configured. For more information, see Configure indexes.
Background information
When you collect logs, the LogReduce feature can cluster highly similar logs and extract patterns from the logs. This way, you can quickly have an overall understanding of the logs. The feature can cluster text logs in multiple formats. You can use the feature to perform O&M operations in DevOps scenarios. For example, you can use the feature to identify errors, detect anomalies, and roll back versions. You can also use the feature to detect intrusions in security scenarios. You can save log clustering results as charts to a dashboard and view the clustered data in real time.
Benefits
- The feature can cluster logs in multiple formats. Examples: Log4j, JSON, and single-line logs.
- The feature can cluster hundreds of millions of logs in seconds.
- The feature can cluster logs by multiple patterns.
- You can retrieve raw logs that are clustered by pattern based on pattern signatures.
- You can compare patterns that are extracted in different time periods.
- You can adjust the precision of log clustering based on your business requirements.
Index traffic
Size of raw logs | Index percentage | Size of indexes that are generated by LogReduce | Total size of indexes |
---|---|---|---|
100 GB | 20% (20 GB) | 100 * 10% | 30 GB |
100 GB | 40% (40 GB) | 100 * 10% | 50 GB |
100 GB | 100% (100 GB) | 100 * 10% | 110 GB |
Enable the LogReduce feature
- Log on to the Log Service console.
- In the Projects section, click the project that you want to view.
- Choose . On the Logstores tab, click the Logstore that you want to view.
- Enable the LogReduce feature.
View log clustering results and raw logs
Change the precision of log clustering
- If you adjust the slider toward Many, you can obtain a more precise log clustering result that has more detailed patterns.
- If you adjust the slider toward Little, you can obtain a less precise log clustering result that has less detailed patterns.
Compare the number of logs that are clustered in different time periods
Examples of query statements
You can use query statements to obtain log clustering results.
- Obtain log clustering results
- Query statement
* | select a.pattern, a.count,a.signature, a.origin_signatures from (select log_reduce(3) as a from log) limit 1000
Note When you view log clustering results, you can click Copy Query to obtain the query statement of the log clustering results. - Modify parameters
Modify the parameter settings in log_reduce(precision) of the query statement. The precision parameter specifies the precision of log clustering. A smaller value indicates a higher precision and more patterns. Valid values: 1 to 16. Default value: 3.
- Returned fields
You can view log clustering details on the Graph tab.
Parameter Description pattern The log pattern. count The number of logs for the pattern in the time range that is specified for the query. signature The signature of the log pattern. origin_signatures The secondary signature of the log pattern. You can use the secondary signature to retrieve the raw logs.
- Query statement
- Compare the number of logs that are clustered in different time periods.
- Query statement
* | select v.pattern, v.signature, v.count, v.count_compare, v.diff from (select compare_log_reduce(3, 86400) as v from log) order by v.diff desc limit 1000
Note When you use Log Compare to compare log clustering results in different time periods, you can click Copy Query to obtain the query statement of the log clustering results. - Modify parameters
Modify the parameter settings in compare_log_reduce(precision, compare_interval) of the query statement.
- The precision parameter specifies the precision of log clustering. A smaller value indicates a higher precision and more patterns. Valid values: 1 to 16. Default value: 3.
- The compare_interval parameter specifies the time difference between the two time ranges for comparison. The value is a positive integer. Unit: seconds.
- Returned fields
Parameter Description pattern The log pattern. count_compare The number of logs for the pattern in the previous time range that is specified for comparison. count The number of logs for the pattern in the time range that is specified for the query. diff The difference between the numbers of logs in the count and count_compare columns. signature The signature of the log pattern.
- Query statement
Disable the LogReduce feature
If you no longer need to use the LogReduce feature, you can disable the feature.
- On the query and analysis page of the Logstore for which you want to disable this feature, choose .
- Turn off LogReduce.
- Click OK.