This topic describes how to use the LogReduce feature of Log Service. You can enable the feature, view log clustering results and raw logs, adjust clustering precision, and compare the number of log entries in different time periods.
Background information
The LogReduce feature allows you to cluster similar logs and extract patterns from the logs. You can use this feature to cluster text logs of multiple formats. You can use this feature to locate errors, detect anomalies, roll back versions, and perform other O&M operations in DevOps scenarios. You can also use this feature to detect network intrusions to ensure data security. In addition, you can save log clustering results as charts to a dashboard, and then view the clustered data in real time.
Benefits
- You can cluster logs of multiple formats, such as Log4j logs, JSON-formatted logs, and single-line logs.
- Hundreds of millions of log entries can be clustered in seconds.
- Log data can be clustered in multiple modes.
- Raw log data can be retrieved based on pattern signatures.
- You can compare log patterns of different time periods.
- You can adjust the clustering precision of logs.
Indexes
Raw log size | Index percentage | Size of indexes that are generated by LogReduce | Index size |
---|---|---|---|
100 GB | 20% (20 GB) | 100 * 10% | 30 GB |
100 GB | 40% (40 GB) | 100 * 10% | 50 GB |
100 GB | 100% (100 GB) | 100 * 10% | 110 GB |
Enable LogReduce of a Logstore
- Log on to the Log Service console.
- In the Projects section, click the destination project.
- On the tab, click the destination Logstore.
- Enable the LogReduce feature.
View log clustering results and raw logs
Adjust the precision of log clustering
- If you drag the slider towards Many, you can obtain a more precise log clustering result with more detailed patterns.
- If you drag the slider towards Little, you can obtain a less precise log clustering result with fewer detailed patterns.
Compare the number of log entries that are clustered in different time periods
Examples
You can use search and analytic statements to query clustered log data.
- Obtain log clustering results.
- Search and analytic statement
* | select a.pattern, a.count,a.signature, a.origin_signatures from (select log_reduce(3) as a from log) limit 1000
Note When you view log clustering results, you can click Copy Query to obtain the search and analytic statement. - Modify parameter settings
In the search and analytic statement, the log_reduce(precision) function specifies the clustering precision. Valid values: 1 to 16. A smaller value indicates a higher clustering precision and more patterns. Default value: 3.
- Returned fields
You can view the clustering details on the Graph tab.
Parameter Description pattern The log pattern. count The number of log entries in a pattern. The log entries are obtained in the current time range. signature The signature of the log pattern. origin_signatures The secondary signature of the log pattern. You can use this signature to query corresponding raw data.
- Search and analytic statement
- Compare the number of log entries that are clustered in different time periods.
- Search and analytic statement
* | select v.pattern, v.signature, v.count, v.count_compare, v.diff from (select compare_log_reduce(3, 86400) as v from log) order by v.diff desc limit 1000
Note When you use Log Compare to compare log clustering results in different time periods, you can click Copy Query to obtain the search and analytic statement. - Modify parameter settings
Modify parameter settings in the compare_log_reduce(precision, compare_interval) function.
- The precision parameter specifies the clustering precision. Valid values: 1 to 16. A smaller number indicates a higher clustering precision and more patterns. Default value: 3.
- The compare_interval parameter specifies the time difference between the two time ranges when the number of log entries are compared. The value is a positive integer. Unit: seconds.
- Returned fields
Parameter Description pattern The log pattern. count_compare The number of log entries in a pattern that are obtained in the time range specified by Log Compare. count The number of log entries in a pattern. The log entries are obtained in the current time range. diff The difference between the number of log entries that are indicated by the count and count_compare fields. signature The signature of the log pattern.
- Search and analytic statement
Enable LogReduce of a Logstore
If you no longer need this feature, you can disable it.
- On the Search and Analysis page of the Logstore for which you want to disable the feature, choose .
- Turn off the LogReduce switch.
- Click OK.