This topic describes how to use LogReduce to group log data. LogReduce is a feature provided by Log Service to group similar log entries by detecting the same log patterns during text log collection.
The LogReduce feature allows you to group text logs of multiple formats. With LogReduce, you can locate problems, detect errors, and perform a version rollback or other O&M operations for DevOps. You can also detect network intrusions to ensure data security. In addition, you can save the log grouping result as an analysis chart to a dashboard, and then view the grouped data in real time.
Benefits
- Log entries in the Log4J, JSON, or Syslog format can be grouped.
- Gigabytes of data can be grouped in seconds.
- Log entries can be grouped by log pattern.
- You can perform reverse lookup on group log entries based on their signatures.
- The number of log entries grouped in different time ranges can be compared by log pattern.
- The precision of log grouping can be adjusted.
Billing method
After the LogReduce feature is enabled, the size of the log indexes increases by 10% of the size of raw log data. For example, if the size of raw log data is 100 GB per day, the size of log indexes increases by 10 GB.
| Raw log size | Index percentage | Size of indexes generated by LogReduce | Index size |
|---|---|---|---|
| 100 GB | 20% (20 GB) | 100 × 10% | 30 GB |
| 100 GB | 40% (40 GB) | 100 × 10% | 50 GB |
| 100 GB | 100% (100 GB) | 100 × 10% | 110 GB |
Enable LogReduce of a Logstore
The LogReduce feature is disabled by default.
- Log on to the Log Service console, and then click the target project name.
- Click the
icon next to the name of the Logstore, and then select Search & Analysis.
- If you have enabled and configured the index of the Logstore, choose . If you have not enabled the index, click Enable.
Figure 1. Enable the index 
Figure 2. Modify the index 
- Set the index attributes and turn on the LogReduce switch.
Figure 3. Enable LogReduce 
- Click OK.
After LogReduce is enabled, Log Service groups collected log data. Then, you can perform the following operations:
icon next to the name of the Logstore, and then select 
View log grouping results and raw logs
- On the Search & Analysis page, enter a query statement in the search box, and then
click Search & Analytics.
You can use keywords to filter grouped log entries.Note SQL statements are not supported. This means analysis results cannot be grouped.
- Click the LogReduce tab to view the log grouping result.
The filtered log grouping result is displayed on the LogReduce tab.
Item Description Number The sequence number of a log group. Count The number of log entries of a log group. Pattern The log pattern. Each log group has one or more sub-patterns. - Move the pointer over a value in the Count column to view the sub-patterns of the corresponding log group and the percentage
of each sub-pattern in the log group. You can also click the plus sign (+) before
the count value to expand the sub-pattern list.
Figure 4. View log grouping details 
- Click a count value to view the raw log entries of the corresponding log group.
Figure 5. View the raw log entries 
Adjust the log grouping precision
- On the Search & Analysis page, click the LogReduce tab.
- In the upper-right corner of the tab, drag the Pattern Count slider to adjust the precision of log grouping.
- If you drag the slider towards Many, you can obtain a more precise log grouping result with more detailed patterns.
- If you drag the slider towards Little, you can obtain a less precise log grouping result with less detailed patterns.
Compare the number of log entries grouped in different time ranges
| Item | Description |
|---|---|
| Number | The sequence number of a log group. |
| Pre_Count | The number of log entries grouped by the current pattern within the previous time range. |
| Count | The number of log entries grouped by the current pattern within the current time range. |
| Diff | The difference between the Pre_Count value and Count value. |
| Pattern | The pattern of a log. |
- Obtain a log grouping result.
- Run the following SQL statement:
* | select a.pattern, a.count,a.signature, a.origin_signatures from (select log_reduce(3) as a from log) limit 1000Note When you view the log grouping result in the Log Service console, you can click Copy Query to obtain the relevant SQL statement. - Input parameter: log_reduce (precision)
precision: An integer from 1 to 16 can be set as the log grouping precision. A lower value indicates a higher precision and more patterns. Default value: 3.
- Returned fields:
- pattern: the sub-patterns of log entries in a log group.
- count: the number of log entries in a log group.
- signature: the log pattern of a log group.
- origin_signatures: the original signature of a log group. You can use this field to query log entries of the log group.
- Run the following SQL statement:
- Compare the number of log entries grouped in different time ranges.
- Run the following SQL statement:
* | select v.pattern, v.signature, v.count, v.count_compare, v.diff from (select compare_log_reduce(3, 86400) as v from log) order by v.diff desc limit 1000Note If you click Log Compare in the Log Service console, you can click Copy Query to obtain the SQL statement. - Input parameters: compare_log_reduce(precision, compare_interval)
- precision: An integer from 1 to 16 can be set as the log grouping precision. A lower value indicates a higher precision and more patterns. Default value: 3.
- compare_interval: the number of seconds before the log entries to be compared with were generated. This parameter must be set to a positive integer.
- Returned fields:
- pattern: the sub-patterns of log entries in a log group.
- signature: the log pattern of a log group.
- count: the number of log entries in a log group.
- count_compare: the number of log entries for a same-pattern log group within the specified time range.
- Diff: the difference between the count field value and the count_compare field value.
- Run the following SQL statement: