This topic describes how to use LogReduce to group log data. LogReduce is a feature provided by Log Service to group similar log entries by detecting the same log patterns during text log collection.

The LogReduce feature allows you to group text logs of multiple formats. With LogReduce, you can locate problems, detect errors, and perform a version rollback or other O&M operations for DevOps. You can also detect network intrusions to ensure data security. In addition, you can save the log grouping result as an analysis chart to a dashboard, and then view the grouped data in real time.

Benefits

  • Log entries in the Log4J, JSON, or Syslog format can be grouped.
  • Gigabytes of data can be grouped in seconds.
  • Log entries can be grouped by log pattern.
  • You can perform reverse lookup on group log entries based on their signatures.
  • The number of log entries grouped in different time ranges can be compared by log pattern.
  • The precision of log grouping can be adjusted.

Billing method

After the LogReduce feature is enabled, the size of the log indexes increases by 10% of the size of raw log data. For example, if the size of raw log data is 100 GB per day, the size of log indexes increases by 10 GB.

Raw log size Index percentage Size of indexes generated by LogReduce Index size
100 GB 20% (20 GB) 100 × 10% 30 GB
100 GB 40% (40 GB) 100 × 10% 50 GB
100 GB 100% (100 GB) 100 × 10% 110 GB

Enable LogReduce of a Logstore

The LogReduce feature is disabled by default.

  1. Log on to the Log Service console, and then click the target project name.
  2. Click the Logstore management icon icon next to the name of the Logstore, and then select Search & Analysis.
  3. If you have enabled and configured the index of the Logstore, choose Index Attributes > Modify. If you have not enabled the index, click Enable.
    Figure 1. Enable the index
    Enable the index
    Figure 2. Modify the index
    Modify the index
  4. Set the index attributes and turn on the LogReduce switch.
    Figure 3. Enable LogReduce
    Enable LogReduce
  5. Click OK.
    After LogReduce is enabled, Log Service groups collected log data. Then, you can perform the following operations:

View log grouping results and raw logs

  1. On the Search & Analysis page, enter a query statement in the search box, and then click Search & Analytics.
    You can use keywords to filter grouped log entries.
    Note SQL statements are not supported. This means analysis results cannot be grouped.
  2. Click the LogReduce tab to view the log grouping result.
    The filtered log grouping result is displayed on the LogReduce tab.
    Item Description
    Number The sequence number of a log group.
    Count The number of log entries of a log group.
    Pattern The log pattern. Each log group has one or more sub-patterns.
  3. Move the pointer over a value in the Count column to view the sub-patterns of the corresponding log group and the percentage of each sub-pattern in the log group. You can also click the plus sign (+) before the count value to expand the sub-pattern list.
    Figure 4. View log grouping details
    Log grouping details
  4. Click a count value to view the raw log entries of the corresponding log group.
    Figure 5. View the raw log entries
    Raw log entries

Adjust the log grouping precision

  1. On the Search & Analysis page, click the LogReduce tab.
  2. In the upper-right corner of the tab, drag the Pattern Count slider to adjust the precision of log grouping.
    • If you drag the slider towards Many, you can obtain a more precise log grouping result with more detailed patterns.
    • If you drag the slider towards Little, you can obtain a less precise log grouping result with less detailed patterns.
Figure 6. Adjust the log grouping precision
Adjust the log grouping precision

Compare the number of log entries grouped in different time ranges

On the LogReduce tab, click Log Compare, set a time range, and then click OK. The comparison result is displayed on the tab.Compare the number of log entries
Item Description
Number The sequence number of a log group.
Pre_Count The number of log entries grouped by the current pattern within the previous time range.
Count The number of log entries grouped by the current pattern within the current time range.
Diff The difference between the Pre_Count value and Count value.
Pattern The pattern of a log.
Use the LogReduce feature through API operations
  • Obtain a log grouping result.
    • Run the following SQL statement:
      * | select a.pattern, a.count,a.signature, a.origin_signatures from (select log_reduce(3) as a from log) limit 1000 
      Note When you view the log grouping result in the Log Service console, you can click Copy Query to obtain the relevant SQL statement.
    • Input parameter: log_reduce (precision)

      precision: An integer from 1 to 16 can be set as the log grouping precision. A lower value indicates a higher precision and more patterns. Default value: 3.

    • Returned fields:
      • pattern: the sub-patterns of log entries in a log group.
      • count: the number of log entries in a log group.
      • signature: the log pattern of a log group.
      • origin_signatures: the original signature of a log group. You can use this field to query log entries of the log group.
  • Compare the number of log entries grouped in different time ranges.
    • Run the following SQL statement:
      * | select v.pattern, v.signature, v.count, v.count_compare, v.diff from (select compare_log_reduce(3, 86400) as v from log) order by v.diff desc limit 1000 
      Note If you click Log Compare in the Log Service console, you can click Copy Query to obtain the SQL statement.
    • Input parameters: compare_log_reduce(precision, compare_interval)
      • precision: An integer from 1 to 16 can be set as the log grouping precision. A lower value indicates a higher precision and more patterns. Default value: 3.
      • compare_interval: the number of seconds before the log entries to be compared with were generated. This parameter must be set to a positive integer.
    • Returned fields:
      • pattern: the sub-patterns of log entries in a log group.
      • signature: the log pattern of a log group.
      • count: the number of log entries in a log group.
      • count_compare: the number of log entries for a same-pattern log group within the specified time range.
      • Diff: the difference between the count field value and the count_compare field value.