Simple Log Service:LogReduce

Benefits

• Supports logs in any format, such as Log4j, JSON, and single-line logs.

• Processes hundreds of millions of log entries and returns results in seconds.

• Clusters log data by any pattern.

• Look up raw logs by using the signature of a pattern.

• Compares patterns across different time periods.

• Dynamically adjust clustering precision.

Index traffic

Enable LogReduce

1. Log on to the Simple Log Service console.

2. In the Projects section, click the one you want.

   

3. On the Log Storage > Logstores tab, click the logstore you want.

   

4. Enable the LogReduce feature.

   1. Click Index Attributes > Attributes.

      If you have not enabled the index feature, click Enable Index.

   2. In the Query and Analysis panel, turn on the LogReduce switch.

   3. Optional: Configure a whitelist or blacklist for fields to include or exclude from clustering.

      Note

      You cannot configure both a whitelist and a blacklist.

      LogReduce filter	Description
      whitelist	If you configure a whitelist, Simple Log Service clusters logs based only on the fields in the whitelist.
      blacklist	If you configure a blacklist, Simple Log Service excludes the blacklisted fields from the clustering process.
      No whitelist or blacklist is configured	If neither a whitelist nor a blacklist is configured, Simple Log Service clusters logs based on all available fields.

   4. Click OK.

Clustering results and raw logs

1. On the Search & Analysis page, enter a query statement, set a time range, and then click Search & Analyze.

   Note

   You can use only query statements to filter logs. LogReduce does not support analysis statements because it cannot process analysis results.

2. Click the LogReduce tab to view the clustering results.

   You can also click Add to New Dashboard to save the clustering results to a dashboard.

   The results page includes a Pattern Count slider (which you can drag to adjust the clustering granularity from Many to Little), a Copy Query button, and a Log Compare button.

   Parameter	Description
   Number	The serial number of the cluster.
   Count	The number of logs that match the pattern within the specified time range.
   Pattern	A pattern generated by LogReduce that represents a type of log entry. A pattern can contain multiple sub-patterns.

   • Hover over a number in the Count column to view the sub-patterns of the current cluster and the percentage of each sub-pattern. Click the plus sign (+) next to the number to expand the sub-pattern list.

   • Click a number in the Count column to go to the Raw Log tab and view the raw logs that match the pattern.

Adjust clustering precision

Compare log clusters

1. On the LogReduce tab, click Log Compare.

2. Set the comparison time range and click OK.

   For example, if your current query time range is 15 minutes and you select 1 Day in the Log Compare panel, the system compares the current 15-minute window with the same 15-minute window from the previous day.

   Parameter	Description
   Number	The serial number of the cluster.
   Pre_Count	The number of logs that match this pattern within the time range that you set in Log Compare.
   Count	The number of logs for the pattern in the current query time range.
   Diff	The difference and percentage change in the number of logs for the pattern between the two time periods.
   Pattern	The specific pattern of a log type.

SQL examples

You can also run Search & Analysis statements to get LogReduce results.

• Get LogReduce results

  • Search & Analysis statement

    * | select a.pattern, a.count,a.signature, a.origin_signatures from (select log_reduce(3) as a from log) limit 1000 

    Note

    When you view the clustering results, you can click Copy Query Statement to get the Search & Analysis statement for the current LogReduce results.

  • Modify parameters

    Modify the log_reduce(precision) function in the statement. The precision parameter specifies the clustering precision. Valid values range from 1 to 16. A smaller value indicates higher precision and generates more patterns. The default value is 3.

  • Returned fields

    You can find the detailed LogReduce results on the Aggregation tab.

    Parameter	Description
    pattern	The specific pattern of a log type.
    count	The number of logs matching the pattern in the specified time range.
    signature	The signature of the pattern.
    origin_signatures	The secondary signature of the pattern. You can use the secondary signature to look up raw logs.

• Compare LogReduce results across different time periods

  • Search & Analysis statement

    * | select v.pattern, v.signature, v.count, v.count_compare, v.diff from (select compare_log_reduce(3, 86400) as v from log) order by v.diff desc limit 1000 

    Note

    After you use Log Compare to compare clustering results from different time periods, you can click Copy Query Statement to get the corresponding Search & Analysis statement.

  • Modify parameters

    Modify the compare_log_reduce(precision, compare_interval) function in the query statement.

    • The precision parameter specifies the clustering precision. Valid values range from 1 to 16. A smaller value indicates higher precision and generates more patterns. The default value is 3.

    • The compare_interval parameter specifies how far back in time to compare, in seconds. The value must be a positive integer.

  • Returned fields

    Parameter	Description
    pattern	The specific pattern of a log type.
    count_compare	The number of logs matching the pattern in the comparison time period.
    count	The number of logs matching the pattern in the current time range.
    diff	The difference between 'count' and 'count_compare'.
    signature	The signature of the pattern.

Disable LogReduce

If you no longer need the LogReduce feature, you can disable it.

1. On the Search & Analysis page, click Index Attributes > Attributes.

2. Turn off the LogReduce switch.

3. Click OK.