All Products
Search
Document Center

Simple Log Service:Filter logs

最終更新日:May 30, 2024

If you use Logtail to collect logs, you can use the processor_filter_regex or processor_filter_key_regex plug-in to filter logs. This topic describes the parameters of the processor_filter_regex and processor_filter_key_regex plug-ins. This topic also provides examples on how to configure the plug-ins.

Limits

The input plug-ins for text logs and container stdout and stderr support only form configuration. Other input plug-ins support only editor configuration in JSON.

Entry point

If you want to use a Logtail plug-in to process logs, you can add a Logtail plug-in configuration when you create or modify a Logtail configuration. For more information, see Logtail plug-ins overview.

processor_filter_regex

You can use regular expressions to match the values of log fields and filter logs.

Form configuration

  • Parameters

    Set Processor Type to Filter Logs with Regular Expression (Match Log Field Values). Then, configure other parameters based on the following table.

    Note

    A log is collected only if the values of fields in the log exactly match the regular expressions that are specified in the Collect Logs parameter and do not match the regular expressions that are specified in the Drop Log parameter. In other cases, the log is dropped.

    Parameter

    Description

    Collect Logs

    Specifies the name of a log field and the regular expression based on which the value of the field is matched. If the values of log fields match the specified regular expressions, the log is collected.

    You can add multiple key-value pairs. The key-value pairs are evaluated by using a logical AND.

    Drop Log

    Specifies the name of a log field and the regular expression based on which the value of the field is matched. If the value of a log field matches a specified regular expression, the log is dropped.

    You can add multiple key-value pairs. The key-value pairs are evaluated by using a logical OR.

  • Configuration example

    In this example, only logs whose ip starts with 10, method is POST, and browser is not aliyun.* are collected.

    • Raw logs

      • Log 1

        "ip" : "10.**.**.**"
        "method" : "POST"
        "browser" : "aliyun-sdk-java"
      • Log 2

        "ip" : "10.**.**.**"
        "method" : "POST"
        "browser" : "chrome"
      • Log 3

        "ip" : "192.168.*.*"
        "method" : "POST"
        "browser" : "ali-sls-ilogtail"
    • Logtail plug-in configuration for data processing Filter logs

    • Results

      Log

      Collected

      Reason

      Log 1

      No

      The value of the browser field matches the regular expression that is specified in the Drop Log parameter.

      Log 2

      Yes

      The values of the specified fields match the regular expressions that are specified in the Collect Logs parameter and do not match the regular expression that is specified in the Drop Log parameter.

      Log 3

      No

      The value of the ip field does not match a regular expression that is specified in the Collect Logs parameter.

Editor configuration in JSON

  • Parameters

    Set type to processor_filter_regex. Then, configure other parameters in detail based on the following table.

    Note

    A log is collected only if the values of fields in the log exactly match the regular expressions that are specified in the Include parameter and do not match the regular expressions that are specified in the Exclude parameter. In other cases, the log is dropped.

    Parameter

    Type

    Required

    Description

    Include

    JSON Object

    No

    The key-value pairs that are used as match conditions. In a key-value pair, the key specifies a field, and the value specifies a regular expression based on which the value of the field is matched. The key-value pairs are evaluated by using a logical AND. If the values of log fields match the regular expressions that are specified in the Include parameter, the log is collected.

    Exclude

    JSON Object

    No

    The key-value pairs that are used as match conditions. In a key-value pair, the key specifies a field, and the value specifies a regular expression based on which the value of the field is matched. The key-value pairs are evaluated by using a logical OR. If the value of a log field matches a regular expression that is specified in the Exclude parameter, the log is dropped.

  • Configuration example

    In this example, only logs whose ip starts with 10, method is POST, and browser is not aliyun.* are collected.

    • Raw logs

      • Log 1

        "ip" : "10.**.**.**"
        "method" : "POST"
        "browser" : "aliyun-sdk-java"
      • Log 2

        "ip" : "10.**.**.**"
        "method" : "POST"
        "browser" : "chrome"
      • Log 3

        "ip" : "192.168.*.*"
        "method" : "POST"
        "browser" : "ali-sls-ilogtail"
    • Logtail plug-in configuration for data processing

      {
         "type" : "processor_filter_regex",
          "detail" : {
               "Include" : {
                  "ip" : "10\\..*",
                  "method" : "POST"
               },
               "Exclude" : {
                  "browser" : "aliyun.*"
               }
          }
      }
    • Results

      Log

      Collected

      Reason

      Log 1

      No

      The value of the browser field matches the regular expression that is specified in the Exclude parameter.

      Log 2

      Yes

      The values of the specified fields match the regular expressions that are specified in the Include parameter and do not match the regular expression that is specified in the Exclude parameter.

      Log 3

      No

      The value of the ip field does not match a regular expression that is specified in the Include parameter.

processor_filter_key_regex

You can use regular expressions to match the names of log fields and filter logs.

Form configuration

  • Parameters

    Set Processor Type to Filter Logs with Regular Expression (Match Log Field Names). Then, configure other parameters based on the following table.

    Note

    A log is collected only if the names of fields in the log exactly match the regular expressions that are specified in the Collect Logs parameter and do not match the regular expressions that are specified in the Drop Log parameter. In other cases, the log is dropped.

    Parameter

    Description

    Collect Logs

    Specifies the regular expression based on which the name of a log field is matched. Multiple regular expressions are evaluated by using a logical AND. If the names of log fields match the specified regular expressions, the log is collected.

    Drop Log

    Specifies the regular expression based on which the name of a log field is matched. Multiple regular expressions are evaluated by using a logical OR. If the name of a log field matches a specified regular expression, the log is dropped.

  • Configuration example

    In this example, logs whose field names start with request are collected.

    • Raw logs

      • Log 1

        "request_time" : 20
        "request_length": 2314
        "request_method" : "POST"
        "browser" : "aliyun-sdk-java"
      • Log 2

        "request_time" : 70
        "request_method" : "GET"
        "ip" : "192.168.**.**"
      • Log 3

        "browser" : "ali-sls-ilogtail"
        "status" : 200
    • Logtail plug-in configuration for data processing Filter logs

    • Results

      Log

      Collected

      Reason

      Log 1

      Yes

      The names of the request_time, request_length, and request_method fields match the regular expression that is specified in the Collect Logs parameter.

      Log 2

      Yes

      The names of the request_time and request_length fields match the regular expression that is specified in the Collect Logs parameter.

      Log 3

      No

      No field names in the log match the regular expression that is specified in the Collect Logs parameter.

Editor configuration in JSON

  • Parameters

    Set type to processor_filter_key_regex. Then, configure other parameters in detail based on the following table.

    Note

    A log is collected only if the names of fields in the log exactly match the regular expressions that are specified in the Include parameter and do not match the regular expressions that are specified in the Exclude parameter. In other cases, the log is dropped.

    Parameter

    Type

    Required

    Description

    Include

    string[]

    No

    Specifies the regular expression based on which the name of a log field is matched. Multiple regular expressions are evaluated by using a logical AND. If the names of log fields match the specified regular expressions, the log is collected.

    Exclude

    string[]

    No

    Specifies the regular expression based on which the name of a log field is matched. Multiple regular expressions are evaluated by using a logical OR. If the name of a log field matches a specified regular expression, the log is dropped.

  • Configuration example

    In this example, logs whose field names start with request are collected.

    • Raw logs

      • Log 1

        "request_time" : 20
        "request_length": 2314
        "request_method" : "POST"
        "browser" : "aliyun-sdk-java"
      • Log 2

        "request_time" : 70
        "request_method" : "GET"
        "ip" : "192.168.**.**"
      • Log 3

        "browser" : "ali-sls-ilogtail"
        "status" : 200
    • Logtail plug-in configuration for data processing

      {
         "type" : "processor_filter_key_regex",
          "detail" : {
               "Include" : [
                  "request.*"
               ]
          }
      }
    • Results

      Log

      Collected

      Reason

      Log 1

      Yes

      The names of the request_time, request_length, and request_method fields match the regular expression that is specified in the Include parameter.

      Log 2

      Yes

      The names of the request_time and request_length fields match the regular expression that is specified in the Include parameter.

      Log 3

      No

      No field names in the log match the regular expression that is specified in the Include parameter.