If you use Logtail to collect logs, you can use the processor_filter_regex or processor_filter_key_regex plug-in to filter logs. This topic describes the parameters of the processor_filter_regex and processor_filter_key_regex plug-ins. This topic also provides examples on how to configure the plug-ins.

Important You can use form configuration to add Logtail plug-ins only if you select Kubernetes - Standard Output in the Import Data section.

Entry point

If you want to use a Logtail plug-in to process logs, you can add a Logtail plug-in configuration when you create or modify a Logtail configuration. For more information, see Entry point.

processor_filter_regex

You can use regular expressions to match the values of log fields and filter logs.

Form configuration

  • Parameters
    Set Processor Type to Filter Logs with Regular Expression (Match Log Field Values). Then, configure other parameters based on the following table.
    Note A log is collected only if the values of fields in the log exactly match the regular expressions that are specified in the Collect Logs parameter and do not match the regular expressions that are specified in the Drop Log parameter. In other cases, the log is dropped.
    ParameterDescription
    Collect LogsSpecifies the name of a log field and the regular expression based on which the value of the field is matched. If the values of log fields match the specified regular expressions, the log is collected.

    You can add multiple key-value pairs. The key-value pairs are evaluated by using a logical AND.

    Drop LogSpecifies the name of a log field and the regular expression based on which the value of the field is matched. If the value of a log field matches a specified regular expression, the log is dropped.

    You can add multiple key-value pairs. The key-value pairs are evaluated by using a logical OR.

  • Configuration example

    In this example, only logs whose ip starts with 10, method is POST, and browser is not aliyun.* are collected.

    • Raw logs
      • Log 1
        "ip" : "10.**.**.**"
        "method" : "POST"
        "browser" : "aliyun-sdk-java"
      • Log 2
        "ip" : "10.**.**.**"
        "method" : "POST"
        "browser" : "chrome"
      • Log 3
        "ip" : "192.168.*.*"
        "method" : "POST"
        "browser" : "ali-sls-ilogtail"
    • Logtail plug-in configuration for data processing Filter logs
    • Results
      LogCollectedReason
      Log 1NoThe value of the browser field matches the regular expression that is specified in the Drop Log parameter.
      Log 2YesThe values of the specified fields match the regular expressions that are specified in the Collect Logs parameter and do not match the regular expression that is specified in the Drop Log parameter.
      Log 3NoThe value of the ip field does not match a regular expression that is specified in the Collect Logs parameter.

Editor configuration in JSON

  • Parameters
    Set type to processor_filter_regex. Then, configure other parameters in detail based on the following table.
    Note A log is collected only if the values of fields in the log exactly match the regular expressions that are specified in the Include parameter and do not match the regular expressions that are specified in the Exclude parameter. In other cases, the log is dropped.
    ParameterTypeRequiredDescription
    IncludeJSON ObjectNoThe key-value pairs that are used as match conditions. In a key-value pair, the key specifies a field, and the value specifies a regular expression based on which the value of the field is matched. The key-value pairs are evaluated by using a logical AND. If the values of log fields match the regular expressions that are specified in the Include parameter, the log is collected.
    ExcludeJSON ObjectNoThe key-value pairs that are used as match conditions. In a key-value pair, the key specifies a field, and the value specifies a regular expression based on which the value of the field is matched. The key-value pairs are evaluated by using a logical OR. If the value of a log field matches a regular expression that is specified in the Exclude parameter, the log is dropped.
  • Configuration example

    In this example, only logs whose ip starts with 10, method is POST, and browser is not aliyun.* are collected.

    • Raw logs
      • Log 1
        "ip" : "10.**.**.**"
        "method" : "POST"
        "browser" : "aliyun-sdk-java"
      • Log 2
        "ip" : "10.**.**.**"
        "method" : "POST"
        "browser" : "chrome"
      • Log 3
        "ip" : "192.168.*.*"
        "method" : "POST"
        "browser" : "ali-sls-ilogtail"
    • Logtail plug-in configuration for data processing
      {
         "type" : "processor_filter_regex",
          "detail" : {
               "Include" : {
                  "ip" : "10\\..*",
                  "method" : "POST"
               },
               "Exclude" : {
                  "browser" : "aliyun.*"
               }
          }
      }
    • Results
      LogCollectedReason
      Log 1NoThe value of the browser field matches the regular expression that is specified in the Exclude parameter.
      Log 2YesThe values of the specified fields match the regular expressions that are specified in the Include parameter and do not match the regular expression that is specified in the Exclude parameter.
      Log 3NoThe value of the ip field does not match a regular expression that is specified in the Include parameter.

processor_filter_key_regex

You can use regular expressions to match the names of log fields and filter logs.

Form configuration

  • Parameters
    Set Processor Type to Filter Logs with Regular Expression (Match Log Field Names). Then, configure other parameters based on the following table.
    Note A log is collected only if the names of fields in the log exactly match the regular expressions that are specified in the Collect Logs parameter and do not match the regular expressions that are specified in the Drop Log parameter. In other cases, the log is dropped.
    ParameterDescription
    Collect LogsSpecifies the regular expression based on which the name of a log field is matched. Multiple regular expressions are evaluated by using a logical AND. If the names of log fields match the specified regular expressions, the log is collected.
    Drop LogSpecifies the regular expression based on which the name of a log field is matched. Multiple regular expressions are evaluated by using a logical OR. If the name of a log field matches a specified regular expression, the log is dropped.
  • Configuration example
    In this example, logs whose field names start with request are collected.
    • Raw logs
      • Log 1
        "request_time" : 20
        "request_length": 2314
        "request_method" : "POST"
        "browser" : "aliyun-sdk-java"
      • Log 2
        "request_time" : 70
        "request_method" : "GET"
        "ip" : "192.168.**.**"
      • Log 3
        "browser" : "ali-sls-ilogtail"
        "status" : 200
    • Logtail plug-in configuration for data processing Filter logs
    • Results
      LogCollectedReason
      Log 1YesThe names of the request_time, request_length, and request_method fields match the regular expression that is specified in the Collect Logs parameter.
      Log 2YesThe names of the request_time and request_length fields match the regular expression that is specified in the Collect Logs parameter.
      Log 3NoNo field names in the log match the regular expression that is specified in the Collect Logs parameter.

Editor configuration in JSON

  • Parameters
    Set type to processor_filter_key_regex. Then, configure other parameters in detail based on the following table.
    Note A log is collected only if the names of fields in the log exactly match the regular expressions that are specified in the Include parameter and do not match the regular expressions that are specified in the Exclude parameter. In other cases, the log is dropped.
    ParameterTypeRequiredDescription
    Includestring[]NoSpecifies the regular expression based on which the name of a log field is matched. Multiple regular expressions are evaluated by using a logical AND. If the names of log fields match the specified regular expressions, the log is collected.
    Excludestring[]NoSpecifies the regular expression based on which the name of a log field is matched. Multiple regular expressions are evaluated by using a logical OR. If the name of a log field matches a specified regular expression, the log is dropped.
  • Configuration example
    In this example, logs whose field names start with request are collected.
    • Raw logs
      • Log 1
        "request_time" : 20
        "request_length": 2314
        "request_method" : "POST"
        "browser" : "aliyun-sdk-java"
      • Log 2
        "request_time" : 70
        "request_method" : "GET"
        "ip" : "192.168.**.**"
      • Log 3
        "browser" : "ali-sls-ilogtail"
        "status" : 200
    • Logtail plug-in configuration for data processing
      {
         "type" : "processor_filter_key_regex",
          "detail" : {
               "Include" : [
                  "request.*"
               ]
          }
      }
    • Results
      LogCollectedReason
      Log 1YesThe names of the request_time, request_length, and request_method fields match the regular expression that is specified in the Include parameter.
      Log 2YesThe names of the request_time and request_length fields match the regular expression that is specified in the Include parameter.
      Log 3NoNo field names in the log match the regular expression that is specified in the Include parameter.