configure a plug-in to filter logs - Simple Log Service - Alibaba Cloud Documentation Center

When you use Logtail to collect logs, you can use the processor_filter_regex or processor_filter_key_regex plug-in to filter them. This topic describes the parameters and provides configuration examples for each plug-in.

Limits

Text logs and container standard output support only form-based configuration. Other input plugins support only JSON configuration.
The Go regular expression engine is based on RE2. It has the following limitations compared with the Perl Compatible Regular Expressions (PCRE) engine:
- Differences in named group syntax
  Go uses the (?P<name>...) syntax, not the (?<name>...) syntax used by PCRE.
- Unsupported regex patterns
  - Assertions: (?=...), (?!...), (?<=...), and (?<!...).
  - Conditional expressions: (?(condition)true|false).
  - Recursive matching: (?R) and (?0).
  - Subroutine references: (?&name) and (?P>name).
  - Atomic groups: (?>...).
When you debug regular expressions using tools such as Regex101, avoid the unsupported patterns. Otherwise, the plugin will not be able to process the expressions.

Entry point

To use a Logtail plugin to process logs, you can add a plugin configuration when you create or modify a Logtail collection configuration. For more information, see Overview of Logtail plugins for data processing.

processor_filter_regex

Filters logs by matching the values of log fields against a regular expression.

Form-based configuration

Parameters

Set Processor Type to Filter Logs With Regular Expression (Match Log Field Values). The following table describes the parameters.

Note

A log is collected only if its field value exactly matches the regular expression in Collect Logs and does not match the regular expression in Drop Logs. Otherwise, the log is dropped.

Parameter

Description

Collect Logs

Enter the name of a log field and the regular expression to match the field value. If the value of the target field in a raw log matches the regular expression, the log is collected.

You can add multiple key-value pairs. The key-value pairs have a logical AND relationship.

Drop Logs

Enter the name of a log field and the regular expression to match the field value. If the value of any field in a log matches the corresponding regular expression, the log is dropped.

You can add multiple key-value pairs. The key-value pairs have a logical OR relationship.

Configuration example

This example shows how to collect only logs where the ip value starts with 10, the method value is POST, and the browser value is not aliyun.*.

Raw logs

Log 1

"ip" : "10.**.**.**"
"method" : "POST"
"browser" : "aliyun-sdk-java"

Log 2

"ip" : "10.**.**.**"
"method" : "POST"
"browser" : "chrome"

Log 3

"ip" : "192.168.*.*"
"method" : "POST"
"browser" : "ali-sls-ilogtail"

Logtail plug-in configuration

Processing results

Log	Collected?	Reason
Log 1	No collection	The value of the browser field matches the regular expression in Drop Logs.
Log 2	Data Collection	The conditions are met.
Log 3	No data collection	The value of the ip field does not match the regular expression in Collect Logs.

JSON configuration

Parameters

Set type to processor_filter_regex. The following table describes the parameters in the detail object.

Note

A log is collected only if the value of a log field exactly matches the regular expression in Include and does not match the regular expression in Exclude. Otherwise, the log is dropped.

Parameter	Type	Required	Description
Include	JSON Object	No	The key specifies a log field and the value specifies the regular expression to match the field value. The key-value pairs have a logical AND relationship. If the value of a log field matches the corresponding regular expression, the log is collected.
Exclude	JSON Object	No	The key specifies a log field and the value specifies the regular expression to match the field value. The key-value pairs have a logical OR relationship. If the value of any field in a log matches the corresponding regular expression, the log is dropped.

Example configuration

This example shows how to collect only logs where the ip value starts with 10, the method is POST, and the browser is not aliyun.*.

Raw logs

Log 1

"ip" : "10.**.**.**"
"method" : "POST"
"browser" : "aliyun-sdk-java"

Log 2

"ip" : "10.**.**.**"
"method" : "POST"
"browser" : "chrome"

Log 3

"ip" : "192.168.*.*"
"method" : "POST"
"browser" : "ali-sls-ilogtail"

Logtail plug-in configuration

{
   "type" : "processor_filter_regex",
    "detail" : {
         "Include" : {
            "ip" : "10\..*",
            "method" : "POST"
         },
         "Exclude" : {
            "browser" : "aliyun.*"
         }
    }
}

Processing results

Log	Collected?	Reason
Log 1	Do not collect	The value of the browser field matches the regular expression in Exclude.
Log 2	Collection	The conditions are met.
Log 3	Collection is disabled.	The value of the ip field does not match the regular expression in Include.

processor_filter_key_regex

Filters logs by matching the names of log fields against a regular expression.

Form-based configuration

Parameters

Set Processor Type to Filter Logs With Regular Expression (Match Log Field Names). The following table describes the parameters.

Note

A log is collected only if a log field name exactly matches the regular expression in Collect Logs and does not match the regular expression in Drop Logs. Otherwise, the log is dropped.

Parameter	Description
Collect Logs	Enter a regular expression to match log field names. The expressions have a logical AND relationship. If a log field name matches the corresponding regular expression, the log is collected.
Drop Logs	Enter a regular expression to match log field names. The expressions have a logical OR relationship. If the name of any field in a log matches the corresponding regular expression, the log is dropped.

Configuration example

This example shows how to collect logs that contain field names starting with request.

Raw logs

Log 1

"request_time" : 20
"request_length": 2314
"request_method" : "POST"
"browser" : "aliyun-sdk-java"

Log 2

"request_time" : 70
"request_method" : "GET"
"ip" : "192.168.**.**"

Log 3

"browser" : "ali-sls-ilogtail"
"status" : 200

Logtail plug-in configuration

Processing results

Log	Collected?	Reason
Log 1	Collection	The names of the request_time, request_length, and request_method fields match the regular expression in Collect Logs.
Log 2	Collection	The names of the request_time and request_method fields match the regular expression in Collect Logs.
Log 3	Not collected	No field name in the log matches the regular expression in Collect Logs.

JSON configuration

Parameters

Set type to processor_filter_key_regex. The following table describes the parameters in the detail object.

Note

A log is collected only if a log field name exactly matches the regular expression in Include and does not match the regular expression in Exclude. Otherwise, the log is dropped.

Parameter	Type	Required	Description
Include	string[]	No	Enter a regular expression to match log field names. The expressions have a logical AND relationship. If a log field name matches the corresponding regular expression, the log is collected.
Exclude	string[]	No	Enter a regular expression to match log field names. The expressions have a logical OR relationship. If the name of any field in a log matches the corresponding regular expression, the log is dropped.

Configuration example

This example shows how to collect logs that contain field names starting with request.

Raw logs

Log 1

"request_time" : 20
"request_length": 2314
"request_method" : "POST"
"browser" : "aliyun-sdk-java"

Log 2

"request_time" : 70
"request_method" : "GET"
"ip" : "192.168.**.**"

Log 3

"browser" : "ali-sls-ilogtail"
"status" : 200

Logtail plug-in configuration

{
   "type" : "processor_filter_key_regex",
    "detail" : {
         "Include" : [
            "request.*"
         ]
    }
}

Processing results

Log	Collected?	Reason
Log 1	Collection	The names of the request_time, request_length, and request_method fields match the regular expression in Include.
Log 2	Collection	The names of the request_time and request_method fields match the regular expression in Include.
Log 3	Collection disabled	No field name in the log matches the regular expression in Include.

References

Configure a Logtail pipeline using API operations:
Configure a processor plugin in the console:
- Collect text logs from a host
- Collect container logs from a cluster in the console (stdout/file)
Collect container logs from a Kubernetes cluster using a CRD (stdout/file)