Log Collection Filter Plugins Overview - Simple Log Service - Alibaba Cloud - Simple Log Service

Data filtering plugin overview

Log Service offers several data filtering plugins. Choose one based on your requirements.

Plugin name

Type

Description

Filtering processor

built-in

Collects only logs where the field value exactly matches a specified allowlist pattern.

filter_regex

custom

Supports the following filtering modes:

Regex filter (by value): Uses a regular expression to match the field value. You can combine a collection condition and a drop condition to specify which logs to collect or discard.
Regex filter (by key): Uses a regular expression to match the field name (key) and filters logs based on the presence of the specified field.

Entry point

To use a Logtail plugin for log processing, add it when you create or modify a Logtail configuration. For more information, see Overview.

Differences between built-in and custom plugins

Built-in plugins: Implemented in C++ for high performance.

Custom plugins: Implemented in Go to provide flexibility and a rich ecosystem. If your service logs are too complex for built-in plugins, consider using a custom plugin.

Performance limitations of custom plugins
- Using a custom plugin to process logs consumes more LoongCollector resources, primarily CPU. If needed, you can use configuration management to adjust the LoongCollector parameter settings.
- If the raw data generation rate exceeds 5 MB/s, avoid using complex plugin combinations. Instead, use a custom plugin for simple processing, followed by data transformation for advanced processing.
Log collection limitations
- Custom plugins process text logs in line mode, storing file-level metadata, such as __tag__:__path__ and __topic__, in each log entry.
- Adding a custom plugin affects tag-related features:
  - The context query and LiveTail feature are unavailable. To use these features, you must add an aggregators configuration.
  - The __topic__ field is renamed to __log_topic__. If you add an aggregators configuration, both the __topic__ and __log_topic__ fields will be present in the logs. If you do not need the __log_topic__ field, you can use the processor_drop plugin to delete it.
  - Fields such as __tag__:__path__ no longer have a built-in field index. You must create a field index for them.

Native filter plugin

The native filter plugin filters logs by their field values.

Configuration

Parameter

Description

Whitelist

Define an allowlist to collect only logs that meet specific conditions. You must specify the target field name and a regular expression for filtering. The regular expression must match the entire string. Partial matches are not supported. For details on writing a regular expression, see Regular Expression Tutorial.

Multiple conditions in the allowlist are joined by a logical AND. The following are examples:

Collecting logs that match a condition
- Set Field Name to level and Field Value to WARNING|ERROR. This configuration collects only logs whose level field value is WARNING or ERROR.
- Set Field Name to content and Field Value to .*05\/Jan\/2023.*. This configuration collects only logs whose content field value contains 05/Jan/2023.
Dropping logs based on a condition
- Set Field Name to level and Field Value to ^(?!.*(INFO|DEBUG)).*. This configuration drops logs whose level field value contains INFO or DEBUG.
- Set Field Name to level and Field Value to ^(?!(INFO|DEBUG)$).*. This configuration drops logs whose level field value is exactly INFO or DEBUG.
- Set Field Name to url and Field Value to .*^(?!.*(healthcheck)).*. This configuration drops logs whose url field value contains healthcheck.

Log filtering plug-ins (advanced)

Use the processor_filter_regex plug-in or the processor_filter_key_regex plug-in to filter logs. This topic describes the parameters and provides configuration examples for each plug-in.

Limitations

The form-based configuration is available only for text logs and container standard output. For all other sources, you must use the JSON-based configuration.
The Go regular expression engine is based on RE2 and has the following limitations compared to the PCRE engine:
- Syntax differences in named capturing groups
  
  Go uses the (?P<name>...) syntax, whereas PCRE uses (?<name>...).
- Unsupported regular expression patterns
  - Lookaround: (?=...), (?!...), (?<=...), and (?<!...).
  - Conditional expressions: (?(condition)true|false).
  - Recursive matching: (?R) and (?0).
  - Subroutine references: (?&name) and (?P>name).
  - Atomic groups: (?>...).
When you debug regular expressions with tools like Regex101, avoid using the unsupported patterns listed above to prevent processing failures.

`processor_filter_regex` (Filter by value)

Filters logs by matching log field values against regular expressions.

Form

Parameters

Set the processor type to Filter Logs with Regular Expression (Match Log Field Values). The following table describes the parameters.

Note

Logtail collects a log only if its field values meet all conditions in Collect Logs and none of the conditions in Drop Log. Otherwise, Logtail discards the log.

Parameter

Description

Collect Logs

Enter a log field name and a regular expression that its value must match. A log is collected if the field's value in the raw log matches the expression.

You can add multiple key-value pairs. These pairs are joined by a logical AND.

Drop Log

Enter a log field name and a regular expression that its value must match. A log is discarded if the field's value matches the expression.

You can add multiple key-value pairs. These pairs are joined by a logical OR.

Configuration example

To collect only logs where the ip starts with 10, the method is POST, and the browser does not match aliyun.*, use the following configuration:

Raw logs

Log 1

"ip" : "10.**.**.**"
"method" : "POST"
"browser" : "aliyun-sdk-java"

Log 2

"ip" : "10.**.**.**"
"method" : "POST"
"browser" : "chrome"

Log 3

"ip" : "192.168.*.*"
"method" : "POST"
"browser" : "ali-sls-ilogtail"

Logtail plug-in configuration

Set the processor type to Filter Logs by Using Regular Expressions (Matching Log Field Values). In the Logs to Be Collected section, add rules for the method field (POST) and the ip field (10\..*). In the Logs to Be Discarded section, add a rule for the browser field (aliyun.*).

Processing results

Log	Collected	Reason
Log 1	No	The value of the browser field matches the regular expression in Drop Log.
Log 2	Yes	The log meets the filtering criteria.
Log 3	No	The value of the ip field does not match the regular expression in Collect Logs.

JSON

Parameters

Set type to processor_filter_regex. The following table describes the parameters in the detail object.

Note

Logtail collects a log only if its field values match all regular expressions in Include and none in Exclude. Otherwise, Logtail discards the log.

Parameter	Type	Required	Description
`Include`	JSON Object	No	Specifies a map where each key is a log field name and the value is a regular expression to match that field's value. These pairs are joined by a logical AND.
`Exclude`	JSON Object	No	Specifies a map where each key is a log field name and the value is a regular expression to match that field's value. These pairs are joined by a logical OR. A log is discarded if any field's value matches its corresponding expression.

Configuration example

To collect only logs where the ip starts with 10, the method is POST, and the browser does not match aliyun.*, use the following configuration:

Raw logs

Log 1

"ip" : "10.**.**.**"
"method" : "POST"
"browser" : "aliyun-sdk-java"

Log 2

"ip" : "10.**.**.**"
"method" : "POST"
"browser" : "chrome"

Log 3

"ip" : "192.168.*.*"
"method" : "POST"
"browser" : "ali-sls-ilogtail"

Logtail plug-in configuration

{
   "type" : "processor_filter_regex",
    "detail" : {
         "Include" : {
            "ip" : "10\..*",
            "method" : "POST"
         },
         "Exclude" : {
            "browser" : "aliyun.*"
         }
    }
}

Processing results

Log	Collected	Reason
Log 1	No	The value of the browser field matches the regular expression in `Exclude`.
Log 2	Yes	The log meets the filtering criteria.
Log 3	No	The value of the ip field does not match the regular expression in `Include`.

`processor_filter_key_regex` (Filter by key)

Filters logs by matching log field names against regular expressions.

Form

Parameters

Set the processor type to Filter Logs with Regular Expression (Match Log Field Names). The following table describes the parameters.

Note

Logtail collects a log only if its field names meet all conditions defined in Collect Logs and meet none of the conditions in Drop Log. Otherwise, Logtail discards the log.

Parameter	Description
Collect Logs	Enter one or more regular expressions to match log field names. A log is collected only if it contains field names that match all specified expressions, which represents a logical AND.
Drop Log	Enter one or more regular expressions to match log field names. A log is discarded if it contains a field name that matches any specified expression, which represents a logical OR.

Configuration example

To collect logs that contain a field name starting with request, use the following configuration:

Raw logs

Log 1

"request_time" : 20
"request_length": 2314
"request_method" : "POST"
"browser" : "aliyun-sdk-java"

Log 2

"request_time" : 70
"request_method" : "GET"
"ip" : "192.168.**.**"

Log 3

"browser" : "ali-sls-ilogtail"
"status" : 200

Set the processor type to Filter Logs by Using Regular Expressions (Matching Log Field Names). In the Logs to Be Collected section, add the regular expression request.*, and leave the Logs to Be Discarded section empty.

Processing results

Log	Collected	Reason
Log 1	Yes	The request_time, request_length, and request_method field names match the regular expression in Collect Logs.
Log 2	Yes	The request_time and request_method field names match the regular expression in Collect Logs.
Log 3	No	The log does not contain any field name that matches the regular expression in Collect Logs.

JSON

Parameters

Set type to processor_filter_key_regex. The following table describes the parameters in the detail object.

Note

Logtail collects a log only if its field names meet all conditions defined in Include and meet none of the conditions in Exclude. Otherwise, Logtail discards the log.

Parameter	Type	Required	Description
`Include`	string[]	No	An array of regular expressions for log field names. These expressions are joined by a logical AND, meaning a log is collected only if it contains field names that match all expressions in the array.
`Exclude`	string[]	No	An array of regular expressions for log field names. These expressions are joined by a logical OR, meaning a log is discarded if it contains a field name that matches any expression in the array.

Configuration example

To collect logs that contain a field name starting with request, use the following configuration:

Raw logs

Log 1

"request_time" : 20
"request_length": 2314
"request_method" : "POST"
"browser" : "aliyun-sdk-java"

Log 2

"request_time" : 70
"request_method" : "GET"
"ip" : "192.168.**.**"

Log 3

"browser" : "ali-sls-ilogtail"
"status" : 200

Logtail plug-in configuration

{
   "type" : "processor_filter_key_regex",
    "detail" : {
         "Include" : [
            "request.*"
         ]
    }
}

Processing results

Log	Collected	Reason
Log 1	Yes	The request_time, request_length, and request_method field names match the regular expression in `Include`.
Log 2	Yes	The request_time and request_method field names match the regular expression in `Include`.
Log 3	No	The log does not contain any field name that matches the regular expression in `Include`.

References

Configure a Logtail pipeline using the API:
Configure a processor plug-in in the console:
- Continuously collect text logs from hosts
- Collect container logs (stdout/file) from a cluster using the console
Collect container logs (stdout/file) from a cluster using a Kubernetes CRD

Data filtering plugin overview

Entry point

Differences between built-in and custom plugins

Native filter plugin

Configuration

Log filtering plug-ins (advanced)

Limitations

processor_filter_regex (Filter by value)

Form

JSON

processor_filter_key_regex (Filter by key)

Form

JSON

References

`processor_filter_regex` (Filter by value)

`processor_filter_key_regex` (Filter by key)