If you use Logtail to collect logs, you can use the processor_filter_regex or processor_filter_key_regex plug-in to filter logs. This topic describes the parameters of the processor_filter_regex and processor_filter_key_regex plug-ins. This topic also provides examples on how to configure the plug-ins.
Entry point
If you want to use a Logtail plug-in to process logs, you can add a Logtail plug-in configuration when you create or modify a Logtail configuration. For more information, see Entry point.
processor_filter_regex
You can use regular expressions to match the values of log fields and filter logs.
Form configuration
- ParametersSet Processor Type to Filter Logs with Regular Expression (Match Log Field Values). Then, configure other parameters based on the following table.Note A log is collected only if the values of fields in the log exactly match the regular expressions that are specified in the Collect Logs parameter and do not match the regular expressions that are specified in the Drop Log parameter. In other cases, the log is dropped.
Parameter Description Collect Logs Specifies the name of a log field and the regular expression based on which the value of the field is matched. If the values of log fields match the specified regular expressions, the log is collected. You can add multiple key-value pairs. The key-value pairs are evaluated by using a logical AND.
Drop Log Specifies the name of a log field and the regular expression based on which the value of the field is matched. If the value of a log field matches a specified regular expression, the log is dropped. You can add multiple key-value pairs. The key-value pairs are evaluated by using a logical OR.
- Configuration example
In this example, only logs whose ip starts with 10, method is POST, and browser is not aliyun.* are collected.
- Raw logs
- Log 1
"ip" : "10.**.**.**" "method" : "POST" "browser" : "aliyun-sdk-java"
- Log 2
"ip" : "10.**.**.**" "method" : "POST" "browser" : "chrome"
- Log 3
"ip" : "192.168.*.*" "method" : "POST" "browser" : "ali-sls-ilogtail"
- Log 1
- Logtail plug-in configuration for data processing
- Results
Log Collected Reason Log 1 No The value of the browser field matches the regular expression that is specified in the Drop Log parameter. Log 2 Yes The values of the specified fields match the regular expressions that are specified in the Collect Logs parameter and do not match the regular expression that is specified in the Drop Log parameter. Log 3 No The value of the ip field does not match a regular expression that is specified in the Collect Logs parameter.
- Raw logs
Editor configuration in JSON
- ParametersSet type to processor_filter_regex. Then, configure other parameters in detail based on the following table.Note A log is collected only if the values of fields in the log exactly match the regular expressions that are specified in the Include parameter and do not match the regular expressions that are specified in the Exclude parameter. In other cases, the log is dropped.
Parameter Type Required Description Include JSON Object No The key-value pairs that are used as match conditions. In a key-value pair, the key specifies a field, and the value specifies a regular expression based on which the value of the field is matched. The key-value pairs are evaluated by using a logical AND. If the values of log fields match the regular expressions that are specified in the Include parameter, the log is collected. Exclude JSON Object No The key-value pairs that are used as match conditions. In a key-value pair, the key specifies a field, and the value specifies a regular expression based on which the value of the field is matched. The key-value pairs are evaluated by using a logical OR. If the value of a log field matches a regular expression that is specified in the Exclude parameter, the log is dropped. - Configuration example
In this example, only logs whose ip starts with 10, method is POST, and browser is not aliyun.* are collected.
- Raw logs
- Log 1
"ip" : "10.**.**.**" "method" : "POST" "browser" : "aliyun-sdk-java"
- Log 2
"ip" : "10.**.**.**" "method" : "POST" "browser" : "chrome"
- Log 3
"ip" : "192.168.*.*" "method" : "POST" "browser" : "ali-sls-ilogtail"
- Log 1
- Logtail plug-in configuration for data processing
{ "type" : "processor_filter_regex", "detail" : { "Include" : { "ip" : "10\\..*", "method" : "POST" }, "Exclude" : { "browser" : "aliyun.*" } } }
- Results
Log Collected Reason Log 1 No The value of the browser field matches the regular expression that is specified in the Exclude parameter. Log 2 Yes The values of the specified fields match the regular expressions that are specified in the Include parameter and do not match the regular expression that is specified in the Exclude parameter. Log 3 No The value of the ip field does not match a regular expression that is specified in the Include parameter.
- Raw logs
processor_filter_key_regex
You can use regular expressions to match the names of log fields and filter logs.
Form configuration
- ParametersSet Processor Type to Filter Logs with Regular Expression (Match Log Field Names). Then, configure other parameters based on the following table.Note A log is collected only if the names of fields in the log exactly match the regular expressions that are specified in the Collect Logs parameter and do not match the regular expressions that are specified in the Drop Log parameter. In other cases, the log is dropped.
Parameter Description Collect Logs Specifies the regular expression based on which the name of a log field is matched. Multiple regular expressions are evaluated by using a logical AND. If the names of log fields match the specified regular expressions, the log is collected. Drop Log Specifies the regular expression based on which the name of a log field is matched. Multiple regular expressions are evaluated by using a logical OR. If the name of a log field matches a specified regular expression, the log is dropped. - Configuration exampleIn this example, logs whose field names start with request are collected.
- Raw logs
- Log 1
"request_time" : 20 "request_length": 2314 "request_method" : "POST" "browser" : "aliyun-sdk-java"
- Log 2
"request_time" : 70 "request_method" : "GET" "ip" : "192.168.**.**"
- Log 3
"browser" : "ali-sls-ilogtail" "status" : 200
- Log 1
- Logtail plug-in configuration for data processing
- Results
Log Collected Reason Log 1 Yes The names of the request_time, request_length, and request_method fields match the regular expression that is specified in the Collect Logs parameter. Log 2 Yes The names of the request_time and request_length fields match the regular expression that is specified in the Collect Logs parameter. Log 3 No No field names in the log match the regular expression that is specified in the Collect Logs parameter.
- Raw logs
Editor configuration in JSON
- ParametersSet type to processor_filter_key_regex. Then, configure other parameters in detail based on the following table.Note A log is collected only if the names of fields in the log exactly match the regular expressions that are specified in the Include parameter and do not match the regular expressions that are specified in the Exclude parameter. In other cases, the log is dropped.
Parameter Type Required Description Include string[] No Specifies the regular expression based on which the name of a log field is matched. Multiple regular expressions are evaluated by using a logical AND. If the names of log fields match the specified regular expressions, the log is collected. Exclude string[] No Specifies the regular expression based on which the name of a log field is matched. Multiple regular expressions are evaluated by using a logical OR. If the name of a log field matches a specified regular expression, the log is dropped. - Configuration exampleIn this example, logs whose field names start with request are collected.
- Raw logs
- Log 1
"request_time" : 20 "request_length": 2314 "request_method" : "POST" "browser" : "aliyun-sdk-java"
- Log 2
"request_time" : 70 "request_method" : "GET" "ip" : "192.168.**.**"
- Log 3
"browser" : "ali-sls-ilogtail" "status" : 200
- Log 1
- Logtail plug-in configuration for data processing
{ "type" : "processor_filter_key_regex", "detail" : { "Include" : [ "request.*" ] } }
- Results
Log Collected Reason Log 1 Yes The names of the request_time, request_length, and request_method fields match the regular expression that is specified in the Include parameter. Log 2 Yes The names of the request_time and request_length fields match the regular expression that is specified in the Include parameter. Log 3 No No field names in the log match the regular expression that is specified in the Include parameter.
- Raw logs