Logtail plug-in that is used to parse data in regex mode - Simple Log Service

You can use a Logtail plug-in to extract fields from logs based on a regular expression. In this case, a log is parsed into multiple key-value pairs.

Entry point

If you want to use a Logtail plug-in to process logs, you can add a Logtail plug-in configuration when you create or modify a Logtail configuration. For more information, see Overview.

Configuration description

Parameter	Description
Original Field	The original field that stores the log content before parsing. Default value: content.
Regular Expression	The regular expression that is used to match logs. After you specify a sample log in the Log Sample field in the Processor Configurations section, the sample log is displayed in the dialog box that appears after you set the Processor Type parameter to Data Parsing (Regex Mode). Simple Log Service can automatically generate a regular expression or use the regular expression that you manually specify to match the beginning of the first line of a log. For more information, see Configuration example. Click Generate. In the Log Sample field, select the log content that you want to extract and click Generate Regular Expression. Simple Log Service generates a regular expression based on the selected content. Click Manually Enter Regular Expression to specify a regular expression. After you configure the settings, click Validate to check whether the regular expression can parse and extract the log content as expected. For more information, see How do I test a regular expression? If you do not specify a sample log, you must specify a regular expression based on the actual log content.
Extracted Field	The extracted fields. Configure the Key parameter for each Value parameter. The Key parameter specifies a new field name. The Value parameter specifies the content that is extracted from logs.
Retain Original Field If Parsing Fails	Specifies whether to retain the original field in the new log that is obtained after the raw log fails to be parsed.
Retain Original Field If Parsing Succeeds	Specifies whether to retain the original field in the new log that is obtained after parsing.
New Name of Original Field	The new name of the original field that you want to retain. If you select Retain Original Field If Parsing Fails or Retain Original Field If Parsing Succeeds, you can rename the original field that stores the original log content.

Configuration example

Raw log

127.0.0.1 - - [16/Oct/2023:12:36:49 +0800] "GET /index.html HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"

Logtail plug-in configuration
In this example, the regular expression is set to (\S+)\s-\s(\S+)\s\[([^]]+)]\s"(\w+)\s(\S+)\s([^"]+)"\s(\d+)(\s\d+)[^-]+([^"]+)"\s"([^"]+).*.
Parsing result