All Products
Search
Document Center

:Parsing in regex mode

Last Updated:Dec 20, 2023

You can use a Logtail plug-in to extract log fields from logs based on a regular expression. The logs are parsed into key-value pairs.

Entry point

If you want to use a Logtail plug-in to process logs, you can add a Logtail plug-in configuration when you create or modify a Logtail configuration. For more information, see Overview.

Configuration description

Parameter

Description

Original Field

The original field that is used to store the content of a log before the log is parsed. Default value: content.

Regular Expression

The regular expression that is used to match logs.

  • If you specify a sample log, Simple Log Service can automatically generate a regular expression or use a regular expression that you manually specify.

    • Click Generate. In the Sample Log field, select the log content that you want to extract and click Generate Regular Expression. Simple Log Service generates a regular expression based on the content that you specified.

    • Click Manual to specify a regular expression. After you configure the settings, click Validate to check whether the regular expression can parse and extract log content as expected. For more information, see How do I test a regular expression?

  • If you do not specify a sample log, you must specify a regular expression based on the actual log content.

Extracted Field

The extracted fields. Configure the Key parameter for each Value parameter. The Key parameter specifies a new field name. The Value parameter specifies the content that is extracted from logs.

Retain Original Field if Parsing Fails

If you select the Retain Original Field if Parsing Fails parameter and parsing fails, the original field is retained.

Retain Original Field if Parsing Succeeds

If you select the Retain Original Field if Parsing Succeeds parameter and parsing is successful, the original field is retained.

New Name of Original Field

If you select the Retain Original Field if Parsing Fails or Retain Original Field if Parsing Succeeds parameter, you can rename the original field to store the original log content.

Configuration example

  • Raw log

    127.0.0.1 - - [16/Oct/2023:12:36:49 +0800] "GET /index.html HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
  • Logtail plug-in configuration

    In this example, the regular expression is set to (\S+)\s-\s(\S+)\s\[([^]]+)]\s"(\w+)\s(\S+)\s([^"]+)"\s(\d+)(\s\d+)[^-]+([^"]+)"\s"([^"]+).*.

    image.png

  • Parsing resultimage.png