Logtail plug-in that is used to parse logs based on delimiters - Simple Log Service

You can use a Logtail plug-in to parse logs into structured data by using delimiters. In this case, a log is parsed into multiple key-value pairs.

Entry point

If you want to use a Logtail plug-in to process logs, you can add a Logtail plug-in configuration when you create or modify a Logtail configuration. For more information, see Overview.

Configuration description

Parameter	Description
Original Field	The original field that stores the log content before parsing. Default value: content.
Delimiter	The delimiter. Select a delimiter based on the actual log content. For example, you can select Vertical Bar (\|). Note If you set the Delimiter parameter to Non-printable Character, you must enter a character in the following format: `0x<Hexadecimal ASCII code of the non-printable character>`. For example, if you want to use a non-printable character whose hexadecimal ASCII code is 01, you must enter 0x01.
Quote	The quote. If a log field contains delimiters, you must specify a quote to enclose the field. Simple Log Service parses the content that is enclosed in a pair of quotes into a complete field. Select a quote based on the format of the logs that you want to collect. Note If you set the Quote parameter to Non-printable Character, you must enter a character in the following format: `0x<Hexadecimal ASCII code of the non-printable character>`. For example, if you want to use a non-printable character whose hexadecimal ASCII code is 01, you must enter 0x01.
Extracted Field	If you specify a sample log, Simple Log Service can automatically extract the log content based on the specified sample log and the delimiter. Configure the Key parameter for each Value parameter. The Key parameter specifies the field name. The Value parameter specifies the extracted content. If you do not specify a sample log, the Value column is unavailable. You must specify keys based on the actual logs and the delimiter. A key can contain only letters, digits, and underscores (_) and must start with a letter or an underscore (_). A key can be up to 128 bytes in length.
Allow Missing Field	Specifies whether to upload a log to Simple Log Service if the number of extracted values is less than the number of specified keys. If you select the Allow Missing Field parameter, the log is uploaded to Simple Log Service. In this example, a log is `11\|22\|33\|44`, the Delimiter parameter is set to Vertical Bar (\|), and the keys are set to `A`, `B`, `C`, `D`, and `E`: The value of the `E` field is empty. If you select the Allow Missing Field parameter, the log is uploaded to Simple Log Service. If you do not select the Allow Missing Field parameter, the log is discarded. Note If you use Linux Logtail V1.0.28 and later or Windows Logtail V1.0.28.0 and later, you can configure the Allow Missing Field parameter when you collect logs in delimiter mode.
Processing Method of Field to which Excess Part is Assigned	The method that is used to process excess values that are extracted if the number of extracted values is greater than the number of specified keys. Valid values: Expand: retains the excess values and adds the values to the fields in the `__column$i__` format. `$i` specifies the sequence number of the excess field. The sequence number starts from 0. Examples: `__column0__` and `__column1__`. Retain: retains the excess values and adds the values to the `__column0__` field. Discard: discards the excess values.
Retain Original Field if Parsing Fails	Specifies whether to retain the original field in the new log that is obtained after the raw log fails to be parsed.
Retain Original Field if Parsing Succeeds	Specifies whether to retain the original field in the new log that is obtained after parsing.
New Name of Original Field	The new name of the original field that you want to retain. If you select Retain Original Field if Parsing Fails or Retain Original Field if Parsing Succeeds, you can rename the original field that stores the original log content.

Appendix

The Logtail plug-in for parsing data in delimiter mode supports single-character delimiters and multi-character delimiters.

Single-character delimiter

The following examples show logs that you can parse by using single-character delimiters:

05/May/2022:13:30:28,10.10.*.*,"POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1",200,18204,aliyun-sdk-java
05/May/2022:13:31:23,10.10.*.*,"POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1",401,23472,aliyun-sdk-java

If you want to use a single-character delimiter, you must specify both the delimiter and quote.

Delimiter: Available single-character delimiters include the tab character (\t), vertical bar (|), space, comma (,), semicolon (;), and non-printable characters. You cannot specify a double quotation mark (") as the delimiter.
However, a double quotation mark (") can be used as a quote. A double quotation mark (") can appear at the border of a field, or in the field. If a double quotation mark (") is included in a log field, it must be escaped as a pair of double quotation marks ("") when the log is processed. When the log is parsed, a pair of double quotation marks ("") are restored to a double quotation mark ("). For example, you can specify a comma (,) as the delimiter and a double quotation mark (") as the quote. If a log field contains the specified delimiter and quote, the field is enclosed in a pair of quotes, and the double quotation mark (") in the field is escaped as a pair of double quotation marks (""). If a processed log is in the 1999,Chevy,"Venture ""Extended Edition, Very Large""","",5000.00 format, the log is parsed into five fields: 1999, Chevy, Venture "Extended Edition, Very Large", an empty field, and 5000.00.
Quote: If a log field contains delimiters, you must specify a quote to enclose the field. Simple Log Service parses the content that is enclosed in a pair of quotes into a complete field.
Available quotes include the tab character (\t), vertical bar (|), space, comma (,), semicolon (;), and non-printable characters.
For example, if you specify a comma (,) as the delimiter and a double quotation mark (") as the quote, the log 1997,Ford,E350,"ac, abs, moon",3000.00 is parsed into five fields: 1997, Ford, E350, ac, abs, moon, and 3000.00.

Multi-character delimiter

The following examples show logs that you can parse by using multi-character delimiters:

05/May/2022:13:30:28&&10.200.**.**&&POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1&&200&&18204&&aliyun-sdk-java
05/May/2022:13:31:23&&10.200.**.**&&POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1&&401&&23472&&aliyun-sdk-java

A multi-character delimiter can contain two or three characters, such as ||, &&&, and ^_^. Simple Log Service parses logs based on delimiters. You do not need to use quotes to enclose log fields.

Important

Make sure that each log field does not contain the exact delimiter. Otherwise, Simple Log Service cannot parse the logs as expected.

For example, if you specify && as the delimiter, the log 1997&&Ford&&E350&&ac&abs&moon&&3000.00 is parsed into five fields: 1997, Ford, E350, ac&abs&moon, and 3000.00.