All Products
Search

This Product

    Simple Log Service:Field processing plugins

    Document Center

    Simple Log Service:Field processing plugins

    Last Updated:Jun 12, 2026

    Field processing plugins add, delete, modify, pack, expand, and extract fields.

    Field processing example

    This table details the data structure of a raw log stored in SLS. It compares using the field extraction plugin in anchor mode to not using a plugin. Using the field extraction plugin structures your data, simplifying subsequent queries.

    Raw log

    Without plugin

    Anchor mode

    		 
    "time:2022.09.12 20:55:36\t json:{\"key1\" : \"xx\", \"key2\": false, \"key3\":123.456, \"key4\" : { \"inner1\" : 1, \"inner2\" : false}}"

    Content: "time:2022.09.12 20:55:36\t json:{\"key1\" : \"xx\", \"key2\": false, \"key3\":123.456, \"key4\" : { \"inner1\" : 1, \"inner2\" : false}}"

    Use anchor mode to extract field values, and set the field names to time, val_key1, val_key2, val_key3, value_key4_inner1, and value_key4_inner2.

    "time" : "2022.09.12 20:55:36"
"val_key1" : "xx"
"val_key2" : "false"
"val_key3" : "123.456"
"value_key4_inner1" : "1"
"value_key4_inner2" : "false"

    Field processing plugins

    SLS provides the following types of field processing plugins. Select one that meets your needs.

    Plugin name

    Type

    Description

    Extract fields

    Extended

    Supports the following modes:

    • Regex mode: Extracts fields using regular expression matching.

    • Anchor mode: Extracts fields by position or marker.

    • CSV mode: Extracts fields in CSV format.

    • Single-character delimiter mode: Extracts fields using a single-character delimiter.

    • Multi-character delimiter mode: Extracts fields using a multi-character delimiter.

    • Key-value pair mode: Extracts fields from a key-value pair format.

    • Grok mode: Extracts structured fields using Grok syntax.

    Add fields

    Extended

    Adds new fields to a log.

    Drop fields

    Extended

    Removes specified fields.

    Rename fields

    Extended

    Renames fields.

    Pack fields

    Extended

    Packs multiple fields into a single JSON object.

    Expand JSON fields

    Extended

    Expands a JSON string field into separate fields.

    Map field values

    Extended

    Replaces or transforms field values based on a mapping table.

    Replace strings

    Extended

    Performs full-text replacement, regular expression-based replacement, or escape character removal in text logs.

    Entry point

    To use a Logtail plugin for log processing, add it when you create or modify a Logtail configuration. For more information, see Overview.

    Limitations

    • Text logs and container standard output support only form-based configuration, while all other input sources support only JSON-based configuration.

    • The following limitations apply when extracting fields in regex mode.

      The Go regular expression engine, based on RE2, has the following limitations compared to the PCRE engine:

      • Differences in named group syntax

        Go uses the (?P<name>...) syntax instead of the PCRE syntax (?<name>...).

      • Unsupported regular expression patterns

        • Assertion: (?=...), (?!...), (?<=...), and (?<!...).

        • Conditional expression: (?(condition)true|false).

        • Recursive matching: (?R) and (?0).

        • Subroutine reference: (?&name) and (?P>name).

        • Atomic group: (?>...).

      When debugging regular expressions with tools such as Regex101, avoid the unsupported patterns listed above, as the plugin cannot process logs that use them.

    Field extraction plugin

    Extracts a log field using regex mode, anchor mode, CSV mode, single-character delimiter mode, multi-character delimiter mode, key-value pair mode, or Grok mode.

    Regex mode

    Extracts fields using a regular expression.

    Form

    • Parameters

      Set processor type to Extract Field (Regex Mode). The following table describes the parameters.

      Parameter

      Description

      Original Field

      The name of the source field.

      Regular Expression

      The regular expression. You must use parentheses () to specify the fields to extract.

      New Field

      The names to assign to the extracted content. You can add multiple field names.

      Report Original Field Missing Error

      Select this option to report an error if the specified source field is not found in the raw log.

      Report Regex Mismatch Error

      Select this option to report an error if the value of the source field does not match the regular expression.

      Retain Original Field

      Select this option to retain the source field in the parsed log.

      Retain Original Field If Parsing Fails

      Select this option to retain the source field in the parsed log if parsing fails.

      Full Regex Match

      Select this option to extract values only if the regular expression finds matches for all fields that you defined in New Field.

    • Configuration example

      Use regex mode to extract the value of the content field, and set the field names to ip, time, method, url, request_time, request_length, status, length, ref_url, and browser. An example configuration is as follows:

      • Raw log

        "content" : "10.200.**.** - - [10/Aug/2022:14:57:51 +0800] \"POST /PutData?
Category=YunOsAccountOpLog&AccessKeyId=<yourAccessKeyId>&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=<yourSignature> HTTP/1.1\" 0.024 18204 200 37 \"-\" \"aliyun-sdk-java"

      • Logtail plugin configuration: Set source field to content, configure the regular expression, and set the result field names to ip, time, method, url, request_time, request_length, status, length, ref_url, and browser. Select the report error if source field is missing and report error on regex mismatch options.

      • Result

        "ip" : "10.200.**.**"
"time" : "10/Aug/2022:14:57:51"
"method" : "POST"
"url" : "/PutData?Category=YunOsAccountOpLog&AccessKeyId=<yourAccessKeyId>&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=<yourSignature>"
"request_time" : "0.024"
"request_length" : "18204"
"status" : "200"
"length" : "27"
"ref_url" : "-"
"browser" : "aliyun-sdk-java"

    JSON

    • Parameters

      Set type to processor_regex. The following table describes the parameters for the detail object.

      Parameter

      Type

      Required

      Description

      SourceKey

      String

      Yes

      The name of the source field.

      Regex

      String

      Yes

      The regular expression. You must use parentheses () to specify the fields to extract.

      Keys

      Array of strings

      Yes

      The names to assign to the extracted content. Example: ["ip", "time", "method"].

      NoKeyError

      Boolean

      No

      Specifies whether to report an error if the source field is not found in the raw log.

      • true: Reports an error.

      • false (default): Does not report an error.

      NoMatchError

      Boolean

      No

      Specifies whether to report an error if the value of the source field does not match the regular expression.

      • true (default): Reports an error.

      • false: Does not report an error.

      KeepSource

      Boolean

      No

      Specifies whether to retain the source field in the parsed log.

      • true: Retains the source field.

      • false (default): Does not retain the source field.

      FullMatch

      Boolean

      No

      Specifies whether to require a full match for extraction.

      • true (default value): Field values are extracted only if the regular expression in the Regex parameter finds a match for all fields specified in the Keys parameter within the source field value.

      • false: Field values are extracted even if only a partial match is found.

      KeepSourceIfParseError

      Boolean

      No

      Specifies whether to retain the source field in the parsed log if parsing fails.

      • true (default): Retains the source field.

      • false: Does not retain the source field.

    • Configuration example

      This example shows how to use regex mode to extract values from the content field and assign them to the ip, time, method, url, request_time, request_length, status, length, ref_url, and browser result fields.

      • Raw log

        "content" : "10.200.**.** - - [10/Aug/2022:14:57:51 +0800] \"POST /PutData?
Category=YunOsAccountOpLog&AccessKeyId=<yourAccessKeyId>&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=<yourSignature> HTTP/1.1\" 0.024 18204 200 37 \"-\" \"aliyun-sdk-java"

      • Logtail plugin configuration

        {
    "type" : "processor_regex",
    "detail" : {"SourceKey" : "content",
         "Regex" : "([\\d\\.]+) \\S+ \\S+ \\[(\\S+) \\S+\\] \"(\\w+) ([^\\\"]*)\" ([\\d\\.]+) (\\d+) (\\d+) (\\d+|-) \"([^\\\"]*)\" \"([^\\\"]*)\" (\\d+)",
         "Keys"   : ["ip", "time", "method", "url", "request_time", "request_length", "status", "length", "ref_url", "browser"],
         "NoKeyError" : true,
         "NoMatchError" : true,
         "KeepSource" : false
    }
}

      • Result

        "ip" : "10.200.**.**"
"time" : "10/Aug/2022:14:57:51"
"method" : "POST"
"url" : "/PutData?Category=YunOsAccountOpLog&AccessKeyId=<yourAccessKeyId>&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=<yourSignature>"
"request_time" : "0.024"
"request_length" : "18204"
"status" : "200"
"length" : "27"
"ref_url" : "-"
"browser" : "aliyun-sdk-java"

    Anchor mode

    Extracts fields by specifying start and end keywords. If a field is in JSON format, you can also perform JSON expansion.

    Console

    • Parameters

      Set processor type to Extract Field (Anchor Mode). The following table describes the parameters.

      Parameter

      Description

      Original Field

      The name of the original field.

      Anchor Parameters

      A list of anchor configurations for field extraction.

      Start Keyword

      The keyword marking the start of the text to extract. If left empty, extraction begins at the start of the original field's value.

      End Keyword

      The keyword marking the end of the text to extract. If left empty, extraction continues to the end of the original field's value.

      New Field

      The name of the field that stores the extracted content.

      Field Type

      The data type of the field. Valid values are string and json.

      JSON Expansion

      Specifies whether to perform JSON expansion.

      Character to Concatenate Expanded Keys

      The character used to join keys during JSON expansion. The default is an underscore (_).

      Maximum Depth of JSON Expansion

      The maximum recursion depth for JSON expansion. The default value is 0, which indicates no limit.

      Report Original Field Missing Error

      If enabled, reports an error if the specified original field is not found in the raw log.

      Report Keywords Missing Error

      If enabled, reports an error if no matching anchor is found in the raw log.

      Retain Original Field

      If enabled, retains the original field in the parsed log.

    • Configuration example

      This example shows how to use anchor mode to extract values from the content field and create the result fields time, val_key1, val_key2, val_key3, value_key4_inner1, and value_key4_inner2.

      • Raw log 

        "content" : "time:2022.09.12 20:55:36\t json:{\"key1\" : \"xx\", \"key2\": false, \"key3\":123.456, \"key4\" : { \"inner1\" : 1, \"inner2\" : false}}"

      • Logtail plugin configuration: Configure two rules. For the first rule, set the source field name to time, the start delimiter to \t, the destination field name to time, and the destination type to string. For the second rule, set the source field name to json, the destination field name to val, the destination type to json, and enable JSON expansion.

      • Result 

        "time" : "2022.09.12 20:55:36"
"val_key1" : "xx"
"val_key2" : "false"
"val_key3" : "123.456"
"value_key4_inner1" : "1"
"value_key4_inner2" : "false"

    JSON

    • Parameters

      Set type to processor_anchor. The detail parameter is described in the following table.

      Parameter

      Type

      Required

      Description

      SourceKey

      String

      Yes

      The name of the original field.

      Anchors

      Array of Anchor objects

      Yes

      The list of anchor parameters.

      Start

      String

      Yes

      The start keyword. If left empty, matching starts at the beginning of the string.

      Stop

      String

      Yes

      The end keyword. If left empty, the match extends to the end of the string.

      FieldName

      String

      Yes

      The name of the field for the extracted content.

      FieldType

      String

      Yes

      The data type of the field. Valid values are string and json.

      ExpandJson

      Boolean

      No

      Specifies whether to perform JSON expansion.

      • true: Perform JSON expansion.

      • false (default): Do not perform JSON expansion.

      Valid only when FieldType is json.

      ExpandConnector

      String

      No

      The character used to join keys during JSON expansion. The default is an underscore (_).

      MaxExpandDepth

      Int

      No

      The maximum depth of JSON expansion. The default value is 0, which indicates no limit.

      NoAnchorError

      Boolean

      No

      Specifies whether to report an error if no matching anchor is found.

      • true: Reports an error.

      • false (default): Does not report an error.

      NoKeyError

      Boolean

      No

      Specifies whether to report an error if the specified original field is not found in the raw log.

      • true: Reports an error.

      • false (default): Does not report an error.

      KeepSource

      Boolean

      No

      Specifies whether to retain the original field after parsing.

      • true: Retains the original field.

      • false (default): Discards the original field.

    • Configuration example

      Use anchor mode to extract the value of the content field, and set the field names to time, val_key1, val_key2, val_key3, value_key4_inner1, and value_key4_inner2. The following provides a configuration example:

      • Raw log 

        "content" : "time:2022.09.12 20:55:36\t json:{\"key1\" : \"xx\", \"key2\": false, \"key3\":123.456, \"key4\" : { \"inner1\" : 1, \"inner2\" : false}}"

      • Logtail plugin configuration 

        {
   "type" : "processor_anchor",
   "detail" : {"SourceKey" : "content",
      "Anchors" : [
          {
              "Start" : "time:",
              "Stop" : "\t",
              "FieldName" : "time",
              "FieldType" : "string",
              "ExpandJson" : false
          },
          {
              "Start" : "json:",
              "Stop" : "",
              "FieldName" : "val",
              "FieldType" : "json",
              "ExpandJson" : true 
          }
      ]
  }
}

      • Result 

        "time" : "2022.09.12 20:55:36"
"val_key1" : "xx"
"val_key2" : "false"
"val_key3" : "123.456"
"value_key4_inner1" : "1"
"value_key4_inner2" : "false"

    CSV mode

    Use CSV mode to parse CSV-formatted logs.

    Form

    • Parameters

      Set Processor type to Extract Field (CSV Mode). The following table describes the parameters.

      Parameter

      Description

      Original Field

      The name of the source field.

      New Field

      Specify one or more field names for the extracted values.

      Important

      If the number of fields to be split is less than the number of fields in New Field, the excess fields in New Field are ignored.

      Delimiter

      The delimiter used to separate values. The default is a comma (,).

      Retain Excess Part

      If you select this option and the number of fields to be split is greater than the number of fields in the New Field, the system retains the excess part.

      Parse Excess Part

      Select this option to parse the excess content. Use Name Prefix of Field to which Excess Part is Assigned to specify the name prefix for the excess fields.

      If you select Retain Excess Part but not Parse Excess Part, the excess content is saved to the _decode_preserve_ field.

      Note

      If the excess content does not conform to the CSV format, you must normalize it before storage.

      Name Prefix of Field to which Excess Part is Assigned

      The name prefix for the excess fields. For example, if you set this parameter to expand_, the fields are named expand_1, expand_2, and so on.

      Ignore Spaces before Field

      Select this option to trim leading spaces from field values.

      Retain Original Field

      Select this option to retain the source field in the parsed log.

      Report Original Field Missing Error

      Select this option to report an error if the source field is missing.

    • Configuration example

      The following example shows how to extract values from the csv field:

      • Raw log

        {
    "csv": "2022-06-09,192.0.2.0,\"{\"\"key1\"\":\"\"value\"\",\"\"key2\"\":{\"\"key3\"\":\"\"string\"\"}}\"",
    ......
}

      • Logtail plugin configuration: Set Source field to csv, Delimiter to a comma (,), and Result field to date, ip, and content.

      • Result

        {
    "date": "2022-06-09",
    "ip": "192.0.2.0",
    "content": "{\"key1\":\"value\",\"key2\":{\"key3\":\"string\"}}"
    ......

}

    JSON

    • Parameters

      Set type to processor_csv. The following table describes detail.

      Parameter

      Type

      Required

      Description

      SourceKey

      String

      Yes

      The name of the source field.

      SplitKeys

      String array

      Yes

      The names of the result fields. Example: ["date", "ip", "content"].

      Important

      If the number of fields to be split is less than the number of fields in the SplitKeys parameter, the excess fields in the SplitKeys parameter are ignored.

      PreserveOthers

      Boolean

      No

      Specifies whether to retain the excess part of the source field if it contains more values than specified in SplitKeys.

      • true: Retains the excess part.

      • false (default): Does not retain the excess part.

      ExpandOthers

      Boolean

      No

      Specifies whether to parse the excess part.

      • true: Parses the excess part.

        You can use the ExpandOthers parameter to parse the excess part, and then specify a naming prefix for the fields of the excess part by using the ExpandKeyPrefix parameter.

      • false (default): Does not parse the excess part.

        If you set PreserveOthers to true and ExpandOthers to false, the excess content is saved to the _decode_preserve_ field.

        Note

        If the excess content does not conform to the CSV format, you must normalize it before storage.

      ExpandKeyPrefix

      String

      No

      The name prefix for the excess fields. For example, if you set this parameter to expand_, the fields are named expand_1, expand_2, and so on.

      TrimLeadingSpace

      Boolean

      No

      Specifies whether to trim leading spaces from field values.

      • true: Trims leading spaces.

      • false (default): Does not trim leading spaces.

      SplitSep

      String

      No

      The delimiter that separates values. The default is a comma (,).

      KeepSource

      Boolean

      No

      Specifies whether to retain the source field in the parsed log.

      • true: Retains the source field.

      • false (default): Does not retain the source field.

      NoKeyError

      Boolean

      No

      Specifies whether to report an error if the source field is missing from the raw log.

      • true: Reports an error.

      • false (default): Does not report an error.

    • Configuration example

      The following example shows how to extract values from the csv field:

      • Raw log

        {
    "csv": "2022-06-09,192.0.2.0,\"{\"\"key1\"\":\"\"value\"\",\"\"key2\"\":{\"\"key3\"\":\"\"string\"\"}}\"",
    ......
}

      • Logtail plugin configuration

         {
    ......
    "type":"processor_csv",
    "detail":{
        "SourceKey":"csv",
        "SplitKeys":["date", "ip", "content"],
    }
    ......
}

      • Result

        {
    "date": "2022-06-09",
    "ip": "192.0.2.0",
    "content": "{\"key1\":\"value\",\"key2\":{\"key3\":\"string\"}}"
    ......

}

    Single-character delimiter mode

    Note

    Extracts fields using a single-character delimiter. This mode supports using a quote character to enclose fields that contain the delimiter.

    Form

    • Parameters

      Set processor type to Extract Field (Single-character Delimiter Mode). The following table describes the parameters.

      Parameter

      Description

      Original Field

      The name of the source field.

      Delimiter

      Delimiter. This must be a single character and can be set to a non-printable character, such as \u0001.

      New Field

      Specifies the names of the fields to create for the extracted content.

      Use Quote

      Specifies whether to use a quote character.

      Quote

      Quote character. This must be a single character and can be a non-printable character, such as \u0001.

      Report Original Field Missing Error

      Specifies whether to report an error if the specified source field is not found in the raw log.

      Report Delimiter Mismatch Error

      Specifies whether to report an error if the specified delimiter is not found in the raw log.

      Retain Original Field

      Specifies whether to retain the source field in the parsed log.

    • Example

      Use the vertical bar (|) as a delimiter to extract the value of the content field, and set the field names to ip, time, method, url, request_time, request_length, status, length, ref_url, and browser. The following is a configuration example:

      • Raw log

        "content" : "10.**.**.**|10/Aug/2022:14:57:51 +0800|POST|PutData?
Category=YunOsAccountOpLog&AccessKeyId=<yourAccessKeyId>&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=<yourSignature>|0.024|18204|200|37|-|
aliyun-sdk-java"

      • Logtail plugin configuration: The delimiter is set to a vertical bar (|), and the result fields include ip, time, method, url, request_time, request_length, status, length, ref_url, and browser.

      • Result 

        "ip" : "10.**.**.**"
"time" : "10/Aug/2022:14:57:51 +0800"
"method" : "POST"
"url" : "/PutData?Category=YunOsAccountOpLog&AccessKeyId=<yourAccessKeyId>&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=<yourSignature>"
"request_time" : "0.024"
"request_length" : "18204"
"status" : "200"
"length" : "27"
"ref_url" : "-"
"browser" : "aliyun-sdk-java"

    JSON

    • Parameters

      Set type to processor_split_char. The following table describes the detail parameter.

      Parameter

      Type

      Required

      Description

      SourceKey

      String

      Yes

      The name of the source field.

      SplitSep

      String

      Yes

      The delimiter must be a single character and can be set to a non-printable character, such as \u0001.

      SplitKeys

      String array

      Yes

      An array of strings that specifies the names for the new fields. Example: ["ip", "time", "method"].

      PreserveOthers

      Boolean

      No

      Specifies whether to retain the excess part when the number of fields to be split is greater than the number of fields in the SplitKeys parameter.

      • true: Retain the excess part.

      • false (default): Do not retain the excess part.

      QuoteFlag

      Boolean

      No

      Specifies whether to use a quote character.

      • true: Use a quote character.

      • false (default): Do not use a quote character.

      Quote

      String

      No

      Quote character. It must be a single character and can be a non-printable character, such as \u0001.

      This parameter is effective only when QuoteFlag is set to true.

      NoKeyError

      Boolean

      No

      Specifies whether to report an error if the specified source field is not found in the raw log.

      • true: Report an error.

      • false (default): Do not report an error.

      NoMatchError

      Boolean

      No

      Specifies whether to report an error if the specified delimiter is not found in the raw log.

      • true: Report an error.

      • false (default): Do not report an error.

      KeepSource

      Boolean

      No

      Specifies whether to retain the source field in the parsed log.

      • true: Retain the source field.

      • false (default): Do not retain the source field.

    • Example

      Use the vertical bar (|) delimiter to extract the value of the content field, and set the field names to ip, time, method, url, request_time, request_length, status, length, ref_url, and browser. The following is a configuration example:

      • Raw log

        "content" : "10.**.**.**|10/Aug/2022:14:57:51 +0800|POST|PutData?
Category=YunOsAccountOpLog&AccessKeyId=<yourAccessKeyId>&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=<yourSignature>|0.024|18204|200|37|-|
aliyun-sdk-java"

      • Logtail plugin configuration

        {
   "type" : "processor_split_char",
   "detail" : {"SourceKey" : "content",
      "SplitSep" : "|",
      "SplitKeys" : ["ip", "time", "method", "url", "request_time", "request_length", "status", "length", "ref_url", "browser"]     
  }
}

      • Result

        "ip" : "10.**.**.**"
"time" : "10/Aug/2022:14:57:51 +0800"
"method" : "POST"
"url" : "/PutData?Category=YunOsAccountOpLog&AccessKeyId=<yourAccessKeyId>&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=<yourSignature>"
"request_time" : "0.024"
"request_length" : "18204"
"status" : "200"
"length" : "27"
"ref_url" : "-"
"browser" : "aliyun-sdk-java"

    Multi-character delimiter mode

    Note

    You can extract fields with a multi-character separator. This mode does not support escaping the separator with quotation marks.

    Console

    • Parameters

      Set processor type to Extract Field (Multi-character Delimiter Mode). The following table describes the parameters.

      Parameter

      Description

      Original Field

      The name of the original field.

      Delimiter String

      The separator. You can specify non-printable characters, such as \u0001\u0002.

      New Field

      The names of the extracted fields.

      Important

      If the number of resulting fields is less than the number of fields in New Field, the excess fields in New Field are ignored.

      Report Original Field Missing Error

      Select this option to report an error if the specified original field is not found in the log.

      Report Delimiter Mismatch Error

      Select this option to report an error if the specified separator is not found in the original field.

      Retain Original Field

      Select this option to retain the original field in the parsed log.

      Retain Excess Part

      Select this option to retain the excess content if the split results in more values than the number of names specified in New Field.

      Parse Excess Part

      Select this option to parse the excess content into new fields if the split results in more values than the number of names specified in New Field. You can then use Name Prefix of Field to which Excess Part is Assigned to specify a prefix for the names of these new fields.

      Name Prefix of Field to which Excess Part is Assigned

      A prefix for the names of the excess fields. For example, if you set this parameter to expand_, the excess fields are named expand_1, expand_2, and so on.

    • Configuration example

      Use the separator |#| to extract values from the content field and create the fields ip, time, method, url, request_time, request_length, status, expand_1, expand_2, and expand_3. The following is an example configuration:

      • Raw log

        "content" : "10.**.**.**|#|10/Aug/2022:14:57:51 +0800|#|POST|#|PutData?
Category=YunOsAccountOpLog&AccessKeyId=<yourAccessKeyId>&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=<yourSignature>|#|0.024|#|18204|#|200|#|27|#|-|#|
aliyun-sdk-java"

      • Logtail plugin configuration: Set delimiter string to |#|, set excess part field name prefix to expand_, and add ip, time, method, url, request_time, request_length, and status to new field.

      • Result

        "ip" : "10.**.**.**"
"time" : "10/Aug/2022:14:57:51 +0800"
"method" : "POST"
"url" : "/PutData?Category=YunOsAccountOpLog&AccessKeyId=<yourAccessKeyId>&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=<yourSignature>"
"request_time" : "0.024"
"request_length" : "18204"
"status" : "200"
"expand_1" : "27"
"expand_2" : "-"
"expand_3" : "aliyun-sdk-java"

    JSON

    • Parameters

      Set type to processor_split_string. The following table describes the parameters in detail.

      Parameter

      Type

      Required

      Description

      SourceKey

      String

      Yes

      The name of the original field.

      SplitSep

      String

      Yes

      The separator. You can specify non-printable characters, such as \u0001\u0002.

      SplitKeys

      Array of strings

      Yes

      The names for the extracted fields, specified as an array of strings. Example: ["key1","key2"].

      Note

      If the number of fields to be split is less than the number of fields in the SplitKeys parameter, the excess fields in the SplitKeys parameter are ignored.

      PreserveOthers

      Boolean

      No

      Specifies whether to retain the excess part if the split results in more values than the number of keys in the SplitKeys array.

      • true: Retain the excess part.

      • false (default): Do not retain the excess part.

      ExpandOthers

      Boolean

      No

      Specifies whether to parse the excess part if the split results in more values than the number of keys in the SplitKeys array.

      • true: Parse the excess part.

        You can use the ExpandOthers parameter to parse the excess part and then use the ExpandKeyPrefix parameter to specify a naming prefix for the fields of the excess part.

      • false (default): Do not parse the excess part.

      ExpandKeyPrefix

      String

      No

      A prefix for the names of the excess fields. For example, if you set this parameter to expand_, the excess fields are named expand_1, expand_2, and so on.

      NoKeyError

      Boolean

      No

      Specifies whether to report an error if the field specified by SourceKey is not found in the raw log.

      • true: Report an error.

      • false (default): Do not report an error.

      NoMatchError

      Boolean

      No

      Specifies whether to report an error if the separator specified by SplitSep is not found in the value of the SourceKey field.

      • true: Report an error.

      • false (default): Do not report an error.

      KeepSource

      Boolean

      No

      Specifies whether to retain the original field in the parsed log.

      • true: Retain the original field.

      • false (default): Do not retain the original field.

    • Configuration example

      Use the separator |#| to extract values from the content field and create the fields ip, time, method, url, request_time, request_length, status, expand_1, expand_2, and expand_3. The following is an example configuration:

      • Raw log

        "content" : "10.**.**.**|#|10/Aug/2022:14:57:51 +0800|#|POST|#|PutData?
Category=YunOsAccountOpLog&AccessKeyId=<yourAccessKeyId>&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=<yourSignature>|#|0.024|#|18204|#|200|#|27|#|-|#|
aliyun-sdk-java"

      • Logtail plugin configuration

        {
   "type" : "processor_split_string",
   "detail" : {"SourceKey" : "content",
      "SplitSep" : "|#|",
      "SplitKeys" : ["ip", "time", "method", "url", "request_time", "request_length", "status"],
      "PreserveOthers" : true,
      "ExpandOthers" : true,
      "ExpandKeyPrefix" : "expand_"
  }
}

      • Result

        "ip" : "10.**.**.**"
"time" : "10/Aug/2022:14:57:51 +0800"
"method" : "POST"
"url" : "/PutData?Category=YunOsAccountOpLog&AccessKeyId=<yourAccessKeyId>&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=<yourSignature>"
"request_time" : "0.024"
"request_length" : "18204"
"status" : "200"
"expand_1" : "27"
"expand_2" : "-"
"expand_3" : "aliyun-sdk-java"

    Key-value pair mode

    Extracts fields by splitting key-value pairs.

    Note

    The processor_split_key_value plugin is supported in Logtail 0.16.26 and later.

    Form

    • Parameters

      Set processor type to Extract Field (Key-value Pair Mode). The following table describes the parameters.

      Parameter

      Description

      Original Field

      The name of the source field.

      Key-value Pair Delimiter

      The delimiter between key-value pairs is the tab character \t by default.

      Key and Value Delimiter

      The separator between a key and a value in a key-value pair. The default value is a colon (:).

      Retain Original Field

      Retains the source field in the parsed log.

      Report Original Field Missing Error

      Reports an error if the specified source field is not found in the raw log.

      Drop Key-value Pairs That Fail to Match Delimiter

      Discards a key-value pair if it does not contain the specified key-value separator.

      Report Key and Value Delimiter Missing Error

      Reports an error if a key-value pair does not contain the specified key-value separator.

      Error on empty key

      Reports an error if a key is empty after splitting.

      Quote

      If a value is enclosed by the specified quote character, the content within the quotes is extracted. The quote character can consist of multiple characters.

      Important

      A backslash (\) preceding a quote character within a quoted value is treated as a literal character and included in the final value.

    • Examples

      • Example 1: Split key-value pairs.

        Split the value of the content field into key-value pairs. The delimiter between key-value pairs is a tab character \t, and the delimiter between a key and a value is a colon (:). The following is a configuration example:

        • Raw log

          "content": "class:main\tuserid:123456\tmethod:get\tmessage:\"wrong user\""

        • Logtail plugin configuration: Set key-value pair delimiter to \t, set key-value separator to a colon (:), and select the retain source field option.

        • Result

          "content": "class:main\tuserid:123456\tmethod:get\tmessage:\"wrong user\""
"class": "main"
"userid": "123456"
"method": "get"
"message": "\"wrong user\""

      • Example 2: Split key-value pairs with quotes.

        Split the value of the content field by using key-value pairs. The delimiter between pairs is the tab character \t, the delimiter within a pair is a colon (:), and the quote character is a double quote ("). An example configuration is as follows:

        • Raw log

          "content": "class:main http_user_agent:\"User Agent\" \"中文\" \"hello\\t\\\"ilogtail\\\"\\tworld\""

        • Logtail plugin configuration: The key-value pair delimiter is set to \t, the key-value separator is set to a colon (:), the quote character is set to a double quote ("), and the retain source field option is disabled.

        • Result

          "class": "main",
"http_user_agent": "User Agent",
"no_separator_key_0": "中文",
"no_separator_key_1": "hello\t\"ilogtail\"\tworld",

      • Example 3: Split key-value pairs with multi-character quotes.

        Split the value of the content field by using key-value pairs. The delimiter between key-value pairs is the tab character \t, the delimiter within a key-value pair is a colon (:), and the quote character is double quotes (""). An example configuration is as follows:

        • Raw log

          "content": "class:main http_user_agent:\"\"\"User Agent\"\"\" \"\"\"中文\"\"\""

        • Logtail plugin configuration: The configuration is the same as in Example 2, except that quote character is set to three double quotation marks (""").

        • Result

          "class": "main",
"http_user_agent": "User Agent",
"no_separator_key_0": "中文",

    JSON

    • Parameters

      Set the type parameter to processor_split_key_value. The following table describes the parameters in the detail object.

      Parameter

      Type

      Required

      Description

      SourceKey

      string

      Yes

      The name of the source field.

      Delimiter

      string

      No

      The separator between key-value pairs. The default is a tab character \t.

      Separator

      string

      No

      The separator between a key and a value in a key-value pair. The default value is a colon (:).

      KeepSource

      boolean

      No

      Specifies whether to retain the source field in the parsed log.

      • true: Retain the source field.

      • false (default): Do not retain the source field.

      ErrIfSourceKeyNotFound

      boolean

      No

      Specifies whether to report an error if the source field is not found in the raw log.

      • true (default): An error is reported.

      • false: No error is reported.

      DiscardWhenSeparatorNotFound

      boolean

      No

      Specifies whether to discard the key-value pair if the specified key-value separator is not found.

      • true: The key-value pair is discarded.

      • false (default): The key-value pair is not discarded.

      ErrIfSeparatorNotFound

      boolean

      No

      Specifies whether to report an error if the specified key-value separator is not found.

      • true (default): An error is reported.

      • false: No error is reported.

      ErrIfKeyIsEmpty

      boolean

      No

      Specifies whether to report an error if a key is empty after splitting.

      • true (default): An error is reported.

      • false: No error is reported.

      Quote

      string

      No

      Specifies the character used to enclose values. If a value is enclosed by this character, only the content within the quotes is extracted. This character can consist of multiple characters. By default, this parameter is empty and this feature is disabled.

      Important

      • To use a double quotation mark (") as the quote character, you must use a backslash (\) to escape it in your JSON configuration. For example: "Quote": "\"".

      • A backslash (\) preceding a quote character within a quoted value is treated as a literal character and included in the final value.

    • Examples

      • Example 1: Split key-value pairs.

        Split the value of the content field by using key-value pairs. The delimiter between key-value pairs is the tab character \t, and the delimiter within a key-value pair is a colon (:). An example configuration is as follows:

        • Raw log

          "content": "class:main\tuserid:123456\tmethod:get\tmessage:\"wrong user\""

        • Logtail plugin configuration

          {
  "processors":[
    {
      "type":"processor_split_key_value",
      "detail": {
        "SourceKey": "content",
        "Delimiter": "\t",
        "Separator": ":",
        "KeepSource": true
      }
    }
  ]
}

        • Result

          "content": "class:main\tuserid:123456\tmethod:get\tmessage:\"wrong user\""
"class": "main"
"userid": "123456"
"method": "get"
"message": "\"wrong user\""

      • Example 2: Split key-value pairs with quotes.

        Split the value of the content field by using key-value pairs. The delimiter between key-value pairs is the tab character \t, the delimiter within a key-value pair is a colon (:), and the quote character is a double quote ("). The following is a configuration example:

        • Raw log

          "content": "class:main http_user_agent:\"User Agent\" \"中文\" \"hello\\t\\\"ilogtail\\\"\\tworld\""

        • Logtail plugin configuration

          {
  "processors":[
    {
      "type":"processor_split_key_value",
      "detail": {
        "SourceKey": "content",
        "Delimiter": " ",
        "Separator": ":",
        "Quote": "\""
      }
    }
  ]
}

        • Result

          "class": "main",
"http_user_agent": "User Agent",
"no_separator_key_0": "中文",
"no_separator_key_1": "hello\t\"ilogtail\"\tworld",

      • Example 3: Split key-value pairs with multi-character quotes.

        Split the value of the content field into key-value pairs. The delimiter between key-value pairs is a tab character (\t), the delimiter within a key-value pair is a colon (:), and the quote character is a double quote ("). The following is a sample configuration:

        • Raw log

          "content": "class:main http_user_agent:\"\"\"User Agent\"\"\" \"\"\"中文\"\"\""

        • Logtail plugin configuration

          {
  "processors":[
    {
      "type":"processor_split_key_value",
      "detail": {
        "SourceKey": "content",
        "Delimiter": " ",
        "Separator": ":",
        "Quote": "\"\"\""
      }
    }
  ]
}

        • Result

          "class": "main",
"http_user_agent": "User Agent",
"no_separator_key_0": "中文",

    Grok mode

    Extracts target fields using grok expressions.

    Note

    The processor_grok plugin is supported in Logtail 1.2.0 and later.

    Form

    • Parameters

      Set Processor Type to Extract Field (Grok Mode). The following table describes the parameters.

      Parameter

      Description

      Original Field

      The name of the source field.

      Grok Expression Array

      An array of grok expressions. The processor_grok plugin tries each expression in order and returns the fields from the first successful match.

      For a list of default expressions, see processor_grok. If a required expression is unavailable, enter it in Custom Grok Pattern.

      Note

      Configuring multiple grok expressions may impact performance. We recommend using no more than five expressions.

      Custom Grok Pattern

      Enter a custom rule name and grok expression.

      Custom Grok Pattern File Directory

      The directory containing your custom grok pattern files. The processor_grok plugin reads all files in this directory.

      Important

      After you update a custom grok pattern file, you must restart Logtail for the changes to take effect.

      Maximum timeout

      The maximum time, in milliseconds, for a grok expression to match and extract fields. If you set this to 0 or leave it empty, the operation does not time out.

      Retain Logs that Fails to be Parsed

      Retains the log if parsing fails.

      Retain Original Field

      Retains the source field in the parsed log.

      Report Original Field Missing Error

      Reports an error if the source field is not found in the raw log.

      Report No Expressions Matched Error

      Reports an error if no expression in the Grok Expression Array matches the log.

      Report Match Timeout Error

      Reports an error if a match times out.

    • Configuration example

      This example shows how to use Grok mode to extract values from the content field and assign them to the year, month, and day fields.

      • Raw log

        "content" : "2022 October 17"

      • Logtail plugin configuration: Set Grok expression array to %{YEAR:year} %{MONTH:month} %{MONTHDAY:day} and enable the Retain logs that fail to be parsed option.

      • Result

        "year":"2022"
"month":"October"
"day":"17"

    JSON

    • Parameters

      Set type to processor_grok. The following table describes the parameters in detail.

      Parameter

      Type

      Required

      Description

      CustomPatternDir

      Array of strings

      No

      The directory containing your custom grok pattern files. The processor_grok plugin reads all files in this directory.

      If you do not specify this parameter, no custom grok pattern files are imported.

      Important

      After you update a custom grok pattern file, you must restart Logtail for the changes to take effect.

      CustomPatterns

      Map

      No

      The custom grok patterns, where the key is the rule name and the value is the grok expression.

      For the expressions that are supported by default, see processor_grok. If the link does not contain the expression that you need, enter a custom grok expression in Match.

      If you do not specify this parameter, custom grok patterns are not used.

      SourceKey

      String

      No

      The name of the source field. The default value is the content field.

      Match

      Array of strings

      Yes

      An array of grok expressions. The processor_grok plugin tries each expression in order and returns the fields from the first successful match.

      Note

      Configuring multiple grok expressions may impact performance. We recommend using no more than five expressions.

      TimeoutMilliSeconds

      Long

      No

      The maximum time, in milliseconds, for a grok expression to match and extract fields.

      If you do not specify this parameter or set it to 0, the operation does not time out.

      IgnoreParseFailure

      Boolean

      No

      Specifies whether to retain logs that fail to parse.

      • true (default): Retain the log.

      • false: Discard the log.

      KeepSource

      Boolean

      No

      Specifies whether to retain the source field after successful parsing.

      • true (default): Retain the source field.

      • false: Discard the source field.

      NoKeyError

      Boolean

      No

      Specifies whether to report an error if the source field is not found in the raw log.

      • true: An error is reported.

      • false (default): No error is reported.

      NoMatchError

      Boolean

      No

      Specifies whether to report an error if no expression in the Match parameter matches the log.

      • true (default): An error is reported.

      • false: No error is reported.

      TimeoutError

      Boolean

      No

      Specifies whether to report an error if a match times out.

      • true (default): An error is reported.

      • false: No error is reported.

    • Example 1

      This example shows how to use Grok mode to extract values from the content field and assign them to the year, month, and day fields.

      • Raw log

        "content" : "2022 October 17"

      • Logtail plugin configuration

        {
   "type" : "processor_grok",
   "detail" : {
      "KeepSource" : false,
      "Match" : [
         "%{YEAR:year} %{MONTH:month} %{MONTHDAY:day}"
      ],
      "IgnoreParseFailure" : false
   }
}

      • Result

        "year":"2022"
"month":"October"
"day":"17"

    • Example 2

      Use Grok mode to extract the values of the content field from multiple logs and parse them into different results based on different grok expressions. An example configuration is as follows:

      • Raw logs

        {
    "content" : "begin 123.456 end"
}
{
    "content" : "2019 June 24 \"I am iron man"\"
}
{
    "content" : "WRONG LOG"
}
{
    "content" : "10.0.0.0 GET /index.html 15824 0.043"
}

      • Logtail plugin configuration

        {
        "type" : "processor_grok",
        "detail" : {
                "CustomPatterns" : {
                        "HTTP" : "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}"
                },
                "IgnoreParseFailure" : false,
                "KeepSource" : false,
                "Match" : [
                        "%{HTTP}",
                        "%{WORD:word1} %{NUMBER:request_time} %{WORD:word2}",
                        "%{YEAR:year} %{MONTH:month} %{MONTHDAY:day} %{QUOTEDSTRING:motto}"
                ],
                "SourceKey" : "content"
        },
}

      • Result

        • For the first log, the processor_grok plugin fails to find a match by using the first expression %{HTTP} in the Match parameter, but succeeds by using the second expression %{WORD:word1} %{NUMBER:request_time} %{WORD:word2}. Therefore, the plugin returns the extracted results based on the second expression.

          Because the KeepSource parameter is set to false, the content field from the raw log is discarded.

        • For the second log entry, the processor_grok plugin fails to match the log by using the first expression %{HTTP} and the second expression %{WORD:word1} %{NUMBER:request_time} %{WORD:word2} in the Match parameter. The plugin successfully matches the log by using the third expression %{YEAR:year} %{MONTH:month} %{MONTHDAY:day} %{QUOTEDSTRING:motto}. Therefore, the plugin returns the extracted results based on the third expression.

        • The processor_grok plugin fails to match the third log with any expression in the Match array. Because the IgnoreParseFailure parameter is set to false, the third log is discarded.

        • For the fourth log, the processor_grok plugin successfully matches it with the first expression in the Match array, %{HTTP}, and returns the extracted fields.

        {
  "word1":"begin",
  "request_time":"123.456",
  "word2":"end",
}
{
  "year":"2019",
  "month":"June",
  "day":"24",
  "motto":"\"I am iron man"\",
}
{
  "client":"10.0.0.0",
  "method":"GET",
  "request":"/index.html",
  "bytes":"15824",
  "duration":"0.043",
}

    Add fields plugin

    Use the processor_add_fields plugin to add log fields. This topic describes the plugin's parameters and provides configuration examples.

    Configuration

    Important

    The processor_add_fields plugin is supported in Logtail 0.16.28 and later.

    Form

    • Parameters

      Set Processor Type to Add Field. The following table describes the parameters.

      Parameter

      Description

      Add Field

      The key-value pairs to add as new log fields.

      Ignore New Fields with Same Name

      If enabled, prevents a new field from overwriting an existing field with the same name.

    • Configuration example

      This example shows how to add the aaa2 and aaa3 fields.

      • Raw log

        "aaa1":"value1"

      • Logtail plugin configuration: Add two fields: aaa2 with the value value2, and aaa3 with the value value3.

      • Result

        "aaa1":"value1"
"aaa2":"value2"
"aaa3":"value3"

    JSON

    • Parameters

      Set type to processor_add_fields. The following table describes the parameters in the detail object.

      Parameter

      Type

      Required

      Description

      Fields

      Map

      Yes

      The log fields to add, specified as a map of key-value pairs.

      IgnoreIfExist

      Boolean

      No

      Specifies the behavior when a field to be added has the same name as an existing field.

      • true: The new field is ignored.

      • false (default): The existing field is overwritten by the new field.

    • Configuration example

      This example shows how to add the aaa2 and aaa3 fields.

      • Raw log

        "aaa1":"value1"

      • Logtail plugin configuration

        {
  "processors":[
    {
      "type":"processor_add_fields",
      "detail": {
        "Fields": {
          "aaa2": "value2",
          "aaa3": "value3"
        }
      }
    }
  ]
}

      • Result

        "aaa1":"value1"
"aaa2":"value2"
"aaa3":"value3"

    Drop fields plugin

    The processor_drop plugin drops specified log fields. This topic covers the plugin's parameters and provides configuration examples.

    Configuration

    Important

    The processor_drop plugin is supported in Logtail 0.16.28 and later.

    Form

    • Parameters

      Set Processor Type to Drop Field.

      Parameter

      Description

      Drop Field

      The log fields to drop. You can specify multiple fields.

    • Configuration example

      To drop the aaa1 and aaa2 fields from a log, configure the plugin as follows:

      • Raw log

        "aaa1":"value1"
"aaa2":"value2"
"aaa3":"value3"

      • Logtail plugin configuration: Drop the aaa1 and aaa2 fields.

      • Result

        "aaa3":"value3"

    JSON

    • Parameters

      Set type to processor_drop. The following table describes the detail parameter.

      Parameter

      Type

      Required

      Description

      DropKeys

      string array

      Yes

      The keys of the log fields to drop. You can specify multiple keys.

    • Configuration example

      To drop the aaa1 and aaa2 fields from a log, configure the plugin as follows:

      • Raw log

        "aaa1":"value1"
"aaa2":"value2"
"aaa3":"value3"

      • Logtail plugin configuration

        {
  "processors":[
    {
      "type":"processor_drop",
      "detail": {
        "DropKeys": ["aaa1","aaa2"]
      }
    }
  ]
}

      • Result

        "aaa3":"value3"

    Rename fields plugin

    You can use the processor_rename plugin to rename fields. This topic describes the parameters and configuration examples of the processor_rename plugin.

    Configuration

    Important

    The processor_rename plugin requires Logtail 0.16.28 or later.

    Console

    • Parameters

      Set Processor Type to Rename Fields and configure the parameters described in the following table.

      Parameter

      Description

      Original Field

      The original field to rename.

      New Field

      The new name for the field.

      Report Original Field Missing Error

      Specifies whether to report an error if the specified original field is not found in the log.

    • Example

      This example renames the aaa1 field to bbb1 and the aaa2 field to bbb2.

      • Raw log

        "aaa1":"value1"
"aaa2":"value2"
"aaa3":"value3"

      • Logtail plugin configuration: Rename aaa1 to bbb1 and aaa2 to bbb2.

      • Result

        "bbb1":"value1"
"bbb2":"value2"
"aaa3":"value3"

    JSON

    • Parameters

      Set type to processor_rename and configure the parameters for the detail object as described in the following table.

      Parameter

      Type

      Required

      Description

      NoKeyError

      Boolean

      No

      Specifies whether to report an error if the specified original field is not found in the log.

      • true: Reports an error.

      • false (default): Does not report an error.

      SourceKeys

      String array

      Yes

      The original fields to rename.

      DestKeys

      String array

      Yes

      The new names for the fields.

    • Example

      This example renames the aaa1 field to bbb1 and the aaa2 field to bbb2.

      • Raw log

        "aaa1":"value1"
"aaa2":"value2"
"aaa3":"value3"

      • Logtail plugin configuration

        {
  "processors":[
    {
      "type":"processor_rename",
      "detail": {
        "SourceKeys": ["aaa1","aaa2"],
        "DestKeys": ["bbb1","bbb2"],
        "NoKeyError": true
      }
    }
  ]
}

      • Result

        "bbb1":"value1"
"bbb2":"value2"
"aaa3":"value3"

    Pack fields plugin

    Use the processor_packjson plugin to pack one or more source fields into a destination field as a JSON object. This topic describes the parameters of the processor_packjson plugin and provides configuration examples.

    Configuration

    Important

    The processor_packjson plugin is supported in Logtail 0.16.28 and later.

    Console

    • Parameters

      Set Processor Type to Encapsulate Fields. The following table describes the parameters.

      Parameter

      Description

      Original Field

      The source fields to pack.

      New Field

      The destination field for the packed JSON object.

      Retain Original Field

      If you select this option, the source fields are retained in the parsed log.

      Report Original Field Missing Error

      If you select this option, the plugin reports an error if a specified source field is not found in the raw log.

    • Example

      This example shows how to pack the a and b fields into a destination field named d_key.

      • Raw log

        "a":"1"
"b":"2"

      • Logtail plugin configuration: Set Source Field to a and b, and set Destination Field to d_key.

      • Result

        "a":"1"
"b":"2"
"d_key":"{\"a\":\"1\",\"b\":\"2\"}"

    JSON

    • Parameters

      Set type to processor_packjson. The following table describes the parameters in the detail object.

      Parameter

      Type

      Required

      Description

      SourceKeys

      String array

      Yes

      The source fields to pack.

      DestKey

      String

      No

      The destination field for the packed JSON object.

      KeepSource

      Boolean

      No

      Specifies whether to retain the source fields in the parsed log.

      • true (default): The source fields are retained.

      • false: The source fields are removed after packing.

      AlarmIfIncomplete

      Boolean

      No

      Specifies whether to report an error if a specified source field is not found in the raw log.

      • true (default): An error is reported.

      • false: No error is reported.

    • Example

      This example shows how to pack the a and b fields into a destination field named d_key.

      • Raw log

        "a":"1"
"b":"2"

      • Logtail plugin configuration

        {
  "processors":[
    {
      "type":"processor_packjson",
      "detail": {
        "SourceKeys": ["a","b"],
        "DestKey":"d_key",
        "KeepSource":true,
        "AlarmIfIncomplete":true
      }
    }
  ]
}

      • Result

        "a":"1"
"b":"2"
"d_key":"{\"a\":\"1\",\"b\":\"2\"}"

    Expand JSON field plugin

    Use the processor_json plugin to expand a JSON field. This topic describes the parameters for the processor_json plugin and provides configuration examples.

    Configuration

    Important

    The processor_json plugin is supported in Logtail 0.16.28 and later.

    Console

    • Parameters

      Set Processor Type to Expand JSON Field. The following table describes the parameters.

      Parameter

      Description

      Original Field

      The name of the source field to expand.

      JSON Expansion Depth

      The maximum depth to expand the JSON object. The default value is 0, which means there is no depth limit. A value of 1 expands only the top-level fields.

      Character to Concatenate Expanded Keys

      The character used to connect nested keys. The default value is an underscore (_).

      Name Prefix of Expanded Keys

      A prefix for the names of expanded fields.

      Expand Array

      Specifies whether to expand arrays. This parameter is supported in Logtail 1.8.0 and later.

      Retain Original Field

      Specifies whether to keep the source field in the log after parsing.

      Report Original Field Missing Error

      Specifies whether to report an error if the specified source field is not found in the raw log.

      Use Name of Original Field as Name Prefix of Expanded Keys

      Specifies whether to use the name of the source field as a prefix for all expanded field names.

      Retain Raw Logs If Parsing Fails

      Specifies whether to keep the raw log if parsing fails.

    • Example

      This example expands the s_key field, using j as a custom prefix and the source field name s_key as an additional prefix for the expanded field names. The following is an example configuration:

      • Raw log (file path read by Logtail)

        {"s_key":"{\"k1\":{\"k2\":{\"k3\":{\"k4\":{\"k51\":\"51\",\"k52\":\"52\"},\"k41\":\"41\"}}}}"}

      • Logtail plugin configuration

        Set SourceKey to s_key, enable NoKeyError, set ExpandDepth to 0 (unlimited), set ExpandConnector to a hyphen (-), set Prefix to j, enable UseSourceKeyAsPrefix, and disable KeepSource.

      • Result

        The expanded log contains the following fields: j_s_key-k1-k2-k3-k4-k51 with the value 51, j_s_key-k1-k2-k3-k4-k52 with the value 52, and j_s_key-k1-k2-k3-k41 with the value 41. The source s_key field is removed.

    JSON

    • Parameters

      Set type to processor_json. The following table describes the parameters in the detail object.

      Parameter

      Type

      Required

      Description

      SourceKey

      String

      Yes

      The name of the source field to expand.

      NoKeyError

      Boolean

      No

      Specifies whether to report an error if the source field is not found in the raw log.

      • true (default): An error is reported.

      • false: No error is reported.

      ExpandDepth

      Int

      No

      The maximum depth to expand the JSON object. The default value is 0, which means there is no depth limit. A value of 1 expands only the top-level fields.

      ExpandConnector

      String

      No

      The character used to connect nested keys. The default value is an underscore (_).

      Prefix

      String

      No

      The prefix to add to the names of the new fields after expansion.

      KeepSource

      Boolean

      No

      Specifies whether to keep the source field in the log after parsing.

      • true (default): The source field is kept.

      • false: The source field is not kept.

      UseSourceKeyAsPrefix

      Boolean

      No

      Specifies whether to use the source field name as a prefix for all expanded field names.

      KeepSourceIfParseError

      Boolean

      No

      Specifies whether to keep the raw log if parsing fails.

      • true (default): The raw log is kept.

      • false: The raw log is not kept.

      ExpandArray

      Boolean

      No

      Specifies whether to expand arrays. This parameter is supported in Logtail 1.8.0 and later.

      • false (default): Arrays are not expanded.

      • true: Expand arrays. For example, {"k":["1","2"]} is expanded to {"k[0]":"1","k[1]":"2"}.

    • Example

      This example expands the s_key field, using j as a custom prefix and the source field name s_key as an additional prefix for the expanded field names. The following is an example configuration:

      • Raw log (file path read by Logtail)

        {"s_key":"{\"k1\":{\"k2\":{\"k3\":{\"k4\":{\"k51\":\"51\",\"k52\":\"52\"},\"k41\":\"41\"}}}}"}

      • Logtail plugin configuration

        {
  "processors":[
    {
      "type":"processor_json",
      "detail": {
        "SourceKey": "s_key",
        "NoKeyError":true,
        "ExpandDepth":0,
        "ExpandConnector":"-",
        "Prefix":"j",
        "KeepSource": false,
        "UseSourceKeyAsPrefix": true
      }
    }
  ]
}

      • Result

        The expanded log contains the following fields: j_s_key-k1-k2-k3-k4-k51 with the value 51, j_s_key-k1-k2-k3-k4-k52 with the value 52, and j_s_key-k1-k2-k3-k41 with the value 41. The source s_key field is removed.

    Field value mapping plugin

    The processor_dict_map plugin maps field values. This topic describes the plugin's parameters and provides configuration examples.

    Configuration

    Form

    Set processor type to Field Value Mapping. The following table describes the parameters.

    Parameter

    Description

    Original Field

    The name of the source field.

    New Field

    The name of the mapped field.

    Mapping Dictionary

    A dictionary that maps source values to target values.

    For small dictionaries, you can configure mappings directly in this field instead of using a local dictionary file.

    Important

    If you set Local Dictionary, the Mapping Dictionary configuration is ignored.

    Local Dictionary

    A CSV file that contains the mapping dictionary. In the file, fields must be enclosed in double quotation marks (") and separated by commas (,).

    advanced parameters>process missing source field

    When selected, if a raw log is missing the source field, the plugin fills the mapped field with the value from Value to Fill New Field.

    advanced parameters>Maximum Mapping Dictionary Size

    The maximum number of rules allowed in the mapping dictionary. The default is 1,000.

    Decrease this value to limit the plugin's server memory usage.

    advanced parameters>Method to Process Raw Log

    Specifies what to do when the mapped field already exists in a raw log.

    • overwrite

    • fill

    JSON

    Set type to processor_dict_map. The following table describes the parameters of detail.

    Parameter

    Type

    Required

    Description

    SourceKey

    String

    Yes

    The name of the source field.

    MapDict

    Map

    No

    A dictionary that maps source values to target values.

    For small dictionaries, you can configure mappings directly here instead of using the DictFilePath parameter.

    Important

    If you set the DictFilePath parameter, the MapDict parameter is ignored.

    DictFilePath

    String

    No

    The path to a CSV file that contains the mapping dictionary. In the file, fields must be enclosed in double quotation marks (") and separated by commas (,).

    DestKey

    String

    No

    The name of the mapped field.

    HandleMissing

    Boolean

    No

    Specifies whether to take action if the source field is missing from a raw log.

    • true: Take action if the source field is missing.

      The plugin populates the mapped field with the value of the Missing parameter.

    • false (default): No action is taken.

    Missing

    String

    No

    The value for the mapped field when the source field is missing. The default is Unknown.

    This parameter takes effect when HandleMissing is set to true.

    MaxDictSize

    Int

    No

    The maximum number of rules allowed in the mapping dictionary. The default is 1,000.

    Decrease this value to limit the plugin's server memory usage.

    Mode

    String

    No

    Specifies what to do when the mapped field (specified by DestKey) already exists in a raw log.

    • overwrite (default): Replaces the existing value in the mapped field.

    • fill: Keeps the existing value in the mapped field.

    String replacement

    Use the processor_string_replace plugin to perform full-text or regex-based replacement, or to remove escape characters from raw logs.

    Configuration

    Important

    The processor_string_replace plugin is supported in Logtail 1.6.0 and later.

    Form

    Set Processor Type to String Replacement. The following table describes the parameters.

    Parameter

    Description

    Original Field

    The name of the source field.

    Match Mode

    The method used to find content for replacement. Valid values:

    • String Match: Replaces content that matches a specific string.

    • Regular Expression Match: Replaces content that matches a regular expression.

    • Remove Escape Character: Removes escape characters.

    Matched Content

    The pattern or string to match.

    • If Match Mode is set to String Match, enter the target string.

      All matching occurrences are replaced.

    • If Match Mode is set to Regex Match, enter the target regular expression.

      You can also use a regular expression group to match a specific part of the string.

    • This parameter is not required if you set Match Mode to Remove Escape Character.

    Replaced By

    The string to use for replacement.

    • If you set Match Mode to String Match, enter the string to replace the original content.

    • If you set Match Mode to Regex Match, enter the string to replace the original content. You can reference capture groups in the replacement string.

    • This parameter is not required if you set Match Mode to Remove Escape Character.

    New Field

    The field to store the result. If not specified, the source field is overwritten.

    JSON

    Set type to processor_string_replace. The following table describes the parameters within the detail object.

    Parameter

    Type

    Required

    Description

    SourceKey

    String

    Yes

    The name of the source field.

    Method

    String

    Yes

    The replacement method. Valid values:

    • const: Uses string replacement.

    • regex: Uses a regular expression for replacement.

    • unquote: Removes escape characters.

    Match

    String

    No

    The pattern or string to match.

    • If you set Method to const, enter the string that matches the content you want to replace.

      All matching occurrences are replaced.

    • If you set Method to regex, enter the regular expression that matches the content you want to replace.

      You can also use a regular expression group to match a specific part of the string.

    • This parameter is not required if you set Method to unquote.

    ReplaceString

    String

    No

    The string to use for replacement. Defaults to an empty string ("").

    • If you set Method to const, enter the string to replace the original content.

    • If you set Method to regex, enter the string to replace the original content. You can reference capture groups in the replacement string.

    • This parameter is not required if you set Method to unquote.

    DestKey

    String

    No

    The field to store the result. If this parameter is not specified, the source field is overwritten.

    Configuration examples

    Replacing content using a string match

    This example shows how to use a string match to replace Error: in the content field with an empty string.

    Form

    • Raw log:

      "content": "2023-05-20 10:01:23 Error: Unable to connect to database."

    • Logtail plugin configuration: Set match mode to String Match, set source field to content, set match content to Error: , and leave replacement content empty.

    • Result:

      "content": "2023-05-20 10:01:23 Unable to connect to database."

    JSON

    • Raw log:

      "content": "2023-05-20 10:01:23 Error: Unable to connect to database."

    • Logtail plugin configuration:

      {
  "processors":[
    {
      "type":"processor_string_replace",
      "detail": {
        "SourceKey": "content",
        "Method": "const",
        "Match": "Error: ", 
        "ReplaceString": ""
      }
    }
  ]
}

    • Result:

      "content": "2023-05-20 10:01:23 Unable to connect to database."

    Replacing content using a regular expression

    This example shows how to use a regular expression to replace strings in the content field that match \\u\w+\[\d{1,3};*\d{1,3}m|N/A with an empty string.

    Form

    • Raw log:

      "content": "2022-09-16 09:03:31.013 \u001b[32mINFO \u001b[0;39m \u001b[34m[TID: N/A]\u001b[0;39m [\u001b[35mThread-30\u001b[0;39m] \u001b[36mc.s.govern.polygonsync.job.BlockTask\u001b[0;39m : Block collection------end------\r"

    • Logtail plugin configuration: Set match mode to Regex Match, set source field to content, set match content to \\u\w+\[\d{1,3};*\d{1,3}m|N/A, and leave replacement content empty.

    • Result:

      "content": "2022-09-16 09:03:31.013 INFO [TID: ] [Thread-30] c.s.govern.polygonsync.job.BlockTask : Block collection------end------\r"

    JSON

    • Raw log:

      "content": "2022-09-16 09:03:31.013 \u001b[32mINFO \u001b[0;39m \u001b[34m[TID: N/A]\u001b[0;39m [\u001b[35mThread-30\u001b[0;39m] \u001b[36mc.s.govern.polygonsync.job.BlockTask\u001b[0;39m : Block collection------end------\r"

    • Logtail plugin configuration:

      {
  "processors":[
    {
      "type":"processor_string_replace",
      "detail": {
        "SourceKey": "content",
        "Method": "regex",
        "Match": "\\\\u\\w+\\[\\d{1,3};*\\d{1,3}m|N/A", 
        "ReplaceString": ""
      }
    }
  ]
}

    • Result:

      "content": "2022-09-16 09:03:31.013 INFO [TID: ] [Thread-30] c.s.govern.polygonsync.job.BlockTask : Block collection------end------\r"

    Replacing content using capture groups

    This example uses a regular expression capture group to replace 16 in the content field with */24 and store the result in a new field named new_ip.

    Important

    When using capture groups for replacement, the replacement string cannot contain {}. You must use formats such as $1 and $2.

    Form

    • Raw log:

      "content": "10.10.239.16"

    • Logtail plugin configuration: Set match mode to Regex Match, set source field to content, set match content to (\d.*\.)\d+, set replacement content to $1*/24, and set destination field to new_ip.

    • Result:

      "content": "10.10.239.16",
"new_ip": "10.10.239.*/24"

    JSON

    • Raw log:

      "content": "10.10.239.16"

    • Logtail plugin configuration:

      {
  "processors":[
    {
      "type":"processor_string_replace",
      "detail": {
        "SourceKey": "content",
        "Method": "regex",
        "Match": "(\\d.*\\.)\\d+", 
        "ReplaceString": "$1*/24",
        "DestKey": "new_ip"
      }
    }
  ]
}

    • Result:

      "content": "10.10.239.16",
"new_ip": "10.10.239.*/24"

    Removing escape characters

    Form

    • Raw log:

      "content": "{\\x22UNAME\\x22:\\x22\\x22,\\x22GID\\x22:\\x22\\x22,\\x22PAID\\x22:\\x22\\x22,\\x22UUID\\x22:\\x22\\x22,\\x22STARTTIME\\x22:\\x22\\x22,\\x22ENDTIME\\x22:\\x22\\x22,\\x22UID\\x22:\\x222154212790\\x22,\\x22page_num\\x22:1,\\x22page_size\\x22:10}"

    • Logtail plugin configuration: Set match mode to Remove Escape Characters, and set source field to content.

    • Result:

      "content": "{\"UNAME\":\"\",\"GID\":\"\",\"PAID\":\"\",\"UUID\":\"\",\"STARTTIME\":\"\",\"ENDTIME\":\"\",\"UID\":\"2154212790\",\"page_num\":1,\"page_size\":10}"

    JSON

    • Raw log:

      "content": "{\\x22UNAME\\x22:\\x22\\x22,\\x22GID\\x22:\\x22\\x22,\\x22PAID\\x22:\\x22\\x22,\\x22UUID\\x22:\\x22\\x22,\\x22STARTTIME\\x22:\\x22\\x22,\\x22ENDTIME\\x22:\\x22\\x22,\\x22UID\\x22:\\x222154212790\\x22,\\x22page_num\\x22:1,\\x22page_size\\x22:10}"

    • Logtail plugin configuration:

      {
  "processors":[
    {
      "type":"processor_string_replace",
      "detail": {
        "SourceKey": "content",
        "Method": "unquote"
      }
    }
  ]
}

    • Result:

      "content": "{\"UNAME\":\"\",\"GID\":\"\",\"PAID\":\"\",\"UUID\":\"\",\"STARTTIME\":\"\",\"ENDTIME\":\"\",\"UID\":\"2154212790\",\"page_num\":1,\"page_size\":10}"

    Related documentation