All Products
Search
Document Center

Simple Log Service:Expand JSON fields

Last Updated:Jan 16, 2024

You can use the processor_json plug-in to expand JSON fields. This topic describes the parameters of the processor_json plug-in and provides an example on how to configure the plug-in.

Important
  • Form configuration: You can use form configuration to collect text logs and container stdout and stderr.

  • Editor configuration in JSON: You cannot use editor configuration in JSON to collect text logs.

Entry point

If you want to use a Logtail plug-in to process logs, you can add a Logtail plug-in configuration when you create or modify a Logtail configuration. For more information, see Overview.

Usage notes

Important

Only Logtail V0.16.28 and later support the processor_json plug-in.

Form configuration

  • Parameters

    Set Processor Type to Expand JSON Field. Then, configure other parameters based on the following table.

    Parameter

    Description

    Original Field

    The name of the original field to expand.

    JSON Expansion Depth

    The depth of JSON expansion. Default value: 0. This value indicates that the depth of JSON expansion is unlimited. If the value is n, the depth of JSON expansion is n.

    Character to Concatenate Expanded Keys

    The character that is used to connect expanded keys. The default value is an underscore (_).

    Name Prefix of Expanded Keys

    The prefix that is added to the names of expanded keys.

    Expand Array

    Specifies whether to expand the array. Only Logtail 1.8.0 and later support this parameter.

    Retain Original Field

    Specifies whether to retain the original field in the new log that is obtained after parsing.

    Report Original Field Missing Error

    Specifies whether to report an error if the raw log does not contain the original field.

    Use Name of Original Field as Name Prefix of Expanded Keys

    Specifies whether to add the name of the original field as a prefix to all expanded keys.

    Retain Raw Logs If Parsing Fails

    Specifies whether to retain the raw log if the raw log fails to be parsed.

  • Configuration example

    The following example shows how to expand the JSON field s_key and add j and the name of the original field s_key as a prefix to the expanded keys:

    • Raw log

      "s_key":"{\"k1\":{\"k2\":{\"k3\":{\"k4\":{\"k51\":\"51\",\"k52\":\"52\"},\"k41\":\"41\"}}})"
    • Logtail plug-in configuration for data processingimage.png

    • Result

      "js_key-k1-k2-k3-k4-k51":"51"
      "js_key-k1-k2-k3-k4-k52":"52"
      "js_key-k1-k2-k3-k41":"41"

Editor configuration in JSON

  • Parameters

    Set type to processor_json. Then, configure other parameters in detail based on the following table.

    Parameter

    Type

    Required

    Description

    SourceKey

    String

    Yes

    The name of the original field to expand.

    NoKeyError

    Boolean

    No

    Specifies whether to report an error if the raw log does not contain the original field. Valid values:

    • true (default): reports an error.

    • false: does not report an error.

    ExpandDepth

    Int

    No

    The depth of JSON expansion. Default value: 0. This value indicates that the depth of JSON expansion is unlimited. If the value is n, the depth of JSON expansion is n.

    ExpandConnector

    String

    No

    The character that is used to connect expanded keys. The default value is an underscore (_).

    Prefix

    String

    No

    The prefix that is added to the names of expanded keys.

    KeepSource

    Boolean

    No

    Specifies whether to retain the original field in the new log that is obtained after parsing. Valid values:

    • true (default): retains the original field.

    • false: does not retain the original field.

    UseSourceKeyAsPrefix

    Boolean

    No

    Specifies whether to add the name of the original field as a prefix to all expanded keys.

    KeepSourceIfParseError

    Boolean

    No

    Specifies whether to retain the raw log if the raw log fails to be parsed. Valid values:

    • true (default): retains the raw log.

    • false: does not retain the raw log.

    ExpandArray

    Boolean

    No

    Specifies whether to expand the array. Only Logtail 1.8.0 and later support this parameter.

    • false (default): does not expand the array.

    • true: expands the array. For example, {"k":["1","2"]} is expanded to {"k[0]":"1","k[1]":"2"}.

  • Configuration example

    The following example shows how to expand the JSON field s_key and add j and the name of the original field s_key as a prefix to the expanded keys:

    • Raw log

      "s_key":"{\"k1\":{\"k2\":{\"k3\":{\"k4\":{\"k51\":\"51\",\"k52\":\"52\"},\"k41\":\"41\"}}})"
    • Logtail plug-in configuration for data processing

      {
        "processors":[
          {
            "type":"processor_json",
            "detail": {
              "SourceKey": "s_key",
              "NoKeyError":true,
              "ExpandDepth":0,
              "ExpandConnector":"-",
              "Prefix":"j",
              "KeepSource": false,
              "UseSourceKeyAsPrefix": true
            }
          }
        ]
      }
    • Result

      "js_key-k1-k2-k3-k4-k51":"51"
      "js_key-k1-k2-k3-k4-k52":"52"
      "js_key-k1-k2-k3-k41":"41"