How to parse and update JSON data using data transformation - Simple Log Service

This topic describes how to use data transformation to parse and update JSON objects that are included in logs.

Use case 1: Expand and extract JSON objects

If a log contains JSON objects, use the e_json function to expand and extract an object.

Example 1: Expand the JSON object at layer 1

For example, expand the first-layer key-value pairs in the value of the data field.

Raw log

data: {"k1": "v1", "k2": {"k3": "v3", "k4": "v4"}}

Transformation rule
```
e_json("data", depth=1)
```

Result

data: {"k1": "v1", "k2": {"k3": "v3", "k4": "v4"}}
k1: v1
k2: {"k3": "v3", "k4": "v4"}

Example 2: Expand the JSON object at each layer

For example, fully expand the key-value pairs in the value of the data field.

Raw log

data: {"k1": "v1", "k2": {"k3": "v3", "k4": "v4"}}

Transformation rule
```
e_json("data")
```

Result

data:{"k1": "v1", "k2": {"k3": "v3", "k4": "v4"}}
k1:v1
k3:v3
k4:v4

Example 3: Extract a JSON object value by specifying a key

Specify a key of a JSON object to accurately extract the key-value pair.

Raw log

data: {
	"foo": {
    	"bar": "baz"
    },
    "peoples": [{
        "name": "xh",
        "sex": "girl"
    }, {
        "name": "xm",
        "sex": "boy"
    }]
}

Transformation rule

e_json("data", jmes="foo", output="foo")
e_json("data", jmes="foo.bar", output="bar")
e_json("data", jmes="peoples[0].name", output="name")
e_json("data", jmes="peoples[*].name", output="names")

Result

data:{"foo": {"bar": "baz"}, "peoples": [{"name": "xh", "sex": "girl"}, {"name": "xm", "sex": "boy"}]}
foo:{"bar": "baz"}
bar:baz
name:xh
names:["xh", "xm"]

Use case 2: Extract JSON object values

If a log contains JSON objects, use the dct_get function to extract a JSON object value.

Example 1: A JSON object contains the required field

For example, extract the value of the k1 key from a JSON object and assign it to a new key named key1.

Raw log
```
data: {"k1":"v1","k2":"v2"}
```
Transformation rule
```
e_set("key1", dct_get(v("data"), "k1"))
```
Result
```
data:{"k1": "v1", "k2": "v2"}
key1:v1
```

Example 2: A JSON object does not contain the required field

For example, extract a key-value pair from a JSON object. If the specified key does not exist, the key3 key is added with a default value.

Raw log
```
data: {"k1":"v1","k2":"v2"}
```

Transformation rule

e_set("key3", dct_get(v("data"), "k3", default="default"))

Result

data:{"k1": "v1", "k2": "v2"}
key3:default

Use case 3: Update JSON object values

If a log contains JSON objects, use the dct_update function to update a JSON object value.

Example 1: Change a JSON object value

For example, modify the value of the k1 key in a JSON object.

Raw log
```
data: {"k1":"v1","k2":"v2"}
```

Transformation rule

e_set("data", dct_update(v("data"), {"k1": "new_k1"}))

Result
```
data:{"k1": "new_k1", "k2": "v2"}
```

Example 2: Add a key-value pair to a JSON object

For example, add the "k3": "k3" key-value pair to a JSON object.

Raw log
```
data: {"k1":"v1","k2":"v2"}
```

Transformation rule

e_set("data", dct_update(v("data"), {"k3": "k3"}))

Result

data:{"k1": "v1", "k2": "v2", "k3": "k3"}

Use case 4: Delete a JSON object value

If a log contains JSON objects, use the dct_delete function to delete a JSON object value.

For example, delete the "k1":"v1" and "k2":"v2" key-value pairs from a JSON object.

Raw log
```
data: {"k1":"v1","k2":"v2", "k3": "v3"}
```

Transformation rule

e_set("data", dct_delete(v("data"), "k1", "k2"))

Result
```
data:{"k3": "v3"}
```

Use case 5: Parse a value into a JSON object

Use the json_parse function to parse a string into a JSON object.

For example, if the value of the data field is a string, convert it into a JSON object.

Raw log

data: "pre{ \"k1\": \"v1\", \"k2\": \"v2\"}"

Transformation rule

e_set("json_object", json_parse(op_slice(v("data"), 3, 28)))

Result

data:pre{ "k1": "v1", "k2": "v2"}
json_object:{"k1": "v1", "k2": "v2"}