Field extraction modes - Simple Log Service - Alibaba Cloud Documentation Center

This topic describes the values of the mode parameter in different functions. The mode parameter specifies a field extraction mode.

Related functions

The following table describes the functions that use the mode parameter and the default value that is used for the mode parameter in each function.

Category	Function	Default value of mode
Value assignment functions	e_set	overwrite
Value extraction functions	e_regex	fill-auto
	e_json	fill-auto
	e_kv	fill-auto
	e_csv, e_psv, and e_tsv	fill-auto
	e_kv_delimit	fill-auto
	e_anchor	overwrite
	e_syslogrfc	overwrite
Mapping and enrichment functions	e_dict_map	fill-auto
	e_table_map	fill-auto
	e_search_dict_map	overwrite
	e_search_table_map	fill-auto

Field extraction check and overwrite modes

The following table describes the values of the mode parameter.

Value	Description
fill	Sets a new field if the field does not exist or if the field already exists but the value of the field is an empty string.
fill-auto	Sets a new field if the new value is not an empty string and one of the following conditions is met: The field does not exist. The field already exists but the value of the field is an empty string.
add	Sets a new field if the field does not exist.
add-auto	Sets a new field if the new value is not an empty string and the field does not exist.
overwrite	Always sets a new field.
overwrite-auto	Sets a new field if the new value is not an empty string.

The following table provides examples on how the functions work in different modes.

Raw log
```
a:         # An empty string
b: 100
```

Transformation examples

Mode	Transformation rule	Result
add	`e_set("c", "123", mode='add')`	`a:# An empty string b: 100 c: 123`
	`e_set("c", "", mode='add')`	`a:# An empty string b: 100 c:`
	`e_set("a", "123", mode='add')`	`a:# An empty string b: 100`
add-auto	`e_set("c", "", mode='add-auto')`	The `c` field is not added, and the raw log remains unchanged.
fill	`e_set("c", "123", mode='fill')`	`a:# An empty string b: 100 c: 123`
	`e_set("c", "", mode='fill')`	`a:# An empty string b: 100 c:`
	`e_set("a", "123", mode='fill')`	`a: 123 b: 100`
	`e_set("b", "123", mode='fill')`	The b field remains `b: 100`.
fill-auto	`e_set("c", "", mode='fill-auto')`	The `c` field is not added, and the raw log remains unchanged.
overwrite	`e_set("c", "123", mode='overwrite')`	`a:# An empty string b: 100 c: 123`
	`e_set("c", "", mode='overwrite')`	`a:# An empty string b: 100 c:`
	`e_set("b", "200", mode='overwrite')`	`a:# An empty string b: 200`
	`e_set("b", "", mode='overwrite')`	`a:# An empty string b:`
overwrite-auto	`e_set("b", "", mode='overwrite-auto')`	The b field remains `b: 100`.

Limits on field names for extraction

Functions such as e_json, e_kv, e_kv_delimit, and e_regex are supported.

Only the fields whose names abide by the limits can be extracted. The fields whose names do not abide by the limits are discarded. The regular expression u'_*[\u4e00-\u9fa5\u0800-\u4e00a-zA-Z][\u4e00-\u9fa5\u0800-\u4e00\\w\\.\\-]*' is not supported. For example, the fields whose names match 123=abc, __1__:100, 1k=200, or {"123": "456"} are discarded.

The following example shows how to use the default limits of a function on field names:

Raw log

data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } }

Transformation rule

e_json(
    "data",
    fmt="parent",
    sep="@",
    prefix="__",
    suffix="__",
    include_node=r"[\u4e00-\u9fa5\u0800-\u4e00a-zA-Z][\w\-\.]*",
    mode="fill-auto",
)

Result

data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } }
data@__k1__:100
k2@__k3__:200
k4@__k5__:300