Field extraction modes

Related functions

The following table lists the functions that use the mode parameter and their default values.

Category	Function	Default value of mode
Value assignment functions	e_set	overwrite
Value extraction functions	e_regex	fill-auto
	e_json	fill-auto
	e_kv	fill-auto
	e_csv, e_psv, and e_tsv	fill-auto
	e_kv_delimit	fill-auto
	e_anchor	overwrite
	e_syslogrfc	overwrite
Mapping and enrichment functions	e_dict_map	fill-auto
Mapping and enrichment functions	e_table_map	fill-auto

Field extraction check and overwrite modes

The following table describes the values of the mode parameter.

Value	Description
fill	Sets a new field if the field does not exist or if the field value is an empty string.
fill-auto	Sets a new field if the new value is not an empty string and one of the following conditions is met: The field does not exist. The field value is an empty string.
add	Sets a new field if the field does not exist.
add-auto	Sets a new field if the new value is not an empty string and the field does not exist.
overwrite	Always sets a new field.
overwrite-auto	Sets a new field if the new value is not an empty string.

The following table shows how functions behave in different modes.

Raw log
```
a:         # An empty string
b: 100
```

Transformation examples

Mode	Transformation rule	Result
add	`e_set("c", "123", mode='add')`	`a:# An empty string b: 100 c: 123`
	`e_set("c", "", mode='add')`	`a:# An empty string b: 100 c:`
	`e_set("a", "123", mode='add')`	`a:# An empty string b: 100`
add-auto	`e_set("c", "", mode='add-auto')`	The `c` field is not added, and the raw log remains unchanged.
fill	`e_set("c", "123", mode='fill')`	`a:# An empty string b: 100 c: 123`
	`e_set("c", "", mode='fill')`	`a:# An empty string b: 100 c:`
	`e_set("a", "123", mode='fill')`	`a: 123 b: 100`
	`e_set("b", "123", mode='fill')`	The b field remains `b: 100`.
fill-auto	`e_set("c", "", mode='fill-auto')`	The `c` field is not added, and the raw log remains unchanged.
overwrite	`e_set("c", "123", mode='overwrite')`	`a:# An empty string b: 100 c: 123`
	`e_set("c", "", mode='overwrite')`	`a:# An empty string b: 100 c:`
	`e_set("b", "200", mode='overwrite')`	`a:# An empty string b: 200`
	`e_set("b", "", mode='overwrite')`	`a:# An empty string b:`
overwrite-auto	`e_set("b", "", mode='overwrite-auto')`	The b field remains `b: 100`.

Limits on field names for extraction

This section applies to functions such as e_json, e_kv, e_kv_delimit, and e_regex.

Only fields whose names match the naming rules can be extracted. Fields with non-compliant names are discarded. The regular expression u'_*[\u4e00-\u9fa5\u0800-\u4e00a-zA-Z][\u4e00-\u9fa5\u0800-\u4e00\\w\\.\\-]*' is not supported. For example, fields named 123=abc, __1__:100, 1k=200, or {"123": "456"} are discarded.

The following example shows how the default field name limits work:

Raw log

data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } }

Transformation rule

e_json(
    "data",
    fmt="parent",
    sep="@",
    prefix="__",
    suffix="__",
    include_node=r"[\u4e00-\u9fa5\u0800-\u4e00a-zA-Z][\w\-\.]*",
    mode="fill-auto",
)

Result

data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } }
data@__k1__:100
k2@__k3__:200
k4@__k5__:300