Field processing functions - Data Transformation| Alibaba Cloud Documentation Center

Functions


Function	Description
v	Extracts the value of a field from an event. If multiple field names are passed in the function, the value of the first field that exists is returned.
e_set	Adds a field or specifies a new value for an existing field.
e_drop_fields	Deletes the log fields that meet a specified condition.
e_keep_fields	Retains the log fields that meet a specified condition.
e_pack_fields	Encapsulates specified log fields, and then assigns the log fields as a value to a new field.
e_rename	Renames the log fields that meet a specified condition.

v

You can use the v function to extract the value of a field from an event.

Syntax
```
v(Field name, ..., default=None)
```

Parameters


Parameter	Type	Required	Description
Field name	String	Yes	The name of the field whose value you want to extract from an event.
default	Arbitrary	No	The value of this parameter is returned if none of the specified fields exist. Default value: None.

Response
The value of the first field that exists in an event is returned. If none of the specified fields exist, the value of the default parameter is returned.
Example
Assign the value of the content field to the test_content field.
- Raw log entry:
```
content: hello
```
- Transformation rule:
```
e_set("test_content", v("content"))
```
- Result:
```
content: hello
test_content: hello
```

e_set

You can use the e_set function to add a field or to specify a new value for an existing field.

Syntax
```
e_set(key1, value1, key2, value2, mode="overwrite")
```
Notice
- The key1 and value1 parameters must be specified in pairs.
- If you use the e_set function to specify a value for a time field, such as F_TIME or __time__, the value must be a numeric string.
```
e_set(F_TIME, "abc")   # Invalid syntax.
e_set(F_TIME, "12345678")   # Valid syntax.
```

Parameters


Parameter	Type	Required	Description
key	String	Yes	The name of a log field. You can set this parameter to an expression that is used to return a string. For more information about how to specify special field names, see Event structure and fields.
value	Arbitrary	Yes	The new value of a specified field. If the value of this parameter is not a string, the function automatically converts the value to a string. For example, if you set this parameter to a value of the tuple, list, or dictionary type, the function automatically converts the value to a JSON string. For more information about the conversion rule of strings, see Automatic type conversion during assignment. Note If you set this parameter to None, the function does not update the original value of the specified field.
mode	String	No	The overwrite mode of fields. Default value: overwrite. For more information, see Field check and overwrite modes.

Response
The updated log entry is returned.
Examples
- Example 1: Assign a fixed value to a field.
  Add a new field named city and set the value to Shanghai.
```
e_set("city", "Shanghai")
```
- Example 2: Extract the value of an existing field, and then assign the value to another field.
  Call an expression function to extract the value of an existing field named ret, and then assign the value to a new field named result.
```
e_set("result", v("ret"))
```
- Example 3: Assign a dynamic value to a field.
  Call multiple expression functions in sequence to obtain the value in lowercase of the first field from specified existing fields and specify the value for the result field.
```
e_set("result", str_lower(v("ret", "return")))
```
- Example 4: Specify a value for a field multiple times.
  1. Specify a fixed value for the event_type field.
```
e_set("event_type", "login event", "event_info", "login host")
```
  2. If the value of the ret field is fail, set the event_type field to login failed event.
```
e_if(e_search('ret==fail'), e_set("event_type", "login failed event" ))
```

e_drop_fields

You can use the e_drop_fields function to delete the log fields that meet a specified condition.

Syntax

e_drop_fields (field 1, field 2, ....,regex=False)

Parameters


Parameter	Type	Required	Description
field	String	Yes	The name of a log field. The value of this parameter can be a regular expression. If the field name meets the specified condition, the field is deleted. Otherwise, the field is retained. For more information about regular expressions, see Regular expressions. You must specify at least one log field.
regex	Boolean	No	If you set this parameter to False, regular expressions are not used to match log fields. Default value: True.

Example
If the value of the content field is 123, the content and age fields are deleted.
- Raw log entry:
```
age: 18
content: 123
name: twiss
```
- Transformation rule:
```
e_if(e_search("content==123"), e_drop_fields("content", "age",regex=True))
```
- Result:
```
name: twiss
```

e_keep_fields

You can use the e_keep_fields function to retain the log fields that meet a specified condition.

Note Log Service provides built-in meta-fields such as __time__ and __topic__. If you do not retain the __time__ field when you call the e_keep_fields function, the time of the event is reset to the current time. If you do not want to reset the value of a meta-field, add the meta-field to a list in the format of F_TIME, F_META, F_TAGS, "f1", "f2". For more information, see Identifiers.

Syntax

e_keep_fields (field 1, field 2, ....,regex=False)

Parameters


Parameter	Type	Required	Description
field	String	Yes	The name of a log field. The value of this parameter can be a regular expression. If the field name meets the specified condition, the field is retained. Otherwise, the field is deleted. You must specify at least one log field.
regex	Boolean	No	If you set this parameter to False, regular expressions are not used to match log fields. Default value: True.

Example: If the value of the content field is 123, the content and age fields are retained.
- Raw log entry:
```
age: 18
content: 123
name: twiss
```
- Transformation rule:
```
e_if(e_search("content==123"), e_keep_fields("content", "age"))
```
- Result:
```
age: 18
content: 123
```

e_pack_fields

Syntax

e_pack_fields(output_fields,include=".*",exclude=None,drop_packed=True)

Parameters


Parameter	Type	Required	Description
output_field	String	Yes	The name of the output field. The value of the field is log data in the JSON format.
include	String	No	The whitelist. Fields that match the specified regular expression are encapsulated. Default value: ".*". This value indicates that all fields are encapsulated. For more information, see Regular expressions.
exclude	String	No	The blacklist. Fields that match the specified regular expression are not encapsulated. Default value: None. This value indicates that all fields are encapsulated. For more information, see Regular expressions.
drop_packed	Boolean	No	Specifies whether to delete raw fields after the fields are encapsulated. Default value: True. True: The raw fields that are encapsulated are deleted in the output. This is the default value. False: The raw fields that are encapsulated are not deleted in the output.

Response
The log entry in which specified fields are encapsulated is returned.
Examples
- Example 1: Encapsulate all log fields into a value, and then assign the value to the test field. By default, the raw fields that are encapsulated are deleted.
  - Raw log entry:
```
test1:123
test2:456
test3:789
```
  - Transformation rule:
```
e_pack_fields("test")
```
  - Result:
```
test:{"test1": "123", "test2": "456", "test3": "789"}
```
- Example 2: Encapsulate all log fields into a value, and then assign the value to the test field. The raw fields that are encapsulated are not deleted.
  - Raw log entry:
```
test1:123
test2:456
test3:789
```
  - Transformation rule:
```
e_pack_fields("test",drop_packed=False)
```
  - Result:
```
test:{"test1": "123", "test2": "456", "test3": "789"}
test1:123
test2:456
test3:789
```
- Example 3: Encapsulate the test and abcd fields into a value, and then assign the value to the content field. The raw fields that are encapsulated are not deleted.
  - Raw log entry:
```
abcd@#%:123
test:456
abcd:789
```
  - Transformation rule:
```
e_pack_fields("content", include="\w+", drop_packed=False)
```
  - Result:
```
abcd:789
abcd@#%:123
content:{"test": "456", "abcd": "789"}
test:456
```
- Example 4: Encapsulate log fields that exclude the test and abcd fields into a value and assign the value to the content field. The raw fields that are encapsulated are deleted.
  - Raw log entry:
```
abcd@#%:123
test:456
abcd:789
```
  - Transformation rule:
```
e_pack_fields("content", exclude="\w+", drop_packed=True)
```
  - Result:
```
abcd:789
content:{"abcd@#%": "123"}
test:456
```

e_rename

You can use the e_rename function to rename the log fields that meet a specified condition.

Syntax

e_rename("field 1", "renamed field 1", "field 2", "renamed field 2", ..., regex=False)

Note The field and renamed field parameters must be specified in pairs.

Parameters


Parameter	Type	Required	Description
field	String	Yes	The name of a log field. The value of this parameter can be a regular expression. If the field name meets the specified condition, the field is renamed. For more information about regular expressions, see Regular expressions. You must specify at least one log field.
renamed field	String	Yes	The new name of the field.
regex	Boolean	No	If you set this parameter to False, regular expressions are not used to match log fields. Default value: True.

Response
The field with the new name is returned.
Examples
- Example 1
  - Raw log entry:
```
host: 1006
```
  - Transformation rule:
```
e_rename("host","client_host")
```
  - Result:
```
client_host: 1006
```
- Example 2
  - Raw log entry:
```
host: 1006
```
  - Transformation rule:
```
e_rename("url","rename_url")
```
  - Result:
```
host: 1006
```