This topic describes the syntax and parameters of mapping and enrichment functions. This topic also provides examples on how to use the functions.
Functions
Type | Function | Description |
---|---|---|
Field-based mapping | e_dict_map | Maps the value of a specified field to a field based on a dictionary, and then returns the value of the matched field. |
e_table_map | Maps the value of a specified field to a row in a table, and then returns the value of the field in this row. | |
Search-based mapping | e_search_dict_map | Maps a search string to a key based on a dictionary, and then returns the value of the matched key. |
e_search_table_map | Maps a search string to a column in a table, and then returns the value of the field in this column. |
e_dict_map
- Syntax
e_dict_map(data, field, output_field, case_insensitive=True, missing=None, mode="overwrite")
- Parameters
Parameter Type Required Description data Dict Yes The dictionary that is used for mapping. A dictionary includes a collection of key-value pairs. Each key must be a string. field String or string list Yes One or more field names. If the value of this parameter contains multiple field names, one of the following results is returned: - The system maps the field names in sequence.
- If multiple log entries match the specified search condition and the mode parameter is set to overwrite, the system returns the last matched log entry.
- If no field matches the specified search condition, the system returns the value of the missing parameter.
output_field String Yes The field name that you want to return. case_insensitive Bool No Specifies whether data is case-sensitive when the system maps the data. Default value: True. This value indicates that data is not case-sensitive. Note If multiple fields in the dictionary match the specified search condition and the case_insensitive parameter is set to True, the system selects the field that uses the same case as the key. If no key is found, the system randomly selects a field.missing String No The value that is returned if no fields are matched. Default value: None. This value indicates that no operations are performed. Note If the specified dictionary contains a mapping rule that returns the asterisk (*) wildcard and no fields are matched, themissing
parameter is disabled. This is because the asterisk (*) wildcard has a higher priority than themissing
parameter.mode String No The overwrite mode of the fields. Default value: overwrite. For more information, see Field check and overwrite modes. - Response
An event in which new fields and values are added is returned.
- Examples
- Example 1
Raw log entry:
data: 123 pro: 1
Transformation rule:e_dict_map({"1": "TCP", "2": "UDP", "3": "HTTP", "*": "Unknown"}, "pro", "protocol")
Result:data: 123 pro: 1 protocol: TCP
- Example 2
Raw log entries (three log entries):
status: 500
status: 400
status: 200
Transformation rule:e_dict_map({"400": "Error", "200": "Success", "*": "Other"}, "status", "message")
Result:status: 500 message: Other
status: 400 message: Error
status: 200 message: Success
- Example 1
e_table_map
- Syntax
e_table_map(data, field, output_fields, missing=None, mode="fill-auto")
- Parameters
Parameter Type Required Description data Table Yes The table that is used to map values in multiple columns. Note If you use a resource function, such as res_rds_mysql or res_log_logstore_pull, to pull data, we recommend that you set the primary_keys parameter to improve the performance of data transformation. For more information, see Resource functions.field String, string list, or tuple list Yes The source fields that are mapped to the specified table in an event. If no field is matched in the event, no operations are performed on the source fields. output_fields String, string list, or tuple list Yes The fields that are matched in the specified table. missing String No The value that is returned if no fields are matched. Default value: None. This value indicates that no operations are performed. If the source field is mapped to multiple columns, the value of the missing
parameter can be a list of default values. The length of the list is the same as the length of the source field.Note If the specified table contains a mapping rule that returns the asterisk (*) wildcard and no fields are matched, themissing
parameter is disabled. This occurs because the asterisk (*) wildcard has a higher priority than themissing
parameter.mode String No The overwrite mode of the fields. Default value: fill-auto. For more information, see Field check and overwrite modes. - Response
An event in which new fields and values are added is returned.
- Examples
- Example 1: One field is returned.
Raw log entry:
data: 123 city: nj
Transformation rule:e_table_map(tab_parse_csv("city,pop,province\nnj,800,js\nsh,2000,sh"), "city", "province")
Result:data: 123 city: nj province: js
- Example 2: Two fields are returned.
Raw log entry:
data: 123 city: nj
Transformation rule:e_table_map(tab_parse_csv("city,pop,province\nnj,800,js\nsh,2000,sh"), "city", ["province", "pop"])
Result:data: 123 city: nj province: js pop: 800
- Example 3: The sep parameter in the tab_parse_csv function is used.
Raw log entry:
data: 123 city: nj
Transformation rule:e_table_map(tab_parse_csv("city#pop#province\nnj#800#js\nsh#2000#sh", sep='#'), "city", ["province", "pop"])
Result:data: 123 city: nj province: js pop: 800
- Example 4: The quote parameter in the tab_parse_csv function is used.
Raw log entry:
data: 123 city: nj
Transformation rule:e_table_map(tab_parse_csv('city,pop,province\n|nj|,|800|,|js|\n|shang hai|,2000,|SHANG,HAI|', quote='|'), "city", ["province", "pop"])
Result:data: 123 city: nj province: js pop: 800
- Example 5: The value of the specified source field is different from the values of
the fields in the specified table.
Raw log entry:
data: 123 cty: nj
Transformation rule:e_table_map(tab_parse_csv("city,pop,province\nnj,800,js\nsh,2000,sh"), [("cty","city")], "province")
Result:data: 123 cty: nj province: js
- Example 6: The value of the specified source field is different from the values of
the fields in the specified table, and the specified output field is renamed.
Raw log entry:
data: 123 cty: nj
Transformation rule:e_table_map(tab_parse_csv("city,pop,province\nnj,800,js\nsh,2000,sh"), [("cty","city")], [("province","pro")])
Result:data: 123 cty: nj pro: js
- Example 7: Multiple source fields are specified.
Raw log entry:
data: 123 city: nj pop: 800
Transformation rule:e_table_map(tab_parse_csv("city,pop,province\nnj,800,js\nsh,2000,sh"), ["city", "pop"], "province")
Result:data: 123 city: nj pop: 800 province: js
- Example 8: Multiple source fields are specified, and the values of the specified source
fields are different from the values of the fields in the specified table.
Raw log entry:
data: 123 cty: nj pp: 800
Transformation rule:e_table_map(tab_parse_csv("city,pop,province\nnj,800,js\nsh,2000,sh"), [("cty", "city"), ("pp", "pop")], "province")
Result:data: 123 cty: nj pop: 800 province: js
- Example 1: One field is returned.
e_search_dict_map
- Syntax
e_search_dict_map(data, output_field, multi_match=False, multi_join=" ", missing=None, mode="overwrite")
- Parameters
Parameter Type Required Description data Dict Yes The dictionary that is used for mapping. A dictionary includes a collection of key-value pairs. Each key must be a search string. For more information about search strings, see the dct_get function. output_field String Yes The field name that you want to return. multi_match Bool No Specifies whether multiple matched fields can be returned. Default value: False. This value indicates that the system returns only the value of the first matched field. If you set the parameter to True, the system concatenates the values of multiple matched fields by using the character that is specified in the multi_join
parameter.multi_join String No The character that is used to concatenate the values of multiple matched fields. The default value is a space character. This parameter is valid only when the multi_match parameter is set to True. missing String No The value that is returned if no fields are matched. Default value: None. This value indicates that no operations are performed. Note If the specified dictionary contains a mapping rule that returns the asterisk (*) wildcard and no fields are matched, themissing
parameter is disabled. This occurs because the asterisk (*) wildcard has a higher priority than themissing
parameter.mode String No The overwrite mode of the fields. Default value: overwrite. For more information, see Field check and overwrite modes. - Response
The matched values are returned.
- Examples
- Example 1: Data mapping in matching mode
Raw log entry:
data:123 pro:1
Transformation rule:e_search_dict_map ({"pro==1": "TCP", "pro==2": "UDP", "pro==3": "HTTP"}, "protocol")
Result:data:123 pro:1 protocol:TCP
- Example 2: Data mapping based on the first character in the value of each field
Raw log entry:
status:200,300
Transformation rule:e_search_dict_map ({"status:2??": "ok", "status:3??": "redirect", "status:4??": "auth", "status:5??": "server_error"}, "status_desc", multi_match=True, multi_join="test")
Result:status:200,300 status_desc: ok test redirect
- Example 1: Data mapping in matching mode
e_search_table_map
- Syntax
e_search_table_map(data, inpt, output_fields, multi_match=False, multi_join=" ", missing=None, mode="fill-auto")
- Parameters
Parameter Type Required Description data Table Yes The table from which data is obtained. The name of a column in the table must be a search string. inpt String Yes The field names that match the search string in the specified table. output_fields String, string list, or tuple list Yes The fields that are matched in the specified table. The value of this parameter is a string, a string list, or a tuple list. multi_match Bool No Specifies whether multiple matched fields can be returned. Default value: False. This value indicates that the system returns only the value of the first matched field. If you set the parameter to True, the system concatenates the values of multiple matched fields by using the character that is specified in the multi_join
parameter.multi_join String No The character that is used to concatenate the values of multiple matched fields. The default value is a space character. This parameter is valid only when the multi_match
parameter is set to True.missing String No The value that is returned if no fields are matched. Default value: None. This value indicates that no operations are performed. Note If the specified table contains a mapping rule that returns the asterisk (*
) wildcard and no fields are matched, themissing
parameter is disabled. This is because the asterisk (*
) wildcard has a higher priority than themissing
parameter.mode String No The overwrite mode of the fields. Default value: fill-auto. - Source table
Column name Type 1 Type 2 content: guide and title:~"\w guide"'
guide user content: city and title:~"\w shanghai"
food home - Response
The matched values are returned.
- Examples
- Example 1: Data mapping in simple mode
Raw log entry:
data: 123 city: sh
Transformation rule:e_search_table_map(tab_parse_csv("search,pop,province\ncity==nj,800,js\ncity==sh,2000,sh"), "search", ["pop", "province"])
Result:data: 123 city: sh province: sh pop: 2000
- Example 2: Data mapping in overwrite mode
Raw log entry:
data: 123 city: nj province:
Transformation rule:e_search_table_map(tab_parse_csv("search,pop,province\ncity==nj,800,js\ncity==sh,2000,sh"), "search", "province",mode="overwrite")
Result:data: 123 city: nj province: js
- Example 3: Data mapping in missing mode when the table does not contain the province
field
Raw log entry:
data: 123 city: wh province:
Transformation rule:e_search_table_map(tab_parse_csv("search,pop,province\ncity==nj,800,\ncity==sh,2000,sh"), "search", "province",missing="Unknown")
Result:data: 123 city: wh province: Unknown
- Example 4: Data mapping in multi_match mode
Raw log entry:
data: 123 city: nj,sh province:
Transformation rule:e_search_table_map(tab_parse_csv("search,pop,province\ncity:nj,800,js\ncity:sh,2000,sh"), "search", "province",multi_match=True, multi_join=",")
Result:data: 123 city: nj,sh province: js,sh
- Example 1: Data mapping in simple mode