This topic describes the syntax of value extraction functions and provides parameter descriptions and function examples.
Functions
Type | Function | Description |
---|---|---|
Regular expression-based extraction | e_regex | Extracts the values of fields in an event by using a regular expression and assigns the values to other fields. |
JSON-based extraction | e_json | Performs operations on JSON objects in specified event fields. The operations include expanding JSON data, extracting JSON data with JMESPath expressions, and extracting JSON data with JMESPath expressions and then expanding JSON data. |
Delimiter-based extraction | e_csv, e_psv, and e_tsv | Extracts the values of multiple predefined fields based on user-defined delimiters.
|
Key-value-based extraction | e_kv | Extracts key-value pairs of multiple source fields by using the quote parameter. |
e_kv_delimit | Extracts the key-value pairs of multiple source fields by using delimiters. | |
Syslog-based extraction | e_syslogrfc | Calculates the values of the facility and severity fields based on the known value of the priority field. Then, obtains the corresponding level information in the Request for Comments (RFC) protocol. |
Anchor-rules-based extraction | e_anchor | Extracts the values between the specified start and end positions. |
e_regex
- Syntax
e_regex(Source field name, Regular expression with named capturing groups) e_regex(Source field name, Regular expression with non-capturing groups, fields_info(Destination field name)) e_regex(Source field name, Regular expression with non-capturing groups, fields_info(An array of destination field names)) e_regex(Source field name, Regular expression with capturing groups, fields_info(An array of destination field names)) e_regex(Source field name, Regular expression with capturing groups, fields_info(Destination dictionary)) e_regex(... Preceding parameters...mode="fill-auto")
- Parameters
Parameter Type Required Description Source field name Arbitrary Yes The name of a source field. The field name can contain any characters. If the field does not exist in the specified event, no actions are performed. For more information about how to set special field names, see Event structure and fields. Regular expression String Yes The regular expression that is used to extract the value of a field. Note Regular expressions with non-capturing groups are used in some scenarios. A non-capturing group uses the prefix that consists of a question mark and a colon (?:
). For example,\w+@\w+\.\w(?:\.\cn)?
. For more information about non-capturing groups, see Non-capturing group.fields_info String/ List/ Dict No The name of the destination field. You must specify this parameter if you do not use a regular expression with named capturing groups. mode String No The overwrite mode for a field. Default value: fill-auto. For information about other values of this parameter, see Field check and overwrite modes. - Response
An event where new fields and values are added is returned.
- Example 1: Extract the value of a single field.
- Raw log entry:
msg: 192.168.1.1 http://... 127.0.0.0
- Transformation rule:
e_regex("msg",r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}","ip")
- Result:
msg: 192.168.1.1 http://... 127.0.0.0 ip: 192.168.1.1
- Raw log entry:
- Example 2: Extract the values of multiple fields.
- Raw log entry:
msg: 192.168.1.1 http://... 127.0.0.0
- Transformation rule:
e_regex("msg",r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}",["server_ip","client_ip"])
- Result:
msg: 192.168.1.1 http://... 127.0.0.0 server_ip: 192.168.1.1 client_ip: 127.0.0.0
- Raw log entry:
- Example 3: Extract the values of multiple fields by using capturing groups.
- Raw log entry:
content: start sys version: deficience, err: 2
- Transformation rule:
e_regex("content",r"start sys version: (\w+),\s*err: (\d+)",["version","error"])
- Result:
content: start sys version: deficience, err: 2 error: 2 version: deficience
- Raw log entry:
- Example 4: Extract the values of multiple fields by using named capturing groups.
- Raw log entry:
content: start sys version: deficience, err: 2
- Transformation rule:
e_regex("content",r"start sys version: (? P<version>\w+),\s*err: (? P<error>\d+)")
- Result:
content: start sys version: deficience, err: 2 error: 2 version: deficience
- Raw log entry:
- Example 5: Dynamically change the fields and extract the values of fields.
- Raw log entry:
dict: varify:123
- Transformation rule:
e_regex("dict",r"(\w+):(\d+)",{r"k_\1": r"v_\2"})
- Result:
dict: varify:123 k_varify: v_123
- Raw log entry:
e_json
- Syntax
e_json(Source field name, expand=None, depth=100, prefix="__", suffix="__", fmt="simple", sep=".", expand_array=True, fmt_array="{parent}_{index}", include_node=r"[\u4e00-\u9fa5\u0800-\u4e00a-zA-Z][\w\-\.]*", exclude_node="", include_path="", exclude_path="", jmes="", output="", jmes_ignore_none=False, mode='fill-auto' )
- Parameters
Parameter Type Required Description Source field name String Yes The name of a source field. The field name can contain any characters. If the field does not exist in the specified event, no actions are performed. For more information about how to set special field names, see Event structure and fields. expand Bool No Specifies whether to expand the source field. - If the jmes field is not specified, the value of this parameter is set to True by default. This indicates that the source field is expanded.
- If the jmes field is specified, the value of this parameter is set to False by default. This indicates that the field is not expanded.
depth Number No The depth of field expanding. Value range: 1 to 2000. Default value: 100. The value 1 indicates that only the first tier of the field is expanded. prefix String No The prefix added to an expanded field. suffix String No The suffix added to an expanded field. fmt String No The formatting method of a field. Valid values: - simple: The current node name is used as the field name. This is the default value. The
format is
{prefix}{current}{suffix}
. - full: The names of the current node and its closest parent node are combined and used
as the field name. The format is
{parent_list_str}{sep}{prefix}{current}{suffix}
.sep
specifies the delimiter. Default value:.
. - parent: The names of the current node and all its parent nodes are used as the field name.
The format is
{parent}{sep}{prefix}{current}{suffix}
.sep
specifies the delimiter. Default value:.
. - root: specifies to combine the names of the current node and its root node as the field
name. The format is
{parent_list[0]}{sep}{prefix}{current}{suffix}
.sep
specifies the delimiter. Default value:.
.
sep String No The delimiter used to separate parent and child nodes during formatting. You must specify this parameter if the value of the fmt
parameter is set to full, parent, or root. Default value:.
.expand_array Bool No Specifies whether to expand an array of fields. To expand an array, the following format is used: {parent}_{index}
.fmt_array String No The formatting method to expand an array of fields. The format is {parent_rlist[0]}_{index}
. Alternatively, you can use up to five placeholders to expand an array:parent_list
,current
,sep
,prefix
, andsuffix
.include_node String/ Number No The whitelist of node names that are expanded. By default, node names that contain digits, letters, and specific special characters ( _. -
) are expanded.exclude_node String No The blacklist of node names that are not expanded. include_path String No The whitelist of node paths that are expanded. exclude_path String No The blacklist of node paths that are not expanded. jmes String No The JMESPath expression used to convert field values into JSON objects and extract specific values. output String No The field name that is returned for a value extracted by using a JMESPath expression. jmes_ignore_none Bool No Specifies whether to skip a field if the value of this field cannot be extracted by using a JMESpath expression. Default value: True. This values indicates that a field is skipped if the value of this field cannot be extracted by using a JMESpath expression. Otherwise, an empty string is returned. mode String No The overwrite mode for a field. Default value: fill-auto. For information about other values of this parameter, see Field check and overwrite modes. - JSON field expanding and filtering
- If a whitelist is specified, only the content included in the whitelist is displayed
in the result. An example regular expression to specify a whitelist is
e_json("json_data_filed", ...., include_node=r'key\d+')
. - If a blacklist is specified, only the content included in the blacklist is not displayed
in the result. An example regular expression to specify a blacklist is
e_json("json_data_filed", ...., exclude_node=r'key\d+')
. - The regular expressions
include_path
andexclue_path
are used to match each specified node path. Periods (.
) are used to separate the matched paths.
- If a whitelist is specified, only the content included in the whitelist is displayed
in the result. An example regular expression to specify a whitelist is
- Filtering fields by using JMESPath expressions
JMESPath expressions are used to select and compute data.
- Selects a list of element attributes from the specified JSON path:
e_json(..., jmes="cve.vendors[*].product",output="product")
. - Splices element attributes from a specified JSON path by using commas (,):
e_json(..., jmes="join(',', cve.vendors[*].name)",output="vendors")
. - Computes and retrieves the maximum value of each attribute of each element in the
specified JSON path:
e_json(..., jmes="max(words[*].score)",output="hot_word")
. - Returns an empty string if the specified JSON path is not located when computing the
maximum value of each attribute of each element in the specified JSON path:
e_json(..., jmes="max(words[*].score)",output="hot_word")
.
- Selects a list of element attributes from the specified JSON path:
- parent_list and parent_rlist
Examples
Raw log entry:data: { "k1": 100, "k2": { "k3": 200, "k4": { "k5": 300 } } }
- The parent_list expression sorts the parent nodes from left to right.
Result:e_json("data", fmt='{parent_list[0]}-{parent_list[1]}#{current}')
data: ... <Raw log> data-k2#k3: 200 data-k2#k5: 300
- The parent_rlist expression sorts the parent nodes from right to left.
Result:e_json("data", fmt='{parent_list[1]}-{parent_list[0]}#{current}')
data: ... <Raw log> data-k2#k3: 200 k2-k4#k5: 300
- The parent_list expression sorts the parent nodes from left to right.
- Response
An event where new fields and values are added is returned.Note If the e_json function is used to parse a string that does not follow the JSON syntax, the string is not parsed and the raw string is returned.
- Examples
- Example 1: Expand a field.
- Raw log entry:
data: {"k1": 100, "k2": 200}
- Transformation rule:
e_json("data",depth=1)
- Result:
data: {"k1": 100, "k2": 200} k1: 100 k2: 200
- Raw log entry:
- Example 2: Add a prefix and suffix to a field.
- Raw log entry:
data: {"k1": 100, "k2": 200}
- Transformation rule:
e_json("data", prefix="data_", suffix="_end")
- Result:
data: {"k1": 100, "k2": 200} data_k1_end: 100 data_k2_end: 200
- Raw log entry:
- Example 3: Expand a field in different formats.
- Raw log entry:
data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } }
- Transformation rule:
e_json("data", fmt='full') e_json("data", fmt='parent') e_json("data", fmt='root')
- Result:
fmt='full': data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } } data.k1: 100 data.k2.k3: 200 data.k2.k4.k5: 300 fmt='parent' data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } } data.k1: 100 k2.k3: 200 k4.k5: 300 fmt='root': data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } } data.k1: 100 data.k3: 200 data.k5: 300
- Raw log entry:
- Example 4: Specify the sep, prefix, and suffix parameters in the e_json function.
- Raw log entry:
data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } }
- Transformation rule:
e_json("data", fmt='parent', sep="@", prefix="__", suffix="__")
- Result:
data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } } data@__k1__: 100 k2@__k3__: 200 k4@__k5__: 300
- Raw log entry:
- Example 5: Specify the fmt_array parameter in the e_json function.
- Raw log entry:
people: [{"name": "xm", "sex": "boy"}, {"name": "xz", "sex": "boy"}, {"name": "xt", "sex": "girl"}]
- Transformation rule:
e_json("people", fmt='parent', fmt_array="{parent_rlist[0]}-{index}")
- Result:
people: [{"name": "xm", "sex": "boy"}, {"name": "xz", "sex": "boy"}, {"name": "xt", "sex": "girl"}] people-0.name: xm people-0.sex: boy people-1.name: xz people-1.sex: boy people-2.name: xt people-2.sex: girl
- Raw log entry:
- Example 6: Use JMESPath expressions in the e_json function.
- Raw log entry:
data: { "people": [{"first": "James", "last": "d"}, {"first": "Jacob", "last": "e"}], "foo": {"bar": "baz"} }
- Transformation rule:
e_json("data", jmes='foo', output='jmes_output0') e_json("data", jmes='foo.bar', output='jmes_output1') e_json("data", jmes='people[0].last', output='jmes_output2') e_json("data", jmes='people[*].first', output='jmes_output3')
- Result:
jmes_output0: {"bar": "baz"} jmes_output1: baz jmes_output2: d jmes_output3: ["james", "jacob"] data: { "people": [{"first": "James", "last": "d"}, {"first": "Jacob", "last": "e"}], "foo": {"bar": "baz"} }
- Raw log entry:
- Example 1: Expand a field.
e_csv, e_psv, and e_tsv
- Syntax
e_csv(Source field name, Destination field list, sep=",", quote='"', restrict=True, mode="fill-auto") e_psv(Source field name, Destination field list, sep="|", quote='"', restrict=True, mode="fill-auto") e_tsv(Source field name, Destination field list, sep="\t", quote='"', restrict=True, mode="fill-auto")
- Parameters
Parameter Type Required Description Source field name Arbitrary Yes The name of a source field. The field name can contain any characters. If the field does not exist in the specified event, no actions are performed. For more information about how to set special field names, see Event structure and fields. Destination field list Arbitrary Yes Field names that are separated by the specified delimiter. Field names can be an array of strings, such as
["error", "message", "result"]
.If the field names do not contain commas (,), you can use commas (,) as delimiters between the field names, for example,
"error, message, result"
.For more information about how to set special field names, see Event structure and fields.
sep String No The delimiter used to separate each field name. It must be a single character. quote String No The character used to enclose a value. You must specify this parameter if a field value contains delimiters. restrict Bool No Specifies whether the restricted mode is enabled. Default value: False. This value indicates that the restricted mode is disabled. If the number of values that are separated differs from the number of destination field names: - In restricted mode, no actions are performed.
- In non-restricted mode, values are assigned to the fields that can be matched.
mode String No The overwrite mode for a field. Default value: fill-auto. For information about other values of this parameter, see Field check and overwrite modes. - Response
This function returns an event that contains new field values.
- Example: The
e_csv
function is used to extract field values. Thee_psv
ande_tsv
functions work in a similar manner to the e_csv function.- Raw log entry:
content: 106.39.189.28,10/Jun/2019:11:32:16 +0800,m.aliyun.cn,GET /zf/11874.html HTTP/1.1,200,0.077,6404,10.11.186.82:8001,200,0.060,https://yz.m.aliyun.cn/s?q=%E8%9B%8B%E8%8A%B1%E9%BE%99%E9%A1%BB%E9%9D%A2%E7%9A%84%E5%81%9A%E6%B3%95&from=wy878378&uc_param_str=dnntnwvepffrgibijbprsvdsei,-,Mozilla/5.0 (Linux; Android 9; HWI-AL00 Build/HUAWEIHWI-AL00) AppleWebKit/537.36,-,-
- Transformation rule:
e_csv("content2", "remote_addr, time_local,host,request,status,request_time,body_bytes_sent,upstream_addr,upstream_status, upstream_response_time,http_referer,http_x_forwarded_for,http_user_agent,session_id,guid")
- Result:
content: 106.39.189.28,10/Jun/2019:11:32:16 +0800,m.aliyun.cn,GET /zf/11874.html HTTP/1.1,200,0.077,6404,10.11.186.82:8001,200,0.060,https://yz.m.aliyun.cn/s?q=%E8%9B%8B%E8%8A%B1%E9%BE%99%E9%A1%BB%E9%9D%A2%E7%9A%84%E5%81%9A%E6%B3%95&from=wy878378&uc_param_str=dnntnwvepffrgibijbprsvdsei,-,Mozilla/5.0 (Linux; Android 9; HWI-AL00 Build/HUAWEIHWI-AL00) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Mobile Safari/537.36,-,- body_bytes_sent: 6404 guid: - host: m.aliyun.cn http_referer: https://yz.m.aliyun.cn/s?q=%E8%9B%8B%E8%8A%B1%E9%BE%99%E9%A1%BB%E9%9D%A2%E7%9A%84%E5%81%9A%E6%B3%95&from=wy878378&uc_param_str=dnntnwvepffrgibijbprsvdsei http_user_agent: Mozilla/5.0 (Linux; Android 9; HWI-AL00 Build/HUAWEIHWI-AL00) AppleWebKit/537.36 http_x_forwarded_for: - remote_addr: 106.39.189.28 request: GET /zf/11874.html HTTP/1.1 request_time: 0.077 session_id: - status: 200 time_local: 10/Jun/2019:11:32:16 +0800 topic: syslog-forwarder upstream_addr: 10.11.186.82:8001 upstream_response_time: 0.060 upstream_status: 200
- Raw log entry:
e_kv
- Syntax
e_kv(Source field name or source field list, sep="=", quote='"', escape=False, prefix="", suffix="", mode="fill-auto")
- Parameters
Parameter Type Required Description Source field name or source field list String or string list Yes The name of the source field or names of source fields. For more information about how to set special field names, see Event structure and fields. sep String No The delimiter used to separate a key and its value in a regular expression. Default value: =
. A delimiter can contain more than one character.Note You can use non-capturing groups in a regular expression, but you cannot use capturing groups. For more information about grouping, see Group.quote String No The parameter used to enclose a field value. Default value: "
.Note We recommend that you specify this parameter to enclose a dynamic value extracted from a field, for example,a="abc"
andb="xyz"
. If you do not specify this parameter, the extracted field values can contain only the following characters:letters, digits, underscores (_), hyphens (-), percent signs (%), and tildes (~)
. For example, the fields extracted from the expressiona=ab12_-.%~|abc b=123
area: ab12_-.%~
andb: 123
.escape Bool No Specifies whether to extract escape characters contained in a field value. Default value: False
. This value indicates that escape characters contained in a field value are not extracted. For example, the valueabc\"xyz
of thekey
field is extracted from the expressionkey="abc\"xyz"
by default. If theescape
parameter is set to True, the extracted value isabc"xyz
.prefix String No The prefix added to an extracted field name. suffix String No The suffix added to an extracted field name. mode String No The overwrite mode for a field. Default value: fill-auto. For information about other values of this parameter, see Field check and overwrite modes. - Response
An event where new fields and values are added is returned.
- Examples
- Example 1: Use the e_kv function in the default format to extract field values.
- Raw log entry:
http_refer: https://yz.m.sm.cn/s?q=asd&a=1&b=2
Note If the raw log entry isrequest_uri: a1=1&a2=&a3=3
and the value of a2 is empty, a2 cannot be extracted by using the e_kv() function. However, you can extract the value of a2 by using the e_regex() function, for example, e_regex("request_uri",r'(\w +)=([>=&] *)',{r "\1":r "\2"}, mode="overwrite"). - Transformation rule:
e_kv("http_refer")
- Result:
http_refer: https://yz.m.sm.cn/s?q=asd&a=1&b=2 q: asd a: 1 b: 2
- Raw log entry:
- Example 2: Add a prefix and suffix to extracted fields.
- Raw log entry:
http_refer: https://yz.m.sm.cn/s?q=asd&a=1&b=2
- Transformation rule:
e_kv("http_refer",sep="=", quote='"', escape=False, prefix="data_", suffix="_end", mode="fill-auto")
- Result:
http_refer: https://yz.m.sm.cn/s?q=asd&a=1&b=2 data_q_end: asd data_a_end: 1 data_b_end: 2
- Raw log entry:
- Example 3: Avoid extracting escape characters contained in a field value.
- Raw log entry:
content2: k1:"v1\"abc", k2:"v2", k3: "v3"
- Transformation rule:
e_kv("content2", escape=True)
- Result:
content2: k1:"v1\"abc", k2:"v2", k3: "v3" k1: v1"abc k2: v2 k3: v3
- Raw log entry:
- Example 1: Use the e_kv function in the default format to extract field values.
e_kv_delimit
- Syntax
e_kv_delimit(Source field name or source field list, pair_sep=r"\s", kv_sep="=", prefix="", suffix="", mode="fill-auto")
- Parameters
Parameter Type Required Description Source field name or source field list String or string list Yes The name of the source field or names of source fields. For more information about how to set special field names, see Event structure and fields. pair_sep String No The regular expression used to separate key-value pairs. Default value: \s
. You can also use\s\w
andabc\s
to separate key-value pairs.Note If you want to use strings to separate fields, we recommend that you use str_replace or regex_replace to convert a string into characters. Then, you can set the characters as the value of the pair_sep parameter in the e_kv_delimiter function to separate fields.kv_sep String No The regular expression that is used to separate key-value pairs. Default value: =
. The regular expression can contain more than one character.Note You can use non-capturing groups in a regular expression, but you cannot use capturing groups. For more information about grouping, see Group.prefix String No The prefix added to an extracted field name. suffix String No The suffix added to an extracted field name. mode String No The overwrite mode for a field. Default value: fill-auto. For information about other values of this parameter, see Field check and overwrite modes. - Response
An event where new fields and values are added is returned.
- Examples
- Example 1: Use the e_kv_delimit function in the default format to extract field values.
- Raw log entry:
data: "i=c1 k1=v1 k2=v2 k3=v3"
Note If the raw log entry isrequest_uri: a1=1&a2=&a3=3
and the value of a2 is empty, a2 cannot be extracted by using the e_kv() function. However, you can use the e_regex() function, for example, e_regex("request_uri",r'(\w+)=([^=&]*)',{r"\1":r"\2"}, mode="overwrite"). - Transformation rule:
e_kv_delimit("data")
- Result:
data: "i=c1 k1=v1 k2=v2 k3=v3" i: c1 k2: v2 k1: v1 k3: v3
- Raw log entry:
- Example 2: Extract fields whose names and values are separated by
&?
.- Raw log entry:
data: "k1=v1&k2=v2? k3=v3"
- Transformation rule:
e_kv_delimit("data",pair_sep=r"&?")
- Result:
data: "k1=v1&k2=v2? k3=v3" k2: v2 k1: v1 k3: v3
- Raw log entry:
- Example 3: Extract fields by using a regular expression.
- Raw log entry:
data: "k1=v1 k2:v2 k3=v3"
- Transformation rule:
e_kv_delimit("data", kv_sep=r"(?:=|:)")
- Result:
data: "k1=v1 k2:v2 k3=v3" k2: v2 k1: v1 k3: v3
- Raw log entry:
- Example 1: Use the e_kv_delimit function in the default format to extract field values.
e_syslogrfc
- Syntax
e_syslogrfc(Source field name, rfc, fields_info=None, mode='overwrite')
- Parameters
Parameter Type Required Description Source field name Arbitrary Yes The name of the source field. You must include content in the field to retrieve the value of the priority
field.rfc String Yes The RFC protocol that is used by syslog. Valid values: SYSLOGRFC3164 and SYSLOGRFC5424. fields_info Dict No Each field is represented by a key-value pair. The key is the original field name and the value is the new field name. The fields that can be renamed and their new names are shown below: {" _severity_":"sev","_facility_":"fac","_severitylabel_":"sevlabel","_facilitylabel_":"faclabel"}
mode String No The overwrite mode for the field. Default value: overwrite. For information about other values of this parameter, see Field check and overwrite modes. - Response
An event where new fields and values are added is returned.
- Examples
- Example 1: Use the e_syslogrfc function in the default format to extract field values.
- Raw log entry:
receive_time: 1558663265 _priority_: 13 _version_: 1 _log_time_: 2019-05-06 11:50:16.015554+08:00 _hostname_: iZbp1a65x3r1vhpe94fi2qZ _program_: root _procid_: - _msgid_: - _extradata_: - _content_: twish
- Transformation rule:
e_syslogrfc("_priority_","SYSLOGRFC5424")
- Result:
receive_time: 1558663265 _priority_: 13 _version_: 1 _log_time_: 2019-05-06 11:50:16.015554+08:00 _hostname_: iZbp1a65x3r1vhpe94fi2qZ _program_: root _procid_: - _msgid_: - _extradata_: - _content_: twish _facility_: 1 _severity_: 5 _severitylabel_: Notice: normal but significant condition _facilitylabel_: user-level messages
- Raw log entry:
- Example 2: Specify the fields_info parameter to extract renamed fields and values.
- Raw log entry:
receive_time: 1558663265 _priority_: 13 _version_: 1 _log_time_: 2019-05-06 11:50:16.015554+08:00 _hostname_: iZbp1a65x3r1vhpe94fi2qZ _program_: root _procid_: - _msgid_: - _extradata_: - _content_: twish
- Transformation rule:
# Rename fields. e_syslogrfc("_priority_","SYSLOGRFC5424",{"_facility_": "fac", "_severity_": "sev", "_facilitylabel_": "_facility_label_", "_severitylabel_": "_severity_label_"})
- Result:
receive_time: 1558663265 _priority_: 13 _version_: 1 _log_time_: 2019-05-06 11:50:16.015554+08:00 _hostname_: iZbp1a65x3r1vhpe94fi2qZ _program_: root _procid_: - _msgid_: - _extradata_: - _content_: twish _facility_: 1 _severity_: 5 _severity_label_: Notice: normal but significant condition _facility_label_: user-level messages
- Raw log entry:
- Example 1: Use the e_syslogrfc function in the default format to extract field values.
e_anchor
Extracts the values between the specified start and end positions.
- Syntax
e_anchor (Source field name, anchor_rules,fields,restrict=False,mode="overwrite")
- Parameters
Parameter Type Required Description Source field name Arbitrary Yes The field name to be extracted. anchor_rules String Yes The rules that are used to extract strings, for example, User = *; Severity = *;. The content to be extracted is indicated by an asterisk (*). For logs that are displayed in the Log Service console in the Key : Value format, a space exists before the Value by default. When you specify the anchor_rules parameter, do not enter the default space.Note When you specify source fields, you cannot use asterisks (*) as prefixes or suffixes.fields Arbitrary Yes The corresponding field of each value after the values of source fields are extracted. The fields can be in a string list, for example, ["user", "job", "result"]. If the field names do not contain commas (,), you can use commas (,) to separate strings, for example, "user, job, result". For more information about how to set special field names, see Event structure and fields. Special field names can contain special characters except for asterisks (*). You can use an asterisk (*) to skip a field. Example: Only user and result are extracted from "user,*,result". For more information, see Example 10.
restrict Bool No Specifies whether to enable the restricted mode. Default value: False. This value indicates that the restricted mode is disabled. For example, assume that the number of values that are extracted differs from the number of the destination field names. - In restricted mode, no actions are performed.
- In non-restricted mode, values are assigned to the matched fields.
mode String No Default value: overwrite. For more information, see Field check and overwrite modes. - Response
The extracted content is returned.
- Examples
- Example 1: Extract the values of specified fields from a log entry.
e_anchor("content","User=*: severity=*:",["user_field","severity_field"])
Raw log entry:
content : "Aug 2 04:06:08: host=10.1.1.124: local/ssl2 notice mcpd[3772]: User=jsmith@demo.com: severity=warning: 01070638:5: Pool member 172.31.51.22:0 monitor status down."
Result:content : "Aug 2 04:06:08: host=10.1.1.124: local/ssl2 notice mcpd[3772]: User=jsmith@demo.com: severity=warning: 01070638:5: Pool member 172.31.51.22:0 monitor status down." user_field : jsmith@demo.com severity_field : warning
- Example 2: Extract multiple values in the JSON array format.
e_anchor("content",'name_list":*,"university":*},', ["name_list","universities"])
Raw log entry:content : '"information":{"name_list":["Twiss","Evan","Wind","like"],"university":["UCL","Stanford University","CMU"]},"other":"graduate"'
Result:content : '"information":{"name_list":["Twiss","Evan","Wind","like"],"university":["UCL","Stanford University","CMU"]},"other":"graduate"' name_list : ["Twiss","Evan","Wind","like"] universities : ["UCL","Stanford University","CMU"]
- Example 3: Extract values that contain special characters.
e_anchor("content", "(+*) * \"*\"",["Year","Date","Msg"])
Raw log entry:content : (+2019) June 24 "I am iron man"
Result:content : (+2019) June 24 "I am iron man" Year : 2019 Date : June 24 Msg : I am iron man
- Example 4: Extract values that contain special characters.
e_anchor("content", "\x09\x09\x09*/55.0 */537.36",["Google", "Apple"])
Raw log entry:content : \x09\x09\x09Chrome/55.0 Safari/537.36
Result:content : \x09\x09\x09Chrome/55.0 Safari/537.36 Google : Chrome Apple : Safari
- Example 5: Extract values that contain special characters. All the characters that
follow "MESSAGE:" are also the value of the content field.
e_anchor("content","* INFO *: \n To: *\n Subject: *",["time","message","email","subject"])
Raw log entry:content : 12:08:10,651 INFO sample_server ReportEmailer:178 - DEBUG SENDING MESSAGE: To: example@aliyun.com Subject: New line Breaks in Message
Result:content : 12:08:10,651 INFO sample_server ReportEmailer:178 - DEBUG SENDING MESSAGE: To: example@aliyun.com Subject: New line Breaks in Message time : 12:08:10,651 message : sample_server ReportEmailer:178 - DEBUG SENDING MESSAGE email : example@aliyun.com subject : New line Breaks in Message
- Example 6: Extract values that contain special characters. The extracted values are
delimited by invisible tabs (\t).
e_anchor("content","\tI'm * in","word") # You can also use the following transformation rule to copy the value of the content field. However, you do not need to copy the default space character. e_anchor("content"," I'm * in","word")
Raw log entry:content : I'm tabbed in
Result:content : I'm tabbed in word : tabbed
- Example 7: Extract values that contain special characters. The extracted values are
delimited by visible tabs (\t).
e_anchor("content","\tI'm * in","word") # You can also use the following transformation rule: e_anchor("content"," I'm * in","word")
Raw log entry:content : \tI'm tabbed in
Result:content : \tI'm tabbed in word : tabbed
- Example 8: Extract the values between the specified start and end positions in restricted
mode.
e_anchor("content","I * to * having",["v_word", "n_word","asd"],restrict=True)
Raw log entry:content : I used to love having snowball fight with my friends and building snowmen on the streets around our neighborhood
Result:content : I used to love having snowball fight with my friends and building snowmen on the streets around our neighborhood
- Example 9: Extract the values between the specified start and end positions in non-restricted
mode.
e_anchor("content","love * fight with my * and",["test1","test2","test13"],restrict=False)
Raw log entry:content : I used to love having snowball fight with my friends and building snowmen on the streets around our neighborhood
Result:content : I used to love having snowball fight with my friends and building snowmen on the streets around our neighborhood test1 : having snowball test2 : friends
- Example 10: Extract field values and rename field names.
e_anchor('content', 'compare the * of natural disasters to man-made *', 'n-word,*')
Raw log entry:content : I used to love having snowball fight with my friends and building snowmen on the streets around our neighborhood
Result:content : Could you compare the severity of natural disasters to man-made disasters n-word : severity
- Example 1: Extract the values of specified fields from a log entry.