This topic describes the syntax of value extraction functions and provides parameter descriptions and function examples.

Functions

Type Function Description
Regular expression-based extraction e_regex Extracts the values of fields in an event by using a regular expression and assigns the values to other fields.
JSON-based extraction e_json Performs operations on JSON objects in specified event fields. The operations include expanding JSON data, extracting JSON data with JMESPath expressions, and extracting JSON data with JMESPath expressions and then expanding JSON data.
Delimiter-based extraction e_csv, e_psv, and e_tsv Extracts the values of multiple predefined fields based on user-defined delimiters.
  • e_csv: uses a comma (,) as the default delimiter.
  • e_psv: uses a vertical bar (|) as the default delimiter.
  • e_tsv: uses a tab (\t) as the default delimiter.
Key-value-based extraction e_kv Extracts key-value pairs of multiple source fields by using the quote parameter.
e_kv_delimit Extracts the key-value pairs of multiple source fields by using delimiters.
Syslog-based extraction e_syslogrfc Calculates the values of the facility and severity fields based on the known value of the priority field. Then, obtains the corresponding level information in the Request for Comments (RFC) protocol.
Anchor-rules-based extraction e_anchor Extracts the values between the specified start and end positions.

e_regex

  • Syntax 
    e_regex(Source field name, Regular expression with named capturing groups)
e_regex(Source field name, Regular expression with non-capturing groups, fields_info(Destination field name))
e_regex(Source field name, Regular expression with non-capturing groups, fields_info(An array of destination field names))
e_regex(Source field name, Regular expression with capturing groups, fields_info(An array of destination field names))
e_regex(Source field name, Regular expression with capturing groups, fields_info(Destination dictionary))
e_regex(... Preceding parameters...mode="fill-auto")
  • Parameters
    Parameter Type Required Description
    Source field name Arbitrary Yes The name of a source field. The field name can contain any characters. If the field does not exist in the specified event, no actions are performed. For more information about how to set special field names, see Event structure and fields.
    Regular expression String Yes The regular expression that is used to extract the value of a field.
    Note Regular expressions with non-capturing groups are used in some scenarios. A non-capturing group uses the prefix that consists of a question mark and a colon (?:). For example, \w+@\w+\.\w(?:\.\cn)?. For more information about non-capturing groups, see Non-capturing group.
    fields_info String/ List/ Dict No The name of the destination field. You must specify this parameter if you do not use a regular expression with named capturing groups.
    mode String No The overwrite mode for a field. Default value: fill-auto. For information about other values of this parameter, see Field check and overwrite modes.
  • Response

    An event where new fields and values are added is returned.

  • Example 1: Extract the value of a single field.
    • Raw log entry: 
      msg:  192.168.1.1 http://... 127.0.0.0
    • Transformation rule: 
      e_regex("msg",r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}","ip")
    • Result: 
      msg:  192.168.1.1 http://... 127.0.0.0
ip:  192.168.1.1
  • Example 2: Extract the values of multiple fields.
    • Raw log entry: 
      msg:  192.168.1.1 http://... 127.0.0.0
    • Transformation rule: 
      e_regex("msg",r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}",["server_ip","client_ip"])
    • Result: 
      msg:  192.168.1.1 http://... 127.0.0.0
server_ip:  192.168.1.1
client_ip:  127.0.0.0
  • Example 3: Extract the values of multiple fields by using capturing groups.
    • Raw log entry: 
      content:  start sys version: deficience, err: 2
    • Transformation rule: 
      e_regex("content",r"start sys version: (\w+),\s*err: (\d+)",["version","error"])
    • Result: 
      content:  start sys version: deficience, err: 2
error:  2
version:  deficience
  • Example 4: Extract the values of multiple fields by using named capturing groups.
    • Raw log entry: 
      content:  start sys version: deficience, err: 2
    • Transformation rule: 
      e_regex("content",r"start sys version: (? P<version>\w+),\s*err: (? P<error>\d+)")
    • Result: 
      content:  start sys version: deficience, err: 2
error:  2
version:  deficience
  • Example 5: Dynamically change the fields and extract the values of fields.
    • Raw log entry: 
      dict:  varify:123
    • Transformation rule: 
      e_regex("dict",r"(\w+):(\d+)",{r"k_\1": r"v_\2"})
    • Result: 
      dict:  varify:123
k_varify:  v_123

e_json

  • Syntax 
    e_json(Source field name, expand=None, depth=100, prefix="__", suffix="__", fmt="simple", sep=".", 
     expand_array=True, fmt_array="{parent}_{index}", 
     include_node=r"[\u4e00-\u9fa5\u0800-\u4e00a-zA-Z][\w\-\.]*",  
     exclude_node="", include_path="", exclude_path="",
     jmes="", output="", jmes_ignore_none=False, mode='fill-auto'
)
  • Parameters
    Parameter Type Required Description
    Source field name String Yes The name of a source field. The field name can contain any characters. If the field does not exist in the specified event, no actions are performed. For more information about how to set special field names, see Event structure and fields.
    expand Bool No Specifies whether to expand the source field.
    • If the jmes field is not specified, the value of this parameter is set to True by default. This indicates that the source field is expanded.
    • If the jmes field is specified, the value of this parameter is set to False by default. This indicates that the field is not expanded.
    depth Number No The depth of field expanding. Value range: 1 to 2000. Default value: 100. The value 1 indicates that only the first tier of the field is expanded.
    prefix String No The prefix added to an expanded field.
    suffix String No The suffix added to an expanded field.
    fmt String No The formatting method of a field. Valid values:
    • simple: The current node name is used as the field name. This is the default value. The format is {prefix}{current}{suffix}.
    • full: The names of the current node and its closest parent node are combined and used as the field name. The format is {parent_list_str}{sep}{prefix}{current}{suffix}. sep specifies the delimiter. Default value: ..
    • parent: The names of the current node and all its parent nodes are used as the field name. The format is {parent}{sep}{prefix}{current}{suffix}. sep specifies the delimiter. Default value: ..
    • root: specifies to combine the names of the current node and its root node as the field name. The format is {parent_list[0]}{sep}{prefix}{current}{suffix}. sep specifies the delimiter. Default value:..
    sep String No The delimiter used to separate parent and child nodes during formatting. You must specify this parameter if the value of the fmt parameter is set to full, parent, or root. Default value: ..
    expand_array Bool No Specifies whether to expand an array of fields. To expand an array, the following format is used: {parent}_{index}.
    fmt_array String No The formatting method to expand an array of fields. The format is {parent_rlist[0]}_{index}. Alternatively, you can use up to five placeholders to expand an array: parent_list, current, sep, prefix, and suffix.
    include_node String/ Number No The whitelist of node names that are expanded. By default, node names that contain digits, letters, and specific special characters (_. -) are expanded.
    exclude_node String No The blacklist of node names that are not expanded.
    include_path String No The whitelist of node paths that are expanded.
    exclude_path String No The blacklist of node paths that are not expanded.
    jmes String No The JMESPath expression used to convert field values into JSON objects and extract specific values.
    output String No The field name that is returned for a value extracted by using a JMESPath expression.
    jmes_ignore_none Bool No Specifies whether to skip a field if the value of this field cannot be extracted by using a JMESpath expression. Default value: True. This values indicates that a field is skipped if the value of this field cannot be extracted by using a JMESpath expression. Otherwise, an empty string is returned.
    mode String No The overwrite mode for a field. Default value: fill-auto. For information about other values of this parameter, see Field check and overwrite modes.
  • JSON field expanding and filtering
    • If a whitelist is specified, only the content included in the whitelist is displayed in the result. An example regular expression to specify a whitelist is e_json("json_data_filed", ...., include_node=r'key\d+').
    • If a blacklist is specified, only the content included in the blacklist is not displayed in the result. An example regular expression to specify a blacklist is e_json("json_data_filed", ...., exclude_node=r'key\d+').
    • The regular expressions include_path and exclue_path are used to match each specified node path. Periods (.) are used to separate the matched paths.
  • Filtering fields by using JMESPath expressions
    JMESPath expressions are used to select and compute data.
    • Selects a list of element attributes from the specified JSON path: e_json(..., jmes="cve.vendors[*].product",output="product").
    • Splices element attributes from a specified JSON path by using commas (,): e_json(..., jmes="join(',', cve.vendors[*].name)",output="vendors").
    • Computes and retrieves the maximum value of each attribute of each element in the specified JSON path: e_json(..., jmes="max(words[*].score)",output="hot_word").
    • Returns an empty string if the specified JSON path is not located when computing the maximum value of each attribute of each element in the specified JSON path: e_json(..., jmes="max(words[*].score)",output="hot_word").
  • parent_list and parent_rlist

    Examples

    Raw log entry: 
    data: {
  "k1": 100,
  "k2": {
    "k3": 200,
    "k4": {
      "k5": 300
    }
  }
}
    • The parent_list expression sorts the parent nodes from left to right. 
      e_json("data", fmt='{parent_list[0]}-{parent_list[1]}#{current}')
      Result: 
      data: ... <Raw log>
data-k2#k3: 200
data-k2#k5: 300
    • The parent_rlist expression sorts the parent nodes from right to left. 
      e_json("data", fmt='{parent_list[1]}-{parent_list[0]}#{current}')
      Result: 
      data: ... <Raw log>
data-k2#k3: 200
k2-k4#k5: 300
  • Response
    An event where new fields and values are added is returned.
    Note If the e_json function is used to parse a string that does not follow the JSON syntax, the string is not parsed and the raw string is returned.
  • Examples
    • Example 1: Expand a field.
      • Raw log entry: 
        data: {"k1": 100, "k2": 200}
      • Transformation rule: 
        e_json("data",depth=1)
      • Result: 
        data: {"k1": 100, "k2": 200}
k1: 100
k2: 200
    • Example 2: Add a prefix and suffix to a field.
      • Raw log entry: 
        data: {"k1": 100, "k2": 200}
      • Transformation rule: 
        e_json("data", prefix="data_", suffix="_end")
      • Result: 
        data: {"k1": 100, "k2": 200}
data_k1_end: 100
data_k2_end: 200
    • Example 3: Expand a field in different formats.
      • Raw log entry: 
        data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } }
      • Transformation rule: 
        e_json("data", fmt='full')
e_json("data", fmt='parent')
e_json("data", fmt='root')
      • Result: 
        fmt='full':
  data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } }
  data.k1: 100
  data.k2.k3: 200
  data.k2.k4.k5: 300
fmt='parent'
  data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } }
  data.k1: 100
  k2.k3: 200
  k4.k5: 300
fmt='root':
  data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } }
  data.k1: 100
  data.k3: 200
  data.k5: 300
    • Example 4: Specify the sep, prefix, and suffix parameters in the e_json function.
      • Raw log entry: 
        data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } }
      • Transformation rule: 
        e_json("data", fmt='parent', sep="@", prefix="__", suffix="__")
      • Result: 
        data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } }
data@__k1__: 100
k2@__k3__: 200
k4@__k5__: 300
    • Example 5: Specify the fmt_array parameter in the e_json function.
      • Raw log entry: 
        people: [{"name": "xm", "sex": "boy"}, {"name": "xz", "sex": "boy"}, {"name": "xt", "sex": "girl"}]
      • Transformation rule: 
        e_json("people", fmt='parent', fmt_array="{parent_rlist[0]}-{index}")
      • Result: 
        people: [{"name": "xm", "sex": "boy"}, {"name": "xz", "sex": "boy"}, {"name": "xt", "sex": "girl"}]
people-0.name: xm
people-0.sex: boy
people-1.name: xz
people-1.sex: boy
people-2.name: xt
people-2.sex: girl
    • Example 6: Use JMESPath expressions in the e_json function.
      • Raw log entry: 
        data: { "people": [{"first": "James", "last": "d"}, 
                     {"first": "Jacob", "last": "e"}],  
          "foo": {"bar": "baz"}
        }
      • Transformation rule: 
        e_json("data", jmes='foo', output='jmes_output0')
e_json("data", jmes='foo.bar', output='jmes_output1')
e_json("data", jmes='people[0].last', output='jmes_output2')
e_json("data", jmes='people[*].first', output='jmes_output3')
      • Result: 
        jmes_output0: {"bar": "baz"}
jmes_output1: baz
jmes_output2: d
jmes_output3: ["james", "jacob"]
data: { "people": [{"first": "James", "last": "d"}, 
                     {"first": "Jacob", "last": "e"}],  
          "foo": {"bar": "baz"}
        }

e_csv, e_psv, and e_tsv

  • Syntax 
    e_csv(Source field name, Destination field list, sep=",", quote='"', restrict=True, mode="fill-auto")
e_psv(Source field name, Destination field list, sep="|", quote='"', restrict=True, mode="fill-auto")
e_tsv(Source field name, Destination field list, sep="\t", quote='"', restrict=True, mode="fill-auto")
  • Parameters
    Parameter Type Required Description
    Source field name Arbitrary Yes The name of a source field. The field name can contain any characters. If the field does not exist in the specified event, no actions are performed. For more information about how to set special field names, see Event structure and fields.
    Destination field list Arbitrary Yes Field names that are separated by the specified delimiter.

    Field names can be an array of strings, such as ["error", "message", "result"].

    If the field names do not contain commas (,), you can use commas (,) as delimiters between the field names, for example, "error, message, result".

    For more information about how to set special field names, see Event structure and fields.

    sep String No The delimiter used to separate each field name. It must be a single character.
    quote String No The character used to enclose a value. You must specify this parameter if a field value contains delimiters.
    restrict Bool No Specifies whether the restricted mode is enabled. Default value: False. This value indicates that the restricted mode is disabled. If the number of values that are separated differs from the number of destination field names:
    • In restricted mode, no actions are performed.
    • In non-restricted mode, values are assigned to the fields that can be matched.
    mode String No The overwrite mode for a field. Default value: fill-auto. For information about other values of this parameter, see Field check and overwrite modes.
  • Response

    This function returns an event that contains new field values.

  • Example: The e_csv function is used to extract field values. The e_psv and e_tsv functions work in a similar manner to the e_csv function.
    • Raw log entry: 
      content:  106.39.189.28,10/Jun/2019:11:32:16 +0800,m.aliyun.cn,GET /zf/11874.html HTTP/1.1,200,0.077,6404,10.11.186.82:8001,200,0.060,https://yz.m.aliyun.cn/s?q=%E8%9B%8B%E8%8A%B1%E9%BE%99%E9%A1%BB%E9%9D%A2%E7%9A%84%E5%81%9A%E6%B3%95&from=wy878378&uc_param_str=dnntnwvepffrgibijbprsvdsei,-,Mozilla/5.0 (Linux; Android 9; HWI-AL00 Build/HUAWEIHWI-AL00) AppleWebKit/537.36,-,-
    • Transformation rule: 
      e_csv("content2", "remote_addr, time_local,host,request,status,request_time,body_bytes_sent,upstream_addr,upstream_status, upstream_response_time,http_referer,http_x_forwarded_for,http_user_agent,session_id,guid")
    • Result: 
      content:  106.39.189.28,10/Jun/2019:11:32:16 +0800,m.aliyun.cn,GET /zf/11874.html HTTP/1.1,200,0.077,6404,10.11.186.82:8001,200,0.060,https://yz.m.aliyun.cn/s?q=%E8%9B%8B%E8%8A%B1%E9%BE%99%E9%A1%BB%E9%9D%A2%E7%9A%84%E5%81%9A%E6%B3%95&from=wy878378&uc_param_str=dnntnwvepffrgibijbprsvdsei,-,Mozilla/5.0 (Linux; Android 9; HWI-AL00 Build/HUAWEIHWI-AL00) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Mobile Safari/537.36,-,-
  body_bytes_sent:  6404
guid:  -
host:  m.aliyun.cn
http_referer:  https://yz.m.aliyun.cn/s?q=%E8%9B%8B%E8%8A%B1%E9%BE%99%E9%A1%BB%E9%9D%A2%E7%9A%84%E5%81%9A%E6%B3%95&from=wy878378&uc_param_str=dnntnwvepffrgibijbprsvdsei
http_user_agent:  Mozilla/5.0 (Linux; Android 9; HWI-AL00 Build/HUAWEIHWI-AL00) AppleWebKit/537.36
http_x_forwarded_for:  -
remote_addr:  106.39.189.28
request:  GET /zf/11874.html HTTP/1.1
request_time:  0.077
session_id:  -
status:  200
time_local:  10/Jun/2019:11:32:16 +0800
topic:  syslog-forwarder
upstream_addr:  10.11.186.82:8001
upstream_response_time:  0.060
upstream_status:  200

e_kv

  • Syntax 
    e_kv(Source field name or source field list, sep="=", quote='"', escape=False, prefix="", suffix="", mode="fill-auto")
  • Parameters
    Parameter Type Required Description
    Source field name or source field list String or string list Yes The name of the source field or names of source fields. For more information about how to set special field names, see Event structure and fields.
    sep String No The delimiter used to separate a key and its value in a regular expression. Default value: =. A delimiter can contain more than one character.
    Note You can use non-capturing groups in a regular expression, but you cannot use capturing groups. For more information about grouping, see Group.
    quote String No The parameter used to enclose a field value. Default value: ".
    Note We recommend that you specify this parameter to enclose a dynamic value extracted from a field, for example, a="abc" and b="xyz". If you do not specify this parameter, the extracted field values can contain only the following characters: letters, digits, underscores (_), hyphens (-), percent signs (%), and tildes (~). For example, the fields extracted from the expression a=ab12_-.%~|abc b=123 are a: ab12_-.%~ and b: 123.
    escape Bool No Specifies whether to extract escape characters contained in a field value. Default value: False. This value indicates that escape characters contained in a field value are not extracted. For example, the value abc\"xyz of the key field is extracted from the expression key="abc\"xyz" by default. If the escape parameter is set to True, the extracted value is abc"xyz.
    prefix String No The prefix added to an extracted field name.
    suffix String No The suffix added to an extracted field name.
    mode String No The overwrite mode for a field. Default value: fill-auto. For information about other values of this parameter, see Field check and overwrite modes.
  • Response

    An event where new fields and values are added is returned.

  • Examples
    • Example 1: Use the e_kv function in the default format to extract field values.
      • Raw log entry: 
        http_refer:  https://yz.m.sm.cn/s?q=asd&a=1&b=2
        Note If the raw log entry is request_uri: a1=1&a2=&a3=3 and the value of a2 is empty, a2 cannot be extracted by using the e_kv() function. However, you can extract the value of a2 by using the e_regex() function, for example, e_regex("request_uri",r'(\w +)=([>=&] *)',{r "\1":r "\2"}, mode="overwrite").
      • Transformation rule: 
        e_kv("http_refer")
      • Result: 
        http_refer:  https://yz.m.sm.cn/s?q=asd&a=1&b=2
q: asd
a: 1
b: 2
    • Example 2: Add a prefix and suffix to extracted fields.
      • Raw log entry: 
        http_refer:  https://yz.m.sm.cn/s?q=asd&a=1&b=2
      • Transformation rule: 
        e_kv("http_refer",sep="=", quote='"', escape=False, prefix="data_", suffix="_end", mode="fill-auto")
      • Result: 
        http_refer:  https://yz.m.sm.cn/s?q=asd&a=1&b=2
data_q_end: asd
data_a_end: 1
data_b_end: 2
    • Example 3: Avoid extracting escape characters contained in a field value.
      • Raw log entry: 
        content2: k1:"v1\"abc", k2:"v2", k3: "v3"
      • Transformation rule: 
        e_kv("content2", escape=True)
      • Result: 
        content2:  k1:"v1\"abc", k2:"v2", k3: "v3"
k1: v1"abc
k2: v2
k3: v3

e_kv_delimit

  • Syntax 
    e_kv_delimit(Source field name or source field list, pair_sep=r"\s", kv_sep="=", prefix="", suffix="", mode="fill-auto")
  • Parameters
    Parameter Type Required Description
    Source field name or source field list String or string list Yes The name of the source field or names of source fields. For more information about how to set special field names, see Event structure and fields.
    pair_sep String No The regular expression used to separate key-value pairs. Default value: \s. You can also use \s\w and abc\s to separate key-value pairs.
    Note If you want to use strings to separate fields, we recommend that you use str_replace or regex_replace to convert a string into characters. Then, you can set the characters as the value of the pair_sep parameter in the e_kv_delimiter function to separate fields.
    kv_sep String No The regular expression that is used to separate key-value pairs. Default value: =. The regular expression can contain more than one character.
    Note You can use non-capturing groups in a regular expression, but you cannot use capturing groups. For more information about grouping, see Group.
    prefix String No The prefix added to an extracted field name.
    suffix String No The suffix added to an extracted field name.
    mode String No The overwrite mode for a field. Default value: fill-auto. For information about other values of this parameter, see Field check and overwrite modes.
  • Response

    An event where new fields and values are added is returned.

  • Examples
    • Example 1: Use the e_kv_delimit function in the default format to extract field values.
      • Raw log entry: 
        data: "i=c1 k1=v1 k2=v2 k3=v3"
        Note If the raw log entry is request_uri: a1=1&a2=&a3=3 and the value of a2 is empty, a2 cannot be extracted by using the e_kv() function. However, you can use the e_regex() function, for example, e_regex("request_uri",r'(\w+)=([^=&]*)',{r"\1":r"\2"}, mode="overwrite").
      • Transformation rule: 
        e_kv_delimit("data")
      • Result: 
        data: "i=c1 k1=v1 k2=v2 k3=v3"
i: c1
k2: v2
k1: v1
k3: v3
    • Example 2: Extract fields whose names and values are separated by &?.
      • Raw log entry: 
        data: "k1=v1&k2=v2? k3=v3"
      • Transformation rule: 
        e_kv_delimit("data",pair_sep=r"&?")
      • Result: 
        data: "k1=v1&k2=v2? k3=v3"
k2: v2
k1: v1
k3: v3
    • Example 3: Extract fields by using a regular expression.
      • Raw log entry: 
        data: "k1=v1 k2:v2 k3=v3"
      • Transformation rule: 
        e_kv_delimit("data", kv_sep=r"(?:=|:)")
      • Result: 
        data: "k1=v1 k2:v2 k3=v3"
k2: v2
k1: v1
k3: v3

e_syslogrfc

  • Syntax 
    e_syslogrfc(Source field name, rfc, fields_info=None, mode='overwrite')
  • Parameters
    Parameter Type Required Description
    Source field name Arbitrary Yes The name of the source field. You must include content in the field to retrieve the value of the priority field.
    rfc String Yes The RFC protocol that is used by syslog. Valid values: SYSLOGRFC3164 and SYSLOGRFC5424.
    fields_info Dict No Each field is represented by a key-value pair. The key is the original field name and the value is the new field name. The fields that can be renamed and their new names are shown below: {" _severity_":"sev","_facility_":"fac","_severitylabel_":"sevlabel","_facilitylabel_":"faclabel"}
    mode String No The overwrite mode for the field. Default value: overwrite. For information about other values of this parameter, see Field check and overwrite modes.
  • Response

    An event where new fields and values are added is returned.

  • Examples
    • Example 1: Use the e_syslogrfc function in the default format to extract field values.
      • Raw log entry: 
        receive_time: 1558663265
_priority_: 13
_version_: 1
_log_time_: 2019-05-06 11:50:16.015554+08:00
_hostname_: iZbp1a65x3r1vhpe94fi2qZ
_program_: root
_procid_: -
_msgid_: -
_extradata_: -
_content_: twish
      • Transformation rule: 
        e_syslogrfc("_priority_","SYSLOGRFC5424")
      • Result: 
        receive_time: 1558663265
_priority_: 13
_version_: 1
_log_time_: 2019-05-06 11:50:16.015554+08:00
_hostname_: iZbp1a65x3r1vhpe94fi2qZ
_program_: root
_procid_: -
_msgid_: -
_extradata_: -
_content_: twish
_facility_: 1
_severity_: 5
_severitylabel_: Notice: normal but significant condition
_facilitylabel_: user-level messages
    • Example 2: Specify the fields_info parameter to extract renamed fields and values.
      • Raw log entry: 
        receive_time: 1558663265
_priority_: 13
_version_: 1
_log_time_: 2019-05-06 11:50:16.015554+08:00
_hostname_: iZbp1a65x3r1vhpe94fi2qZ
_program_: root
_procid_: -
_msgid_: -
_extradata_: -
_content_: twish
      • Transformation rule: 
        # Rename fields.
e_syslogrfc("_priority_","SYSLOGRFC5424",{"_facility_": "fac", "_severity_": "sev", "_facilitylabel_": "_facility_label_", "_severitylabel_": "_severity_label_"})
      • Result: 
        receive_time: 1558663265
_priority_: 13
_version_: 1
_log_time_: 2019-05-06 11:50:16.015554+08:00
_hostname_: iZbp1a65x3r1vhpe94fi2qZ
_program_: root
_procid_: -
_msgid_: -
_extradata_: -
_content_: twish
_facility_: 1
_severity_: 5
_severity_label_: Notice: normal but significant condition
_facility_label_: user-level messages

e_anchor

Extracts the values between the specified start and end positions.

  • Syntax 
    e_anchor (Source field name, anchor_rules,fields,restrict=False,mode="overwrite")
  • Parameters
    Parameter Type Required Description
    Source field name Arbitrary Yes The field name to be extracted.
    anchor_rules String Yes The rules that are used to extract strings, for example, User = *; Severity = *;. The content to be extracted is indicated by an asterisk (*).
    For logs that are displayed in the Log Service console in the Key : Value format, a space exists before the Value by default. When you specify the anchor_rules parameter, do not enter the default space.e_anchor
    Note When you specify source fields, you cannot use asterisks (*) as prefixes or suffixes.
    fields Arbitrary Yes The corresponding field of each value after the values of source fields are extracted. The fields can be in a string list, for example, ["user", "job", "result"]. If the field names do not contain commas (,), you can use commas (,) to separate strings, for example, "user, job, result". For more information about how to set special field names, see Event structure and fields. Special field names can contain special characters except for asterisks (*).

    You can use an asterisk (*) to skip a field. Example: Only user and result are extracted from "user,*,result". For more information, see Example 10.

    restrict Bool No Specifies whether to enable the restricted mode. Default value: False. This value indicates that the restricted mode is disabled. For example, assume that the number of values that are extracted differs from the number of the destination field names.
    • In restricted mode, no actions are performed.
    • In non-restricted mode, values are assigned to the matched fields.
    mode String No Default value: overwrite. For more information, see Field check and overwrite modes.
  • Response

    The extracted content is returned.

  • Examples
    • Example 1: Extract the values of specified fields from a log entry. 
      e_anchor("content","User=*: severity=*:",["user_field","severity_field"])

      Raw log entry:

      content : "Aug 2 04:06:08: host=10.1.1.124: local/ssl2 notice mcpd[3772]: User=jsmith@demo.com: severity=warning: 01070638:5: Pool member 172.31.51.22:0 monitor status down."
      Result: 
      content : "Aug 2 04:06:08: host=10.1.1.124: local/ssl2 notice mcpd[3772]: User=jsmith@demo.com: severity=warning: 01070638:5: Pool member 172.31.51.22:0 monitor status down."
user_field : jsmith@demo.com
severity_field : warning
    • Example 2: Extract multiple values in the JSON array format. 
      e_anchor("content",'name_list":*,"university":*},', ["name_list","universities"])
      Raw log entry: 
      content : '"information":{"name_list":["Twiss","Evan","Wind","like"],"university":["UCL","Stanford University","CMU"]},"other":"graduate"'
      Result: 
      content : '"information":{"name_list":["Twiss","Evan","Wind","like"],"university":["UCL","Stanford University","CMU"]},"other":"graduate"'
name_list : ["Twiss","Evan","Wind","like"]
universities : ["UCL","Stanford University","CMU"]
    • Example 3: Extract values that contain special characters. 
      e_anchor("content", "(+*) * \"*\"",["Year","Date","Msg"])
      Raw log entry: 
      content : (+2019) June 24 "I am iron man"
      Result: 
      content : (+2019) June 24 "I am iron man"
Year : 2019
Date : June 24
Msg : I am iron man
    • Example 4: Extract values that contain special characters. 
      e_anchor("content", "\x09\x09\x09*/55.0 */537.36",["Google", "Apple"])
      Raw log entry: 
      content : \x09\x09\x09Chrome/55.0 Safari/537.36
      Result: 
      content : \x09\x09\x09Chrome/55.0 Safari/537.36
Google : Chrome
Apple : Safari
    • Example 5: Extract values that contain special characters. All the characters that follow "MESSAGE:" are also the value of the content field. 
      e_anchor("content","* INFO *: \n    To: *\n    Subject: *",["time","message","email","subject"])
      Raw log entry: 
      content : 12:08:10,651 INFO sample_server ReportEmailer:178 - DEBUG SENDING MESSAGE: 
To: example@aliyun.com
Subject: New line Breaks in Message
      Result: 
      content : 12:08:10,651 INFO sample_server ReportEmailer:178 - DEBUG SENDING MESSAGE: 
To: example@aliyun.com
Subject: New line Breaks in Message

time : 12:08:10,651
message : sample_server ReportEmailer:178 - DEBUG SENDING MESSAGE
email : example@aliyun.com
subject : New line Breaks in Message
    • Example 6: Extract values that contain special characters. The extracted values are delimited by invisible tabs (\t). 
      e_anchor("content","\tI'm * in","word")
# You can also use the following transformation rule to copy the value of the content field. However, you do not need to copy the default space character.
e_anchor("content","    I'm * in","word")
      Raw log entry: 
      content :   I'm tabbed in
      Result: 
      content :   I'm tabbed in
word : tabbed
    • Example 7: Extract values that contain special characters. The extracted values are delimited by visible tabs (\t). 
      e_anchor("content","\tI'm * in","word")
# You can also use the following transformation rule:
e_anchor("content","    I'm * in","word")
      Raw log entry: 
      content : \tI'm tabbed in
      Result: 
      content : \tI'm tabbed in
word : tabbed
    • Example 8: Extract the values between the specified start and end positions in restricted mode. 
      e_anchor("content","I * to * having",["v_word", "n_word","asd"],restrict=True)
      Raw log entry: 
      content :  I used to love having snowball fight with my friends and building snowmen on the streets around our neighborhood
      Result: 
      content : I used to love having snowball fight with my friends and building snowmen on the streets around our neighborhood
    • Example 9: Extract the values between the specified start and end positions in non-restricted mode. 
      e_anchor("content","love * fight with my * and",["test1","test2","test13"],restrict=False)
      Raw log entry: 
      content :  I used to love having snowball fight with my friends and building snowmen on the streets around our neighborhood
      Result: 
      content : I used to love having snowball fight with my friends and building snowmen on the streets around our neighborhood
test1 : having snowball
test2 : friends
    • Example 10: Extract field values and rename field names. 
      e_anchor('content', 'compare the * of natural disasters to man-made *', 'n-word,*')
      Raw log entry: 
      content :  I used to love having snowball fight with my friends and building snowmen on the streets around our neighborhood
      Result: 
      content : Could you compare the severity of natural disasters to man-made disasters
n-word : severity