This topic describes the syntax and parameters of value extraction functions. This topic also provides examples on how to use the functions.

Functions

Type Function Description
Extraction by using regular expressions e_regex Extracts the value of a field by using a regular expression and assigns the value to other fields.
Extraction based on JSON objects e_json Performs operations on JSON objects in a specified field. You can set parameters to expand JSON data, extract JSON data with JMESPath expressions, and expand extracted JSON data.
Extraction by using delimiters e_csv, e_psv, and e_tsv Extracts the value of a specified source field, parses the value into multiple values by using a custom delimiter, and then assigns the parsed values to the predefined fields.
  • e_csv: uses a comma (,) as the default delimiter.
  • e_psv: uses a vertical bar (|) as the default delimiter.
  • e_tsv: uses a tab (\t) as the default delimiter.
Extraction of key-value pairs e_kv Extracts the key-value pairs of multiple source fields and uses the character specified by the quote parameter to enclose the value in each key-value pair.
e_kv_delimit Extracts the key-value pairs from multiple source fields and uses the specified delimiter to separate the key-value pairs.
Extraction based on syslog e_syslogrfc Calculates the values of the facility and severity fields and then returns the corresponding level information. The calculation is based on the value of the priority field in an event based on the syslog protocol.
Extraction based on anchor rules e_anchor Extracts the value between the specified start and end positions.

e_regex

  • Syntax 
    e_regex(Source field name, Regular expression with named capturing groups)
e_regex(Source field name, Regular expression with non-capturing groups, fields_info(Destination field name))
e_regex(Source field name, Regular expression with non-capturing groups, fields_info(An array of destination field names))
e_regex(Source field name, Regular expression with capturing groups, fields_info(An array of destination field names))
e_regex(Source field name, Regular expression with capturing groups, fields_info(Destination dictionary))
e_regex(...Preceding parameters...mode="fill-auto")
  • Parameters
    Parameter Type Required Description
    Source field name Arbitrary Yes The name of a source field. The field name can contain any characters. If the field that you specify does not exist, no operations are performed. For more information about how to specify special field names, see Event structure and fields.
    Regular expression String Yes The regular expression that you want to use to extract the value of the specified field.
    Note Regular expressions with non-capturing groups are used in some scenarios. A non-capturing group uses the prefix that consists of a question mark and a colon (?:), for example, \w+@\w+\.\w(?:\.\cn)?. For more information about non-capturing groups, see Non-capturing group.
    fields_info String/ List/ Dict No The name of the destination field. If you do not use a regular expression with named capturing groups, you must specify this parameter.
    mode String No The overwrite mode of fields. Default value: fill-auto. For information about other values of this parameter, see Field check and overwrite modes.
  • Response

    A log entry in which new fields and values are added is returned.

  • Example 1: Extract the value of a single field.
    • Raw log entry: 
      msg: 192.168.1.1 http://... 127.0.0.0
    • Transformation rule: 
      e_regex("msg",r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}","ip")
    • Result: 
      msg: 192.168.1.1 http://... 127.0.0.0
ip: 192.168.1.1
  • Example 2: Extract the values of multiple fields.
    • Raw log entry: 
      msg: 192.168.1.1 http://... 127.0.0.0
    • Transformation rule: 
      e_regex("msg",r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}",["server_ip","client_ip"])
    • Result: 
      msg: 192.168.1.1 http://... 127.0.0.0
server_ip: 192.168.1.1
client_ip: 127.0.0.0
  • Example 3: Extract the values of multiple fields by using capturing groups.
    • Raw log entry: 
      content: start sys version: deficience, err: 2
    • Transformation rule: 
      e_regex("content",r"start sys version: (\w+),\s*err: (\d+)",["version","error"])
    • Result: 
      content: start sys version: deficience, err: 2
error: 2
version: deficience
  • Example 4: Extract the values of multiple fields by using named capturing groups.
    • Raw log entry: 
      content:  start sys version: deficience, err: 2
    • Transformation rule: 
      e_regex("content",r"start sys version: (?P<version>\w+),\s*err: (?P<error>\d+)")
    • Result: 
      content:  start sys version: deficience, err: 2
error:  2
version:  deficience
  • Example 5: Extract the value of a field and then construct a field based on the extracted value.
    • Raw log entry: 
      dict: varify:123
    • Transformation rule: 
      e_regex("dict",r"(\w+):(\d+)",{r"k_\1": r"v_\2"})
    • Result: 
      dict: varify:123
k_varify: v_123

e_json

  • Syntax 
    e_json(Source field name, expand=None, depth=100, prefix="__", suffix="__", fmt="simple", sep=".", 
     expand_array=True, fmt_array="{parent}_{index}", 
     include_node=r"[\u4e00-\u9fa5\u0800-\u4e00a-zA-Z][\w\-\.]*",  
     exclude_node="", include_path="", exclude_path="",
     jmes="", output="", jmes_ignore_none=False, mode='fill-auto'
)
  • Parameters
    Parameter Type Required Description
    Source field name String Yes The name of a source field. The field name can contain any characters. If the field that you specify does not exist, no operations are performed. For more information about how to specify special field names, see Event structure and fields.
    expand Bool No Specifies whether to expand the source field.
    • If you don not specify the jmes parameter, the value of this parameter is set to True by default. This value indicates that the source field is expanded.
    • If you specify the jmes parameter, the value of this parameter is set to False by default. This value indicates that the field is not expanded.
    depth Number No The depth to which the function expands the source field. Valid values: 1 to 2000. Default value: 100. The value 1 indicates that only the first level of the field is expanded.
    prefix String No The prefix that you want to add to an expanded field.
    suffix String No The suffix that you want to add to an expanded field.
    fmt String No The formatting method of a field. Valid values:
    • simple: The name of the current node is used as the field name. This is the default value. The format is {prefix}{current}{suffix}.
    • full: The names of the current node and its closest parent node are combined and used as the field name. The format is {parent_list_str}{sep}{prefix}{current}{suffix}. sep specifies the delimiter. Default value: ..
    • parent: The names of the current node and all its parent nodes are used as the field name. The format is {parent}{sep}{prefix}{current}{suffix}. sep specifies the delimiter. Default value: ..
    • root: The names of the current node and its root node are combined and used as the field name. The format is {parent_list[0]}{sep}{prefix}{current}{suffix}. sep specifies the delimiter. Default value:..
    sep String No The delimiter that you want to use to separate parent and child nodes when the function formats data. If you set the fmt parameter to full, parent, or root, you must specify this parameter. Default value: ..
    expand_array Bool No Specifies whether to expand an array of fields. The function expands an array in the following format: {parent}_{index}.
    fmt_array String No The formatting method that you want to use to expand an array of fields. The format is {parent_rlist[0]}_{index}. You can also use up to five placeholders to expand an array: parent_list, current, sep, prefix, and suffix.
    include_node String/ Number No The whitelist of node names that you want to expand. By default, node names that contain digits, letters, and special characters _.- are expanded.
    exclude_node String No The blacklist of node names that you do not want to expand.
    include_path String No The whitelist of node paths that you want to expand.
    exclude_path String No The blacklist of node paths that you do not want to expand.
    jmes String No The JMESPath expression that you want to use to convert field values to JSON objects and extract specific values.
    output String No The field name that you want to return for a value extracted by using a JMESPath expression.
    jmes_ignore_none Bool No Specifies whether to skip a field if the value of this field cannot be extracted by using a JMESpath expression. Default value: True. This value indicates that a field is skipped if the value of this field cannot be extracted by using a JMESpath expression. Otherwise, an empty string is returned.
    mode String No The overwrite mode of fields. Default value: fill-auto. For information about other values of this parameter, see Field check and overwrite modes.
  • JSON field expanding and filtering
    • If a whitelist is specified, only the content included in the whitelist is displayed in the result. An example regular expression used to specify a whitelist is e_json("json_data_filed", ...., include_node=r'key\d+').
    • If a blacklist is specified, only the content included in the blacklist is not displayed in the result. An example regular expression used to specify a blacklist is e_json("json_data_filed", ...., exclude_node=r'key\d+').
    • The regular expressions include_path and exclue_path are used to match each specified node path. Periods (.) are used to separate the paths that match the regular expressions.
  • Field filtering by using JMESPath expressions
    JMESPath expressions are used to select and compute data.
    • Selects a list of element attributes from the specified JSON path: e_json(..., jmes="cve.vendors[*].product",output="product").
    • Splices element attributes from the specified JSON path by using commas (,): e_json(..., jmes="join(',', cve.vendors[*].name)",output="vendors").
    • Computes and retrieves the maximum value of each attribute of each element in the specified JSON path: e_json(..., jmes="max(words[*].score)",output="hot_word").
    • Returns an empty string if the specified JSON path is not found when the function computes the maximum value of each attribute of each element in the specified JSON path: e_json(..., jmes="max(words[*].score)",output="hot_word").
  • parent_list and parent_rlist

    The following examples show how to use the parent_list and parent_rlist expressions:

    Raw log entry: 
    data: {
  "k1": 100,
  "k2": {
    "k3": 200,
    "k4": {
      "k5": 300
    }
  }
}
    • The parent_list expression sorts the parent nodes from left to right. 
      e_json("data", fmt='{parent_list[0]}-{parent_list[1]}#{current}')
      Result: 
      data: ...<Raw log entry>
data-k2#k3: 200
data-k2#k5: 300
    • The parent_rlist expression sorts the parent nodes from right to left. 
      e_json("data", fmt='{parent_list[0]}-{parent_list[1]}#{current}')
      Result: 
      data: ...<Raw log entry>
data-k2#k3: 200
k2-k4#k5: 300
  • Response
    A log entry in which new fields and values are added is returned.
    Note If you use the e_json function to parse a string that does not follow the JSON syntax, the function does not parse the string and returns the original string.
  • Examples
    • Example 1: Expand a field.
      • Raw log entry: 
        data: {"k1": 100, "k2": 200}
      • Transformation rule: 
        e_json("data",depth=1)
      • Result: 
        data: {"k1": 100, "k2": 200}
k1: 100
k2: 200
    • Example 2: Add a prefix and a suffix to a field.
      • Raw log entry: 
        data: {"k1": 100, "k2": 200}
      • Transformation rule: 
        e_json("data", prefix="data_", suffix="_end")
      • Result: 
        data: {"k1": 100, "k2": 200}
data_k1_end: 100
data_k2_end: 200
    • Example 3: Expand a field in different formats.
      • Raw log entry: 
        data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } }
      • Transformation rule: 
        e_json("data", fmt='full')
e_json("data", fmt='parent')
e_json("data", fmt='root')
      • Result: 
        fmt='full':
  data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } }
  data.k1: 100
  data.k2.k3: 200
  data.k2.k4.k5: 300
fmt='parent'
  data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } }
  data.k1: 100
  k2.k3: 200
  k4.k5: 300
fmt='root':
  data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } }
  data.k1: 100
  data.k3: 200
  data.k5: 300
    • Example 4: Specify the sep, prefix, and suffix parameters in the e_json function.
      • Raw log entry: 
        data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } }
      • Transformation rule: 
        e_json("data", fmt='parent', sep="@", prefix="__", suffix="__")
      • Result: 
        data: {"k1": 100, "k2": {"k3": 200, "k4": {"k5": 300} } }
data@__k1__: 100
k2@__k3__: 200
k4@__k5__: 300
    • Example 5: Specify the fmt_array parameter in the e_json function.
      • Raw log entry: 
        people: [{"name": "xm", "sex": "boy"}, {"name": "xz", "sex": "boy"}, {"name": "xt", "sex": "girl"}]
      • Transformation rule: 
        e_json("people", fmt='parent', fmt_array="{parent_rlist[0]}-{index}")
      • Result: 
        people: [{"name": "xm", "sex": "boy"}, {"name": "xz", "sex": "boy"}, {"name": "xt", "sex": "girl"}]
people-0.name: xm
people-0.sex: boy
people-1.name: xz
people-1.sex: boy
people-2.name: xt
people-2.sex: girl
    • Example 6: Use JMESPath expressions in the e_json function.
      • Raw log entry: 
        data: { "people": [{"first": "James", "last": "d"}, 
                     {"first": "Jacob", "last": "e"}],  
          "foo": {"bar": "baz"}
        }
      • Transformation rule: 
        e_json("data", jmes='foo', output='jmes_output0')
e_json("data", jmes='foo.bar', output='jmes_output1')
e_json("data", jmes='people[0].last', output='jmes_output2')
e_json("data", jmes='people[*].first', output='jmes_output3')
      • Result: 
        jmes_output0: {"bar": "baz"}
jmes_output1: baz
jmes_output2: d
jmes_output3: ["james", "jacob"]
data: { "people": [{"first": "James", "last": "d"}, 
                     {"first": "Jacob", "last": "e"}],  
          "foo": {"bar": "baz"}
        }

e_csv, e_psv, and e_tsv

  • Syntax 
    e_csv(Source field name, Destination field list, sep=",", quote='"', restrict=True, mode="fill-auto")
e_psv(Source field name, Destination field list, sep="|", quote='"', restrict=True, mode="fill-auto")
e_tsv(Source field name, Destination field list, sep="\t", quote='"', restrict=True, mode="fill-auto")
  • Parameters
    Parameter Type Required Description
    Source field name Arbitrary Yes The name of a source field. The field name can contain any characters. If the field that you specify does not exist, no operations are performed. For more information about how to specify special field names, see Event structure and fields.
    Destination field list Arbitrary Yes The field names that are separated by the specified delimiter.

    The field names can be in a string list, for example, ["error", "message", "result"].

    If the field names do not contain commas (,), you can use commas (,) as delimiters to separate the field names, for example, "error, message, result".

    For more information about how to specify special field names, see Event structure and fields.

    sep String No The delimiter that you want to use to separate field names. You must specify a single character as a delimiter.
    quote String No The character that you want to use to enclose a value. If a field value contains delimiters, you must specify this parameter.
    restrict Bool No Specifies whether to enable the restricted mode. Default value: False. This value indicates that the restricted mode is disabled. If the number of values that are separated differs from the number of destination field names, the operation that is performed by the function varies based on the mode.
    • In restricted mode, the function does not perform operations on the values.
    • In non-restricted mode, the function assigns the values to the fields that match the values.
    mode String No The overwrite mode of fields. Default value: fill-auto. For information about other values of this parameter, see Field check and overwrite modes.
  • Response

    A log entry in which new fields and values are added is returned.

  • Example

    In this example, the e_csv function is used to extract field values. The e_psv and e_tsv functions work in a similar manner to the e_csv function.

    • Raw log entry: 
      content: 106.39.189.28,10/Jun/2019:11:32:16 +0800,m.aliyun.cn,GET /zf/11874.html HTTP/1.1,200,0.077,6404,10.11.186.82:8001,200,0.060,https://yz.m.aliyun.cn/s?q=%E8%9B%8B%E8%8A%B1%E9%BE%99%E9%A1%BB%E9%9D%A2%E7%9A%84%E5%81%9A%E6%B3%95&from=wy878378&uc_param_str=dnntnwvepffrgibijbprsvdsei,-,Mozilla/5.0 (Linux; Android 9; HWI-AL00 Build/HUAWEIHWI-AL00) AppleWebKit/537.36,-,-
    • Transformation rule: 
      e_csv("content", "remote_addr, time_local,host,request,status,request_time,body_bytes_sent,upstream_addr,upstream_status, upstream_response_time,http_referer,http_x_forwarded_for,http_user_agent,session_id,guid")
    • Result: 
      content:  106.39.189.28,10/Jun/2019:11:32:16 +0800,m.aliyun.cn,GET /zf/11874.html HTTP/1.1,200,0.077,6404,10.11.186.82:8001,200,0.060,https://yz.m.aliyun.cn/s?q=%E8%9B%8B%E8%8A%B1%E9%BE%99%E9%A1%BB%E9%9D%A2%E7%9A%84%E5%81%9A%E6%B3%95&from=wy878378&uc_param_str=dnntnwvepffrgibijbprsvdsei,-,Mozilla/5.0 (Linux; Android 9; HWI-AL00 Build/HUAWEIHWI-AL00) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Mobile Safari/537.36,-,-
  body_bytes_sent:  6404
guid:  -
host:  m.aliyun.cn
http_referer:  https://yz.m.aliyun.cn/s?q=%E8%9B%8B%E8%8A%B1%E9%BE%99%E9%A1%BB%E9%9D%A2%E7%9A%84%E5%81%9A%E6%B3%95&from=wy878378&uc_param_str=dnntnwvepffrgibijbprsvdsei
http_user_agent:  Mozilla/5.0 (Linux; Android 9; HWI-AL00 Build/HUAWEIHWI-AL00) AppleWebKit/537.36
http_x_forwarded_for:  -
remote_addr:  106.39.189.28
request:  GET /zf/11874.html HTTP/1.1
request_time:  0.077
session_id:  -
status:  200
time_local:  10/Jun/2019:11:32:16 +0800
topic:  syslog-forwarder
upstream_addr:  10.11.186.82:8001
upstream_response_time:  0.060
upstream_status:  200

e_kv

  • Syntax 
    e_kv(Source field name or source field list, sep="=", quote='"', escape=False, prefix="", suffix="", mode="fill-auto")
  • Parameters
    Parameter Type Required Description
    Source field name or source field list String or string list Yes The name of the source field or names of source fields. For more information about how to specify special field names, see Event structure and fields.
    sep String No The delimiter that you want to use to separate a key and its value in a regular expression. Default value: =. You can specify one or more characters as a delimiter.
    Note You can use non-capturing groups but not capturing groups in a regular expression. For more information about grouping, see Group.
    quote String No The character that you want to use to enclose a field value. Default value: ".
    Note We recommend that you set the quote parameter to a character that you want to use to enclose a dynamic value extracted from a field, for example, a="abc" and b="xyz". If you do not specify the quote parameter, the extracted field values can contain only the following characters: letters, digits, underscores (_), hyphens (-),percent signs (%), and tildes (~). For example, the fields extracted from the expression a=Chinese ab12_-.%~| abc b=123 are a: ab12_-.%~ and b: 123.
    escape Bool No Specifies whether to extract escape characters contained in a field value. Default value: False. This value indicates that escape characters contained in a field value are not extracted. For example, the value abc of the key field is extracted from the expression key="abc\"xyz" by default. If the escape parameter is set to True, the extracted value is abc"xyz.
    prefix String No The prefix that you want to add to an extracted field name.
    suffix String No The suffix that you want to add to an extracted field name.
    mode String No The overwrite mode of fields. Default value: fill-auto. For information about other values of this parameter, see Field check and overwrite modes.
  • Response

    A log entry in which new fields and values are added is returned.

  • Examples
    • Example 1: Use the e_syslogrfc function to extract field values in the default format.
      • Raw log entry: 
        http_refer: https://yz.m.sm.cn/s?q=asd&a=1&b=2
        Note If the raw log entry is request_uri: a1=1&a2=&a3=3 and the value of a2 is empty, the e_kv() function cannot extract the value of a2. You can use the e_regex() function to extract the value of a2, for example, e_regex("request_uri",r'(\w+)=([^=&]*)',{r"\1":r"\2"},mode="overwrite").
      • Transformation rule: 
        e_kv("http_refer")
      • Result: 
        http_refer: https://yz.m.sm.cn/s?q=asd&a=1&b=2
q: asd
a: 1
b: 2
    • Example 2: Add a prefix and a suffix to extracted fields.
      • Raw log entry: 
        http_refer: https://yz.m.sm.cn/s?q=asd&a=1&b=2
      • Transformation rule: 
        e_kv("http_refer",sep="=", quote='"', escape=False, prefix="data_", suffix="_end", mode="fill-auto")
      • Result: 
        http_refer: https://yz.m.sm.cn/s?q=asd&a=1&b=2
data_q_end: asd
data_a_end: 1
data_b_end: 2
    • Example 3: Extract escape characters from a field value.
      • Raw log entry: 
        content2: k1:"v1\"abc", k2:"v2", k3: "v3"
      • Transformation rule: 
        e_kv("content2", sep=":", escape=True)
      • Result: 
        content2:  k1:"v1\"abc", k2:"v2", k3: "v3"
k1: v1"abc
k2: v2
k3: v3

e_kv_delimit

  • Syntax 
    e_kv_delimit(Source field name or source field list, pair_sep=r"\s", kv_sep="=", prefix="", suffix="", mode="fill-auto")
  • Parameters
    Parameter Type Required Description
    Source field name or source field list String or string list Yes The name of the source field or names of source fields. For more information about how to specify special field names, see Event structure and fields.
    pair_sep String No The regular expression that you want to use to separate key-value pairs. Default value: \s. You can also use \s\w and abc\s to separate key-value pairs.
    Note If you want to use strings to separate fields, we recommend that you use str_replace or regex_replace to convert a string into characters. Then, you can specify the characters as the value of the pair_sep parameter in the e_kv_delimiter function to separate fields.
    kv_sep String No The regular expression that you want to use to separate key-value pairs. Default value: =. The regular expression can contain one or more characters.
    Note You can use non-capturing groups but not capturing groups in a regular expression. For more information about grouping, see Group.
    prefix String No The prefix that you want to add to an extracted field name.
    suffix String No The suffix that you want to add to an extracted field name.
    mode String No The overwrite mode of fields. Default value: fill-auto. For information about other values of this parameter, see Field check and overwrite modes.
  • Response

    A log entry in which new fields and values are added is returned.

  • Examples
    • Example 1: Use the e_syslogrfc function to extract field values in the default format.
      • Raw log entry: 
        data: "i=c1 k1=v1 k2=v2 k3=v3"
        Note If the raw log entry is request_uri: a1=1&a2=&a3=3 and the value of a2 is empty, the e_kv() function cannot extract the value of a2. You can use the e_regex() function to extract the value of a2, for example, e_regex("request_uri",r'(\w+)=([^=&]*)',{r"\1":r"\2"}, mode="overwrite").
      • Transformation rule: 
        e_kv_delimit("data")
      • Result: 
        data: "i=c1 k1=v1 k2=v2 k3=v3"
i: c1
k2: v2
k1: v1
k3: v3
    • Example 2: Extract fields whose names and values are separated by &?.
      • Raw log entry: 
        data: "k1=v1&k2=v2?k3=v3"
      • Transformation rule: 
        e_kv_delimit("data",pair_sep=r"&?")
      • Result: 
        data: "k1=v1&k2=v2?k3=v3"
k2: v2
k1: v1
k3: v3
    • Example 3: Extract fields by using a regular expression.
      • Raw log entry: 
        data: "k1=v1 k2:v2 k3=v3"
      • Transformation rule: 
        e_kv_delimit("data", kv_sep=r"(?:=|:)")
      • Result: 
        data: "k1=v1 k2:v2 k3=v3"
k2: v2
k1: v1
k3: v3

e_syslogrfc

  • Syntax 
    e_syslogrfc(Source field name, rfc, fields_info=None, mode='overwrite')
  • Parameters
    Parameter Type Required Description
    Source field name Arbitrary Yes The name of the source field. You must include content in the field to retrieve the value of the priority field.
    rfc String Yes The RFC protocol that is used by syslog. Valid values: SYSLOGRFC3164 and SYSLOGRFC5424.
    fields_info Dict No Each field is represented by a key-value pair. The key is the original field name and the value is the new field name. The following fields can be renamed and the new names can be modified: {"_severity_":"sev","_facility_":"fac","_severitylabel_":"sevlabel","_facilitylabel_":"faclabel"}
    mode String No The overwrite mode of fields. Default value: overwrite. For information about other values of this parameter, see Field check and overwrite modes.
  • Response

    A log entry in which new fields and values are added is returned.

  • Examples
    • Example 1: Use the e_syslogrfc function to extract field values in the default format.
      • Raw log entry: 
        receive_time: 1558663265
_priority_: 13
_version_: 1
_log_time_: 2019-05-06 11:50:16.015554+08:00
_hostname_: iZbp1a65x3r1vhpe94fi2qZ
_program_: root
_procid_: -
_msgid_: -
_extradata_: -
_content_: twish
      • Transformation rule: 
        e_syslogrfc("_priority_","SYSLOGRFC5424")
      • Result: 
        receive_time: 1558663265
_priority_: 13
_version_: 1
_log_time_: 2019-05-06 11:50:16.015554+08:00
_hostname_: iZbp1a65x3r1vhpe94fi2qZ
_program_: root
_procid_: -
_msgid_: -
_extradata_: -
_content_: twish
_facility_: 1
_severity_: 5
_severitylabel_: Notice: normal but significant condition
_facilitylabel_: user-level messages
    • Example 2: Specify the fields_info parameter to extract renamed fields and values.
      • Raw log entry: 
        receive_time: 1558663265
_priority_: 13
_version_: 1
_log_time_: 2019-05-06 11:50:16.015554+08:00
_hostname_: iZbp1a65x3r1vhpe94fi2qZ
_program_: root
_procid_: -
_msgid_: -
_extradata_: -
_content_: twish
      • Transformation rule: 
        # Rename fields.
e_syslogrfc("_priority_","SYSLOGRFC5424",{"_facility_": "fac", "_severity_": "sev", "_facilitylabel_": "_facility_label_", "_severitylabel_": "_severity_label_"})
      • Result: 
        receive_time: 1558663265
_priority_: 13
_version_: 1
_log_time_: 2019-05-06 11:50:16.015554+08:00
_hostname_: iZbp1a65x3r1vhpe94fi2qZ
_program_: root
_procid_: -
_msgid_: -
_extradata_: -
_content_: twish
_facility_: 1
_severity_: 5
_severity_label_: Notice: normal but significant condition
_facility_label_: user-level messages

e_anchor

The e_anchor function is used to extract the value between the specified start and end positions.

  • Syntax 
    e_anchor (Source field name, anchor_rules,fields,restrict=False,mode="overwrite")
  • Parameters
    Parameter Type Required Description
    Source field name Arbitrary Yes The field name that you want to extract.
    anchor_rules String Yes The rules that you want to use to extract strings, for example, User = *; Severity = *;. The content that needs to be extracted is indicated by an asterisk (*).
    For logs that are displayed in the Log Service console in the Key : Value format, a space character exists before the Value by default. When you specify the anchor_rules parameter, remove the default space character. e_anchor
    Note When you specify source fields, you cannot use asterisks (*) as prefixes or suffixes.
    fields Arbitrary Yes The corresponding field of each value after the values of source fields are extracted. The field names can be in a string list, for example, ["user", "job", "result"]. If the field names do not contain commas (,), you can use commas (,) to separate strings, for example, "user, job, result". For more information about how to specify special field names, see Event structure and fields. Special field names can contain special characters except asterisks (*).

    You can use an asterisk (*) to skip a field. For example, only user and result are extracted from "user,*,result". For more information, see Example 10.

    restrict Bool No Specifies whether to enable the restricted mode. Default value: False. This value indicates that the restricted mode is disabled. If the number of values that are separated differs from the number of destination field names, the operation that is performed by the function varies based on the mode.
    • In restricted mode, the function does not perform operations on the values.
    • In non-restricted mode, the function assigns the values to the fields that match the values.
    mode String No Default value: overwrite. For more information, see Field check and overwrite modes.
  • Response

    The extracted content is returned.

  • Examples
    • Example 1: Extract the values of specified fields from a log entry. 
      e_anchor("content","User=*: severity=*:",["user_field","severity_field"])

      Raw log entry:

      content : "Aug 2 04:06:08: host=10.1.1.124: local/ssl2 notice mcpd[3772]: User=jsmith@demo.com: severity=warning: 01070638:5: Pool member 172.31.51.22:0 monitor status down."
      Result: 
      content : "Aug 2 04:06:08: host=10.1.1.124: local/ssl2 notice mcpd[3772]: User=jsmith@demo.com: severity=warning: 01070638:5: Pool member 172.31.51.22:0 monitor status down."
user_field : jsmith@demo.com
severity_field : warning
    • Example 2: Extract multiple values in the JSON array format. 
      e_anchor("content",'name_list":*,"university":*},', ["name_list","universities"])
      Raw log entry: 
      content : '"information":{"name_list":["Twiss","Evan","Wind","like"],"university":["UCL","Stanford University","CMU"]},"other":"graduate"'
      Result: 
      content : '"information":{"name_list":["Twiss","Evan","Wind","like"],"university":["UCL","Stanford University","CMU"]},"other":"graduate"'
name_list : ["Twiss","Evan","Wind","like"]
universities : ["UCL","Stanford University","CMU"]
    • Example 3: Extract values that contain special characters. 
      e_anchor("content", "(+*) * \"*\"",["Year","Date","Msg"])
      Raw log entry: 
      content : (+2019) June 24 "I am iron man"
      Result: 
      content : (+2019) June 24 "I am iron man"
Year : 2019
Date : June 24
Msg : I am iron man
    • Example 4: Extract values that contain special characters. 
      e_anchor("content", "\x09\x09\x09*/55.0 */537.36",["Google", "Apple"])
      Raw log entry: 
      content : \x09\x09\x09Chrome/55.0 Safari/537.36
      Result: 
      content : \x09\x09\x09Chrome/55.0 Safari/537.36
Google : Chrome
Apple : Safari
    • Example 5: Extract values that contain special characters. All the characters that follow "MESSAGE:" are also the value of the content field. 
      e_anchor("content","* INFO *: \n    To: *\n    Subject: *",["time","message","email","subject"])
      Raw log entry: 
      content : 12:08:10,651 INFO sample_server ReportEmailer:178 - DEBUG SENDING MESSAGE: 
To: example@aliyun.com
Subject: New line Breaks in Message
      Result: 
      content : 12:08:10,651 INFO sample_server ReportEmailer:178 - DEBUG SENDING MESSAGE: 
To: example@aliyun.com
Subject: New line Breaks in Message

time : 12:08:10,651
message : sample_server ReportEmailer:178 - DEBUG SENDING MESSAGE
email : example@aliyun.com
subject : New line Breaks in Message
    • Example 6: Extract values that contain special characters. The extracted values are separated by invisible tabs (\t). 
      e_anchor("content","\tI'm * in","word")
# You can also use the following transformation rule to copy the value of the content field. Remove the default space character from the value.
e_anchor("content","    I'm * in","word")
      Raw log entry: 
      content :   I'm tabbed in
      Result: 
      content :   I'm tabbed in
word : tabbed
    • Example 7: Extract values that contain special characters. The extracted values are separated by visible tabs (\t). 
      e_anchor("content","\tI'm * in","word")
# You can also use the following transformation rule:
e_anchor("content","    I'm * in","word")
      Raw log entry: 
      content : \tI'm tabbed in
      Result: 
      content : \tI'm tabbed in
word : tabbed
    • Example 8: Extract the values between the specified start and end positions in restricted mode. 
      e_anchor("content","I * to * having",["v_word", "n_word","asd"],restrict=True)
      Raw log entry: 
      content :  I used to love having snowball fight with my friends and building snowmen on the streets around our neighborhood
      Result: 
      content : I used to love having snowball fight with my friends and building snowmen on the streets around our neighborhood
    • Example 9: Extract the values between the specified start and end positions in non-restricted mode. 
      e_anchor("content","love * fight with my * and",["test1","test2","test13"],restrict=False)
      Raw log entry: 
      content :  I used to love having snowball fight with my friends and building snowmen on the streets around our neighborhood
      Result: 
      content : I used to love having snowball fight with my friends and building snowmen on the streets around our neighborhood
test1 : having snowball
test2 : friends
    • Example 10: Extract the value of a field and assign the extracted value to another field. 
      e_anchor('content', 'compare the * of natural disasters to man-made *', 'n-word,*')
      Raw log entry: 
      content: Could you compare the severity of natural disasters to man-made disasters
      Result: 
      content : Could you compare the severity of natural disasters to man-made disasters
n-word : severity