This topic describes the syntax and parameters of functions that parse JSON, XML, and Protobuf data and a function that provides geolocation data from IP addresses. The topic also gives some examples of these functions.

Functions

Type Function Description
JSON json_select Extracts or calculates specific values from a JSON expression based on the JMESPath syntax.
json_parse Parses a JSON text to a JSON object.
XML xml_to_json Converts XML data to JSON data.
Protobuf pb_to_json Converts Protobuf data to JSON data.
IP geo_parse Identifies the city, province, and country to which data belongs based on an IP address.

json_select

  • Syntax
    json_select(JSON text, "JMESPath expression", default=None, restrict=False)
  • Parameters
    Parameter Type Required Description
    JSON text Arbitrary data type Yes The JSON expression or fields from which a field is extracted.
    JMESPath expression String Yes The JMESPath expression, which represents the field to be extracted.
    default Arbitrary data type No If the field to be extracted does not exist, the value of this parameter is assigned to the field and then returned. The default value of the parameter is None.
    restrict Bool No Indicates whether the restricted mode is enabled. In the restricted mode, an error is returned if non-standard JSON text is specified. Default value: False. If the value of the restrict parameter is changed to True, the restricted mode is enabled.
  • Response

    The extracted field value is returned.

  • Examples
    • Example 1: To extract the value of the name field.
      Raw log entry:
      content:  {"name": "xiaoming", "age": 10}
      Transformation rule:
      e_set("json_filter",json_select(v("content"), "name"))
      Result:
      content:  {"name": "xiaoming", "age": 10}
      json_filter:  xiaoming
    • Example 2: To extract all values of the name field.
      Raw log entry:
      content:  {"name": ["xiaoming", "xiaowang", "xiaoli"], "age": 10}
      Transformation rule:
      e_set("json_filter",json_select(v("content"), "name[*]"))
      e_set("json_filter2",str_join("*", json_select(v('content'), "name[*]")))
      Result:
      content:  {"name": ["xiaoming", "xiaowang", "xiaoli"], "age": 10}
      json_filter:  ["xiaoming", "xiaowang", "xiaoli"]
      json_filter2: xiaoming*xiaowang*xiaoli
    • Example 3: To extract the value of the name field and return the value of the default parameter.
      Raw log entry:
      content:  {"name": "xiaoming", "age": 10}
      Transformation rule:
      e_set("json_filter",json_select(v("content"), "name1.name2", default="default"))
      Result:
      content:  {"name": "xiaoming", "age": 10}
      json_filter: default

json_parse

  • Syntax
    json_parse(JSON text, default=None, restrict=False)
  • Parameters
    Parameter Type Required Description
    JSON text String Yes The fields that need to be parsed.
    default Arbitrary data type No If the field to be extracted does not exist, the value of the parameter is assigned to the field and then returned. The default value of the parameter is None.
    restrict Bool No Indicates whether the restricted mode is enabled. In the restricted mode, an error is returned if non-standard JSON text is specified. Default value: False. If the value of the restrict parameter is changed to True, the restricted mode is enabled.
  • Response

    A JSON object is returned.

  • Examples
    Raw log entry:
    content:  {"abc": 123, "xyz": "test" }
    Transformation rule:
    e_set("json",json_parse(v("content")))
    Result:
    content:  {"abc": 123, "xyz": "test" }
    json:  {"abc": 123, "xyz": "test"}

xml_to_json

  • Syntax
    xml_to_json(XML text)
  • Parameters
    Parameter Type Required Description
    XML text String Yes The XML fields that need to be converted.
  • Response

    JSON-formatted data is returned.

  • Examples
    Raw log entry:
    str : <? xml version="1.0" ? ><data><countryname="Liechtenstein"><rank>1</rank><year>2008</year><gdppc>141100</gdppc><neighborname="Austria"direction="E"/><neighborname="Switzerland"direction="W"/></country><countryname="Singapore"><rank>4</rank><year>2011</year><gdppc>59900</gdppc><neighborname="Malaysia"direction="N"/></country><countryname="Panama"><rank>68</rank><year>2011</year><gdppc>13600</gdppc><neighborname="Costa Rica"direction="W"/><neighborname="Colombia"direction="E"/></country></data>
    Transformation rule:
    e_set("str_json",xml_to_json(v("str")))
    Result:
    str : <? xmlversion="1.0"? ><data><countryname="Liechtenstein"><rank>1</rank><year>2008</year><gdppc>141100</gdppc><neighborname="Austria"direction="E"/><neighborname="Switzerland"direction="W"/></country><countryname="Singapore"><rank>4</rank><year>2011</year><gdppc>59900</gdppc><neighborname="Malaysia"direction="N"/></country><countryname="Panama"><rank>68</rank><year>2011</year><gdppc>13600</gdppc><neighborname="Costa Rica"direction="W"/><neighborname="Colombia"direction="E"/></country></data>
    str_json : {"data": {"country": [{"@name": "Liechtenstein", "rank": "1", "year": "2008", "gdppc": "141100", "neighbor": [{"@name": "Austria", "@direction": "E"}, {"@name": "Switzerland", "@direction": "W"}]}, {"@name": "Singapore", "rank": "4", "year": "2011", "gdppc": "59900", "neighbor": {"@name": "Malaysia", "@direction": "N"}}, {"@name": "Panama", "rank": "68", "year": "2011", "gdppc": "13600", "neighbor": [{"@name": "Costa Rica", "@direction": "W"}, {"@name": "Colombia", "@direction": "E"}]}]}}

pb_to_json

  • Syntax
    pb_to_json(pb_data,pb_py_path_str,pb_fun_name)
  • Parameters
    Parameter Type Required Description
    pb_data String Yes The Protobuf data or fields that need to be converted into JSON-formatted data.
    pb_py_path_str String Yes The name of the Protobuf file. For example, the name of the addressbook.proto file is addressbook.
    pb_fun_name String Yes The name of the message in the Protobuf file.
  • Response

    JSON-formatted data is returned.

  • Examples
    addressbook.proto file:
    '''addressbook.proto file'''
    syntax = "proto3";
    message Person {
      string name = 1;
      int32 id = 2;
      string email = 3;
      string university = 5;
      int32 age = 6;
    
      enum PhoneType {
        MOBILE = 0;
        HOME = 1;
        WORK = 2;
      }
    
      message PhoneNumber {
        string number = 1;
        PhoneType type = 2;
      }
    
      repeated PhoneNumber phones = 4;
    }
    
    message AddressBook {
      repeated Person people = 1;
    }
    Raw log entry:
    content: \n3\n\x05twiss\x10\x01\x1a\x0ftwiss@gkate.com"\x0f\n\x0b13099922287\x10\x01*\x04Henu0\x18\nC\n\x04Iran\x10\x02\x1a\x10Iran@alibaba.com"\x0f\n\x0b13022244455\x10\x01*\x14West Leak University0\x02
    Transformation rule:
    e_set("data_json",pb_to_json(v("content"),"addressbook","AddressBook"))
    Result:
    content: \n3\n\x05twiss\x10\x01\x1a\x0ftwiss@gkate.com"\x0f\n\x0b13099922287\x10\x01*\x04Henu0\x18\nC\n\x04Iran\x10\x02\x1a\x10Iran@alibaba.com"\x0f\n\x0b13022244455\x10\x01*\x14West Leak University0\x02
    data_json : {'people': [{'name': 'twiss', 'id': 1, 'email': 'twiss@gkate.com', 'phones': [{'number': '13099922287', 'type': 1}], 'university': 'Henu', 'age': 24}, {'name': 'Iran', 'id': 2, 'email': 'Iran@alibaba.com', 'phones': [{'number': '13022244455', 'type': 1}], 'university': 'West Leak University', 'age': 2}]}

geo_parse

  • Syntax
    geo_parse("ip",ip_db,keep_fields=None)
  • Parameters
    Parameter Type Required Description
    IP IP address string Yes The IP address.
    ip_db IP address database Yes The IP address database. The res_oss_file(endpoint, ak_id, ak_key, bucket, file, format='text', change_detect_interval=0,fetch_interval=2,refresh_retry_max=60,encoding='utf8',error='ignore') method is used to access the IP address database. The format parameter is set to binary. For more information about parameters in the res_oss_file() method, see res_oss_file.
    keep_fields Tuple No Indicates the keys contained in the returned dictionary. By default, the keys contained in the returned dictionary are as follows:
    • city: the name of the city
    • region: the name of the province
    • country: the name of the country

    For example, keep_fields=("city","country") indicates that only the city and country keys are returned.

    The keep_fields parameter can also be used to rename the keys in the dictionary. For example, (("city","cty"),("country","state")) indicates that the city and country keys are renamed cty and state respectively in the returned data.

  • Response
    A dictionary is returned in the following format:
    {
      "city": "...",
      "province":"...",
      "country": "..."
    }
  • Examples
    • Example 1
      Raw log entry:
      ip : 1.2.3.4
      Transformation rule:
      e_set("geo",geo_parse(v("ip"), ip_db=res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',
                                                       ak_id='your ak_id',
                                                       ak_key='your ak_key',
                                                       bucket='your bucket', file='ipipfree.ipdb',
                                                                     format='binary',change_detect_interval=20)))
      Result:
      ip : 1.2.3.4
      geo :{'city': 'Hangzhou', 'province':'Zhejiang Province','country': 'China'}
    • Example 2: To select keys in the dictionary to return and rename the keys.
      Raw log entry:
      ip : 1.2.3.4
      Transformation rule:
      e_set("geo",geo_parse(v("ip"), ip_db=res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',
                                                       ak_id='your ak_id',
                                                       ak_key='your ak_key',
                                                       bucket='your bucket', file='ipipfree.ipdb',
                                                                     format='binary',change_detect_interval=20),keep_fields=(("city","cty"),("country","state"),("province","pro"))))
      Result:
      ip : 1.2.3.4
      geo :{ "state": "China","pro": "Zhejiang Province","cty": "Hangzhou"}
    • Example 3: To select keys in the dictionary to return.
      Raw log entry:
      ip : 1.2.3.4
      Transformation rule:
      e_set("geo",geo_parse(v("ip"), ip_db=res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',
                                                       ak_id='your ak_id',
                                                       ak_key='your ak_key',
                                                       bucket='your bucket', file='ipipfree.ipdb',
                                                                     format='binary',change_detect_interval=20),keep_fields=("country","province")))
      Result:
      ip : 1.2.3.4
      geo :{ "country": "China","province": "Zhejiang Province"}