This topic describes the syntax and parameters of User-Agent parsing functions. This topic also provides examples on how to use the functions.

Functions

Function Description
ua_parse_device Parses User-Agent and returns the device information.
ua_parse_os Parses User-Agent and returns the operating system information.
ua_parse_agent Parses User-Agent and returns the browser information.
ua_parse_all Parses User-Agent and returns all information.
url_parse Parses a URL and returns the components of the URL.
url_parse_qs Parses the query string of a URL and returns the components of the query string.
Note The User-Agent parsing functions delete fields whose parsed values are None. For example, if the parsed device information is {'brand': None, 'family': 'Other', 'model': None}, the brand and model fields are deleted, and the final parsing result is {'family': 'Other'}.

ua_parse_device

The ua_parse_device function parses User-Agent and returns the device information.
  • Syntax

    ua_parse_device(value)
  • Parameters

    Parameter Type Required Description
    value String Yes The User-Agent string that you want to parse.
  • Response

    A JSON-formatted data set is returned.

  • Examples

    • Raw log
      http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36
    • Transformation rule
      e_set("new_column",ua_parse_device(v("http_user_agent")))
    • Result
      http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36
      new_column:{'family': 'Other'}

ua_parse_os

The ua_parse_os function parses User-Agent and returns the operating system information.
  • Syntax

    ua_parse_os(value)
  • Parameters

    Parameter Type Required Description
    value String Yes The User-Agent string that you want to parse.
  • Response

    A JSON-formatted data set is returned.

  • Examples

    • Raw log
      http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36
    • Transformation rule
      e_set("new_column",ua_parse_os(v("http_user_agent")))
    • Result
      http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36
      new_column:{'family': 'Mac OS X',
                          'major': '10',
                          'minor': '9',
                          'patch': '4'}

ua_parse_agent

The ua_parse_agent function parses User-Agent and returns the browser information.
  • Syntax

    ua_parse_agent(value)
  • Parameters

    Parameter Type Required Description
    value String Yes The User-Agent string that you want to parse.
  • Response

    A JSON-formatted data set is returned.

  • Examples

    • Raw log
      http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36
    • Transformation rule
      e_set("new_column",ua_parse_agent(v("http_user_agent")))
    • Result
      http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36
      new_column:{'family': 'Chrome', 'major': '192', 'minor': '168', 'patch': '0'}

ua_parse_all

The ua_parse_all function parses User-Agent and returns all information.
  • Syntax

    ua_parse_all(value)
  • Parameters

    Parameter Type Required Description
    value String Yes The User-Agent string that you want to parse.
  • Response

    A JSON-formatted data set is returned.

  • Examples

    • Raw log
      http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36
    • Transformation rule
      e_set("new_column",ua_parse_all(v("http_user_agent")))
    • Result
      http_user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36
      new_column: {
        "user_agent": {
          "family": "Chrome",
          "major": "192",
          "minor": "168",
          "patch": "0"
        },
        "os": {
          "family": "Mac OS X",
          "major": "10",
          "minor": "9",
          "patch": "4"
        },
        "device": {
          "family": "Mac",
          "brand": "Apple",
          "model": "Mac"
        }
      }

url_parse

The url_parse function parses a URL and returns the components of the URL.
  • Syntax

    url_parse(url, scheme="", allow_fragments=True)
  • Parameters

    Parameter Type Required Description
    value String Yes The URL that you want to parse.
    scheme String No The network protocol. This parameter is empty by default.

    If the URL that you want to parse does not contain a network protocol, the value of this parameter is returned for the scheme parameter in the result.

    allow_fragments Boolean No Specifies whether to parse the fragment part in the URL. Valid values:
    • True: parses the fragment part in the URL. A specific value is returned for the fragment parameter. This is the default value.
    • False: does not parse the fragment part in the URL. An empty string is returned for the fragment parameter.
  • Response

    JSON-formatted data is returned. The following table describes the parameters in the result.
    Parameter Description
    scheme The network protocol.
    netloc The network location.
    path The hierarchical path identifier.
    query The query component.
    fragment The fragment identifier.
  • Examples

    • Example 1: Use default parameter settings. The components of a URL are returned.
      • Raw log
        content:https://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib
      • Transformation rule
        e_set("url",url_parse(v("content")))
      • Result
        content:https://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib
        url:{
                "scheme": "https",
                "netloc": "username:username@example.com:8083",
                "path": "/hello/asdah/;type=docx",
                "query": "filename=python3.docx",
                "fragment": "urllib",
            }
    • Example 2: Set allow_fragments to False. An empty string is returned for the fragment parameter.
      • Raw log
        content:https://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib
      • Transformation rule
        e_set("url",url_parse(v("content"),allow_fragments=False))
      • Result
        content:https://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib
        url:{
                        "scheme": "https",
                "netloc": "username:username@example.com:8083",
                "path": "/hello/asdah/;type=docx",
                "query": "filename=python3.docx",
                "fragment": "",
            }
    • Example 3: Set scheme to https and set allow_fragments to False. https is returned for the scheme parameter, and an empty string is returned for the fragment parameter.
      • Raw log
        content://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib
      • Transformation rule
        e_set("url",url_parse(v("content"),scheme="https", allow_fragments=False))
      • Result
        content://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib
        url:{
               "scheme": "https",
                "netloc": "username:username@example.com:8083",
                "path": "/hello/asdah/;type=docx",
                "query": "filename=python3.docx",
                "fragment": "",
            }

url_parse_qs

The url_parse_qs function parses the query string of a URL and returns the components of the query string.
  • Syntax

    url_parse_qs(
        url_qs,
        keep_blank_values=False,
        strict_parsing=False,
        encoding="utf-8",
        errors="replace",
        ignore_multi_fields=True,
    )
  • Parameters

    Parameter Type Required Description
    url_qs String Yes The URL query string that you want to parse.
    keep_blank_values Boolean No Specifies whether to return parameters that are empty. Valid values:
    • False: does not return parameters that are empty. This is the default value.
    • True: returns parameters that are empty and returns empty strings as values for the parameters.
    strict_parsing Boolean No Specifies whether to handle a parsing error. Valid values:
    • True: triggers a ValueError exception for the parsing error.
    • False: ignores the parsing error. This is the default value.
    encoding String No The encoding method. Default value: utf-8. Escape characters that contain percent signs (%) are parsed into Unicode characters. ASCII is supported.
    errors String No The method to process characters that cannot be recognized based on the encoding method. Valid values:
    • ignore: ignores the characters.
    • strict: reports an error and discards the log that contains the characters.
    • replace: replaces the characters with question marks (?). This is the default value.
    • xmlcharrefreplace: replaces the characters with XML characters.
    ignore_multi_fields Num No The number of values that are returned for a single parameter. Valid values:
    • True: returns only the first value for each parameter. The return value is a string. This is the default value.
    • False: returns all values for each parameter. The return value is a list.
  • Response

    JSON-formatted data is returned. The following table describes the parameters in the result.
    Parameter Description
    logType The type of the log.
    uid The unique identifier of the log.
    time The time of the log.
    msg The information in the log.
  • Examples

    • Example 1: Set keep_blank_values to True. Parameters that are empty are returned.
      • Raw log
        content:logType=net_wheel_log&uid=62452****&vid=6.1.0_gf_pc&asb=1206427&git=&time=22-11-3+%e4%b8%8a11%e6%97%b649%e5%88%8633%e7%a7%92&operatingSystem=Windows+10++(10.0.0)+64bit&deviceModel=System+Product+Name+(System+manufacturer)&graphicsDeviceName=NVIDIA+GeForce+GTX+1650&graphicsDeviceType=Direct3D11&graphicsDeviceVendor=NVIDIA&graphicsDeviceVersion=Direct3D+11.0+%5blevel+11.1%5d&graphicsMemorySize=3962&systemMemorySize=8127&processorCount=6&processorFrequency=3000&processorType=Intel(R)+Core(TM)+i5-9500F+CPU+%40+3.00GHz&deviceID=96da5902a042a5f84118995f88373f73650e76be166589726****&guessUID=62452****&networkReachability=wifi&msg=GetAuthkeyRsp
      • Transformation rule
        e_set("url",url_parse_qs(v("content"), keep_blank_values=True))
      • Result
        content:logType=net_wheel_log&uid=62452****&vid=6.1.0_gf_pc&asb=1206427&git=&time=22-11-3+%e4%b8%8a11%e6%97%b649%e5%88%8633%e7%a7%92&operatingSystem=Windows+10++(10.0.0)+64bit&deviceModel=System+Product+Name+(System+manufacturer)&graphicsDeviceName=NVIDIA+GeForce+GTX+1650&graphicsDeviceType=Direct3D11&graphicsDeviceVendor=NVIDIA&graphicsDeviceVersion=Direct3D+11.0+%5blevel+11.1%5d&graphicsMemorySize=3962&systemMemorySize=8127&processorCount=6&processorFrequency=3000&processorType=Intel(R)+Core(TM)+i5-9500F+CPU+%40+3.00GHz&deviceID=96da5902a042a5f84118995f88373f73650e76be166589726****&guessUID=62452****&networkReachability=wifi&msg=GetAuthkeyRsp
        url:{
                "logType": "net_wheel_log",
                "uid": "62452****",
                "vid": "6.1.0_gf_pc",
                "asb": "1206427",
                "git": "",
                "time": "22-11-3 11:49:33",
                "operatingSystem": "Windows 10  (10.0.0) 64bit",
                "deviceModel": "System Product Name (System manufacturer)",
                "graphicsDeviceName": "NVIDIA GeForce GTX 1650",
                "graphicsDeviceType": "Direct3D11",
                "graphicsDeviceVendor": "NVIDIA",
                "graphicsDeviceVersion": "Direct3D 11.0 [level 11.1]",
                "graphicsMemorySize": "3962",
                "systemMemorySize": "8127",
                "processorCount": "6",
                "processorFrequency": "3000",
                "processorType": "Intel(R) Core(TM) i5-9500F CPU @ 3.00GHz",
                "deviceID": "96da5902a042a5f84118995f88373f73650e76be166589726****",
                "guessUID": "62452****",
                "networkReachability": "wifi",
                "msg": "GetAuthkeyRsp",
            }
    • Example 2: Set keep_blank_values to False. Parameters that are not empty are returned.
      • Raw log
        content:logType=net_wheel_log&uid=62452****&vid=6.1.0_gf_pc&asb=1206427&git=&time=22-11-3+%e4%b8%8a11%e6%97%b649%e5%88%8633%e7%a7%92&operatingSystem=Windows+10++(10.0.0)+64bit&deviceModel=System+Product+Name+(System+manufacturer)&graphicsDeviceName=NVIDIA+GeForce+GTX+1650&graphicsDeviceType=Direct3D11&graphicsDeviceVendor=NVIDIA&graphicsDeviceVersion=Direct3D+11.0+%5blevel+11.1%5d&graphicsMemorySize=3962&systemMemorySize=8127&processorCount=6&processorFrequency=3000&processorType=Intel(R)+Core(TM)+i5-9500F+CPU+%40+3.00GHz&deviceID=96da5902a042a5f84118995f88373f73650e76be166589726****&guessUID=62452****&networkReachability=wifi&msg=GetAuthkeyRsp
      • Transformation rule
        e_set("url",url_parse_qs(v("content")))
      • Result
        content:logType=net_wheel_log&uid=62452****&vid=6.1.0_gf_pc&asb=1206427&git=&time=22-11-3+%e4%b8%8a11%e6%97%b649%e5%88%8633%e7%a7%92&operatingSystem=Windows+10++(10.0.0)+64bit&deviceModel=System+Product+Name+(System+manufacturer)&graphicsDeviceName=NVIDIA+GeForce+GTX+1650&graphicsDeviceType=Direct3D11&graphicsDeviceVendor=NVIDIA&graphicsDeviceVersion=Direct3D+11.0+%5blevel+11.1%5d&graphicsMemorySize=3962&systemMemorySize=8127&processorCount=6&processorFrequency=3000&processorType=Intel(R)+Core(TM)+i5-9500F+CPU+%40+3.00GHz&deviceID=96da5902a042a5f84118995f88373f73650e76be166589726****&guessUID=62452****&networkReachability=wifi&msg=GetAuthkeyRsp
        url:{
                "logType": "net_wheel_log",
                "uid": "62452****",
                "vid": "6.1.0_gf_pc",
                "asb": "1206427",
                "time": "22-11-3 11:49:33",
                "operatingSystem": "Windows 10  (10.0.0) 64bit",
                "deviceModel": "System Product Name (System manufacturer)",
                "graphicsDeviceName": "NVIDIA GeForce GTX 1650",
                "graphicsDeviceType": "Direct3D11",
                "graphicsDeviceVendor": "NVIDIA",
                "graphicsDeviceVersion": "Direct3D 11.0 [level 11.1]",
                "graphicsMemorySize": "3962",
                "systemMemorySize": "8127",
                "processorCount": "6",
                "processorFrequency": "3000",
                "processorType": "Intel(R) Core(TM) i5-9500F CPU @ 3.00GHz",
                "deviceID": "96da5902a042a5f84118995f88373f73650e76be166589726****",
                "guessUID": "62452****",
                "networkReachability": "wifi",
                "msg": "GetAuthkeyRsp",
            }
    • Example 3: Set ignore_multi_fields to True. Only the first value for each parameter is returned.
      • Raw log
        content:logType=net_log&uid=62452****&x=1&x=2&x=3&asb=123&asb=456
      • Transformation rule
        e_set("url",url_parse_qs(v("content")))
      • Result
        content:logType=net_log&uid=62452****&x=1&x=2&x=3&asb=123&asb=456
        url:{
            "logType": "net_log",
            "uid": "62452****",
            "x": "1",
            "asb": "123"
        }
    • Example 4: Set ignore_multi_fields to False. All values for each parameter are returned.
      • Raw log
        content:logType=net_log&uid=62452****&x=1&x=2&x=3&asb=123&asb=456
      • Transformation rule
        e_set("url",url_parse_qs(v("content"),ignore_multi_fields=False))
      • Result
        content:logType=net_log&uid=62452****&x=1&x=2&x=3&asb=123&asb=456
        url:{
            "logType": ["net_log"],
              "uid": ["62452****"],
              "x": ["1", "2", "3"],
              "asb": ["123", "456"],
        }