All Products
Search
Document Center

Preset split

Last Updated: Apr 28, 2018

This topic describes the preset splitters of ARMS.

More types of splitters, such as single separator, multi-separator, sequence, KV, and JSON splitters, are shipped with ARMS. In different cases, they can be used separately or in combination.

Single separator splitter

Description

A single string serves as the input. The log is splitted into multiple key-value (KV) pairs with user-specified separator.

Example

The sample log is as follows:

  1. 2016-11-08 11:00:01|user_abc|123456|PlaceOrder
  2. 2016-11-08 11:00:02|user_abc|123456|Payment
  3. 2016-11-08 11:10:01|user_abc|123456|Return

The custom splitting is as follows:

Preset Splitter: Single Separator

Configuration description:

  • “|” serves as the separator.
  • After splitting, the string at location 0 is converted into a Date value in the format of yyyy-MM-dd HH:mm:ss (the conversion result is actually saved as a Long value on the ARMS console).
  • The string at location 1 is converted into a String value whose key is username.
  • The string at location 2 is converted into a Long value whose key is userid.
  • The string at location 3 is saved as a String value whose key is event.

Note:

  • Support for non-printable characters: The single separator splitter supports the splitting of non-printable characters, for example, \t and \u001f (ascii char=31). The splitting of some non-printable characters cannot be correctly validated by page test, because the non-printable characters cannot be correctly entered to the textboxes on the page. Therefore, We recommend that you use printable characters for debugging on the test page, and then replace the printable characters with the non-printable characters for the debugging result saving and deployment.

  • Multi-character separator: The single separator splitter also supports multi-character separators. For example, if a::b::c serves as the input, :: is used as the separator.

  • The splitting model contains the _line and _hostIp fields. For more information, see System fields in log cleansing.

Multi-separator splitter

Description

A single string serves as the input. The log is splitted into multiple key-value (KV) pairs with user-specified separator.

Example

The sample log is as follows:

  1. 2014-07-25 17:25:00,aaa|b~1
  2. 2014-07-25 17:26:00,aaa|b~1

The custom splitting is as follows:

Preset Splitter: Multiple Separator

The splitting of each line of log is as follows:

Preset Splitter: Multiple Separator Result

The splitting model is as follows:

Preset Splitter: Multiple Separator Model

Sequential separator splitter

Description

In the complicated splitting cases, for example, in the accesslog splitting case, ARMS provides the sequential separator splitter to split the logs according to the complexity of O(n).

Example

The sample log is as follows:

  1. 117.74.77.48 835158 - [26 May 2014:14:05:28 +0800] "GET http://trade.taobao.com/trade/detail/trade_snap.htm?spm=a1z0f.2.100003.9.8BpicQ&trade_id=664793864233811" 302 0
  2. 117.74.87.58 835158 - [26 May 2014:14:05:28 +0800] "GET http://trade.taobao.com/trade/detail/trade_snap.htm?spm=a1z0f.2.100003.9.8BpicQ&trade_id=664793864233811" 302 0

You may want to get such fields as IP address, average response time, time, and request URL:

Then, you may also find that:

  • The single separator splitter cannot meet the needs obviously.

  • The multi-separator splitter cannot meet the needs either: If you want to get the KV pair rt=835158, you must take “ “ (space) as one of the separators. But if you do this, you split the string 26 May 2014:14:05:28 +0800 into several sections too, which is not what you want. Then, the value date - 1401084328000 cannot be obtained.

  • If you use the regular expression to match a KV, the efficiency is too low because it’s required to traverse for many times.

ARMS provides the “sequential separator splitter”. You can get the correct result as shown in this example with the following configuration. The detailed configuration mode is as follows:

Preset Splitter: Sequential Separator

Define five separator strings, cut the entered strings into six substrings in turn, and assign the specified substrings to different keys, respectively. The target splitting effect can be achieved.

KV splitter

As shown in the following figure, if you want to split the logs into multiple groups of Key=Value (KV pairs):

  1. key1=aaaa;key2=bbbb;key3=cccc;key4=dddd;....
  2. key1=aaaa;key2=bbbb;key3=cccc;key4=dddd;....

And the location of each KV pair is not fixed, then you can use the KV splitter:

The splitting configuration is as follows:

Preset Splitter: KV 1

Note:

  • If the text character<-> key mapping is not configured, the KV splitter regards all the keys as “unspecified keys”, and the values corresponding to all the keys are split as the StringKey values.

  • If Whether to split undefined key is cleared, the undefined keys are not split. In other words, if the text character <-> key mapping is not configured and Whether to split undefined Key is cleared, the KV splitter does not perform any operation.

The following example shows how a specified key is converted into the corresponding value. Assuming that the user log is as follows:

  1. name=abc;item=iphone6;quantity=15;date=2014-07-25 17:25:00;....
  2. name=abc;item=iphone6;quantity=15;date=2014-07-25 17:25:00;....

In the preceding log, quantity is converted into LongKey and date is converted into DateKey:

Preset Splitter: KV 2

JSON splitter

If the log contains a JSON string, you can use the JSON splitter to quickly split the “leaf node” of the JSON string.

Example 1: Automatic splitting

  1. {
  2. "title": "Example Schema",
  3. "type": "object",
  4. "quantity": 200,
  5. "date":"2015-12-12 12:12:12",
  6. "properties": {
  7. "firstName": {
  8. "type": "string"
  9. },
  10. "lastName": {
  11. "type": "string"
  12. },
  13. "age": {
  14. "description": "Age in years",
  15. "type": "integer",
  16. "minimum": 0
  17. }
  18. },
  19. "required": ["firstName", "lastName"]
  20. }

Place the JSON string into a line of text and split it with the JSON splitter.

Preset Splitter: Json 1

The splitting result is as follows:

Preset Splitter: Json 1 Result

Because Whether to split undefined key is enabled, the JSON splitter automatically splits all “Non-Array leaf nodes” and converts them into corresponding Keys (StringKey or LongKey).

NOTE: The required field in this example is an Array field, so it is not split. In addition, the date field is converted into a String field.

Example 2: Custom splitting

To split only specified fields of the JSON string, configure as follows:

Preset Splitter: Json 2

The splitting model is as follows:

Preset Splitter: Json 2 Result

In this example, Whether to split undefined key is cleared, the title, date, and quantity fields are customized, and types are specified. ARMS only splits the custom fields, and converts them into corresponding types (For example, the field date in this example is converted into the Long field preset in ARMS).