edit-icon download-icon

Use custom splitting

Last Updated: Apr 28, 2018

Besides the intelligent splitter, ARMS also provides a custom cleansing configurator, which is used for creating your own log cleansing flow to meet complicated splitting requirement.

The custom cleansing configurator works in the what you see is what you get (WYSIWYG) visual configuration mode. This topic explains how to use the custom cleansing configurator.

User interface of the configurator

The user interface of the custom cleansing configurator consists of three areas:

  • A: Raw log area
  • B: Cleansing flow editing area
  • C: Cleansing result preview area

interface

In the cleansing flow editing area (“editing area”), one or more “blocks” are assembled into the log cleansing logic, which “cleanses” the logs in the raw log area into structural key-value (KV) pairs.

While you edit a cleansing flow in the editing area, you can click Log Splitting Preview on the top of the cleansing result preview area (preview area) to preview the KV pairs generated after each line of the raw log is cleansed with the existing cleansing flow.

If the cleansing flow and previewed result are correct, click Next to start the job. ARMS processes each log entry consumed from the data source with the flow saved in the editing area.

Example 1

This example shows how to configure a simple log cleansing flow.

Assuming that the transaction log of a system is as follows:

  1. 2016-11-08 11:00:01|user_abc|123456|PlaceOrder
  2. 2016-11-08 11:00:02|user_abc|123456|Payment
  3. 2016-11-08 11:10:01|user_abc|123456|Return

Take the first line of the log as an example. Assuming that the log entry is to be split with the following KV pairs:

Key type Key Value
Date date 2016-11-08 11:00:01
String username user_abc
Long userid 123456
String event PlaceOrder

The KV pairs are separated by vertical bars (|), so we use the single separator splitter to split the KV pairs.

  1. On the left-side tool bar of the editing area, drag and drop the single separator splitter to the editing area, as shown in the following figure.

    NOTE: The splitter must be attached to the Start Cleansing module.

    Start Cleansing

    About splitting configuration:

    • Take “|” as the separator and “_line” as the input parameter (_line indicates the raw log line).
    • After splitting, the string at location 0 is converted into a Date value in the format of yyyy-MM-dd HH:mm:ss (the conversion result is actually saved as a Long value on the ARMS console).
    • The string at location 1 is converted into a String value whose key is username.
    • The string at location 2 is converted into a Long value whose key is userid.
    • The string at location 3 is converted into a String value whose key is event.
  2. Click Log Splitting Preview. The splitting result of each log is as follows:

    Log Splitting Preview

  3. Click Save or Next. The cleansing flow takes effect.

This is how a simple log splitting job works.

Example 2

This example shows how to split and cleanse a complicated log.

  1. 2016-11-08 11:00:01|PlaceOrder|user_abc|123456
  2. 2016-11-08 11:00:02|Payment|200
  3. 2016-11-08 11:10:01|Return

In this example, for different logs, the number of fields and the meanings of fields in the same column vary with operation types. This is common in the business systems. Therefore, logs of different types can be distinguished by the if-then or if-else logic. For more information about the syntax, see Aviator.

Logic Splitting

About splitting configuration:

  • Use the single separator splitter to split the fields date and event first.
  • Then, use the logic module if-then. If the values of event are different, use different single separator splitters to split the fields.

In the preview area, you can find that the KV pairs split from the first line are different from those split from the second line. The KV pairs split from the first line contain username and userid, but those split from the second line contain price.

Logic Splitting Result 1

Logic Splitting Result 2

The preceding two examples only used the single separator splitter. More types of splitters, such as single separator, multi-separator, sequence, KV, and JSON splitters, are shipped with ARMS. In different cases, they can be used separately or in combination. For more information, see Preset splitters.

Thank you! We've received your feedback.