You need to select different functions based on the actual scenarios when setting data transformation rules.

For more information about how to use the LOG DSL functions, see Syntax introduction.

Use case 1: use of e_keep and KEEP

By default, event logs that are not processed are retained. Therefore, KEEP is used only in specific scenarios. You can call e_drop or e_drop(e_search()) to discard logs. Alternatively, you can call e_if or e_if_else together with DROP to discard logs. The relevant functions are as follows:
e_keep(e_search(...) )    # Retain the data if it meets the requirements. Otherwise, discard the data.
e_drop(e_search(...) )    # Discard the data if it meets the requirements. Otherwise, retain the data.
e_if(e_search("..."), KEEP)    # This line of code does not make sense. The data that meets the requirements will be retained.
e_if_else(e_search("..."), KEEP, DROP)    # This line of code makes sense.
e_if(e_search("not ..."), DROP)        # This line of code makes sense.

Use case 2: use of the functions

  • 1. Assign a value to a field if the field does not exist or is empty
    • Best practices
      e_set("result", ".....value...", mode="fill")
    • Non-best practices
      e_if(op_not(v("result")), e_set("result", ".....value..."))
    • For more information about how to extract and overwrite a field, see Field check and overwrite modes.
  • 2. Use the Grok function to simplify regular expressions
    • Transformation request

      Extract an IP address from the content field.

    • Raw log
      content:"ip address: 192.168.1.1"
    • Target
      192.168.1.1
    • Best practices
      e_regex("content", grok(r"\w+: (%{IP})"), "addr")
      # Or
      e_regex("content", grok(r"\w+: (%{IP:addr})"))
    • Non-best practices
      e_regex("content", grok(r"\w+: (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"), "addr")
  • 3. Assign values to multiple fields
    • Best practices
      e_set("k1", "v1", "k2", "v2", "k3", "v3", ....)
    • Non-best practices
      e_set("k1", "v1")
      e_set("k2", "v2")
      e_set("k3", "v3")
      ...

Use case 3: use of e_compose to reduce repeated judgments

  • Transformation request

    Assume that the value of the content field is 123. Delete the age and name fields, and then rename content as ctx.

  • Raw log
    content: 123
    age: 23
    name: twiss
  • Transformation result
    ctx: 123
  • Best practices
    We recommend that you call e_compose to combine the operations that are performed by using the same logic.
    e_if(e_search("content==123"), 
       e_compose(e_drop_fields("age|name"), e_rename("content", "ctx")))
  • Non-best practices
    The following code can incur multiple judgments, which is less efficient.
    e_if(e_search("content==123"), e_drop_fields("age|name"))
    e_if(e_search("content==123"), e_rename("content", "ctx"))

Use case 4: field type conversion of expression functions

The fields and values in event logs are always passed to functions as strings. Data of a non-string type is automatically converted to data of the string type. Therefore, you need to pay attention to the types of fields that a function can receive when calling the function. For more information about the types of fields that can be received by each function, see Syntax introduction.
  • Example 1
    The op_add function can receive both data of the string type and data of the numeric type. Therefore, no field type conversion is required.
    e_set("a", 1)
    e_set("b", 2)
    
    op_add(v("a"), v("b"))    # Valid. The return value is 12.
    op_add(ct_int(v("a")), ct_int(v("b")))    # Valid. The return value is 3.
  • Example 2
    The op_mul function can receive only a value of the numeric type for its second field. Therefore, you must convert a value of the string type to a value of the numeric type for the second field.
    e_set("a", 2)
    e_set("b", 5)
    
    op_mul(v("a"), v("b"))    # Invalid.
    op_mul(ct_int(v("a")), ct_int(v("b")))    # Valid. The return value is 10.
    op_mul(v("a"), ct_int(v("b")))    # Valid. The return value is 22222.
    When the field values in event logs are passed to functions, they are automatically converted to the values of the string type. Therefore, v("a") and v("b") are both of the string type. However, the second field of op_mul can receive only numeric values. In this case, you must call ct_int to convert a value of the string type to a value of the integer type, and then pass the value to this function.
  • Example 3
    """
    Transformation request: Convert the time indicated by time1 to a UNIX timestamp.
    """
    e_set("time1", "2019-06-03 2:41:26")
    
    e_set("time2", dt_totimestamp(v("time1"))) # Invalid.
    
    e_set("time2", dt_totimestamp(dt_parse(v("time1")))) # Valid.
    e_set("time2", dt_parsetimestamp(v("time1"))) # Valid.
    • The dt_totimestamp function receives only datetime objects, instead of strings. Therefore, you must call dt_parse to convert the string value of time1 to a datetime object.
    • Alternatively, you can directly call dt_parsetimestamp because it can receive both datetime objects and strings.

Use case 5: exception handling for expression functions

Many expression functions have certain requirements for input fields. If input fields do not meet the requirements, an error is thrown or default values are returned. Note that errors may be further thrown when these default values are passed to subsequent functions. Examples:
e_set("data_len": op_len(v("data")))           # Invalid calling.
e_set("data_len": op_len(v("data", default="")))  # Valid calling.
In the first expression, if the data field does not exist, v("data") returns None and an error is thrown. In the second expression, a valid default value is assigned by using the default field. Therefore, no error is thrown for the expression.

Use case 6: differences between e_if and e_switch

e_if syntax
e_if(condition 1, operation 1, condition 2, operation 2, condition 3, operation 3, ...)
Conditions and operations are paired one to one. Conditions are checked in sequence for condition-based judgments. An operation is performed only after its paired condition is met. If a condition is not met, its paired operation will not be performed and the next condition-based judgment is triggered.
e_switch syntax
e_switch(condition 1, operation 1, condition 2, operation 2, condition 3, operation 3, ..., default=None)
Conditions and operations are paired one to one. Conditions are checked in sequence for condition-based judgments. An operation is performed only after its paired condition is met. Once a condition is met, the corresponding operation result is returned and no further condition-based judgment will be triggered. If a condition is not met, its paired operation will not be performed and the next condition-based judgment is triggered. If no conditions are met and the default field is set, the operation specified by default is performed and the default value is returned.
Examples:
  • Raw log
    status1: 200
    status2: 404
  • e_if transformation rule
    e_if(e_match("status1", "200"), e_set("status1_info", "normal"), 
         e_match("status2", "404"), e_set("status2_info", "error"))
  • e_if transformation result
    status1: 200
    status2: 404
    status1_info: normal
    status2_info: error
    The e_if function checks all conditions for condition-based judgments. An operation is performed only after its paired condition is met.
  • e_switch transformation rule
    e_switch(e_match("status1", "200"), e_set("status1_info", "normal"), 
             e_match("status2", "404"), e_set("status2_info", "error"))
  • e_switch transformation result
    status1: 200
    status2: 404
    status1_info: normal
    As long as one condition is met for e_switch, the system returns the operation result without triggering any further condition-based judgment.