Simple Log Service:Field operation functions

You can call the v function to retrieve the value of a specific log field. If you specify multiple field names, the function returns the value of the first field that exists in the log.

• Syntax

  v(key, ..., default=None)

• Parameters

  Parameter	Parameter Type	Required	Description
  key	String	Yes	Enter a name for the field.
  default	Any	No	If the specified field name does not exist, the function returns the value of `default`. The default value is None.

• Response

  Returns the value of the first existing field in the log. If none of the specified fields exist, the function returns the value of the default parameter.

• Examples

  Assign the value of the `content` field to the `test_content` field.

  • Raw log

    content: hello

  • Transformation rule

    e_set("test_content", v("content"))

  • Result

    content: hello
    test_content: hello

• References

  This function can be used with other functions. For more information, see Cleanse data by calling functions.

You can call the e_set function to add a new field or set a new value for an existing field.

• Syntax

  e_set(key1, value1, key2, value2, mode="overwrite")

  Important

  • The key1 and value1 parameters must be specified in pairs.

  • When you use the e_set function to set a time field, such as F_TIME or __time__, the value must be a numeric string.

    e_set(F_TIME, "abc")   # Incorrect
    e_set(F_TIME, "12345678")   # Correct

• Parameters

  Parameter	Parameter type	Required	Description
  key	String	Yes	The name of the destination field. You can also obtain the field name from a string expression. For more information about how to set special field names, see Event types.
  value	Any	Yes	The new field value. Values that are not strings are converted to strings and added to the log. Tuples, lists, and dictionaries are converted to JSON object strings. For more information about string conversion, see Automatic type conversion during assignment. Note If the passed value is None, no update is performed.
  mode	String	No	The field overwrite mode. The default value is overwrite. For more information, see Field extraction check and overwrite modes.

• Response

  Returns the updated log.

• Examples

  • Example 1: Set a static value for a field.

    Add a new field named city and set its value to Shanghai.

    e_set("city", "Shanghai")

  • Example 2: Copy a field value.

    You can call a single expression function to assign the value of the existing ret field to the new result field.

    e_set("result", v("ret"))

  • Example 3: Set a dynamic value.

    You can call a composite expression function to retrieve the value of the first existing field, convert it to lowercase, and assign it to the result field.

    e_set("result", str_lower(v("ret", "return")))

  • Example 4: Set a field value multiple times.

    1. Set the value of the event_type field.

       e_set("event_type", "login event", "event_info", "login host")

    2. If the value of the ret field is fail, set the value of the event_type field to login failed event.

       e_if(e_search('ret==fail'), e_set("event_type", "login failed event" ))

  • References

    This function can be used with other functions. For more information, see Cleanse data by calling functions.

You can call the e_drop_fields function to delete log fields that meet specified conditions.

• Syntax

  e_drop_fields(key1, key2, ....,regex=False)

• Parameters

  Parameter	Parameter type	Required	Description
  key	String	Yes	The log field name. It can be a regular expression. If the field name fully matches the condition, the field is deleted. Fields that do not match are retained. For more information about regular expressions, see Regular expressions. You must configure at least one log field.
  regex	Boolean	No	If this parameter is set to False, regular expressions are not used for matching. If this parameter is not configured, the default value is True.

• Response

  Returns the log that was deleted.

• Examples

  If the value of the content field is 123, delete the content and age fields.

  • Raw log

    age: 18
    content: 123
    name: twiss

  • Transformation rule

    e_if(e_search("content==123"), e_drop_fields("content", "age",regex=True))

  • Result

    name: twiss

• References

  This function can be used with other functions. For more information, see Transform complex JSON data.

You can call the e_keep_fields function to retain log fields that meet specified conditions.

• Syntax

  e_keep_fields(key1, key2, ....,regex=False)

• Parameters

  Parameter	Parameter type	Required	Description
  key	String	Yes	The log field name. It can be a regular expression. If the field name fully matches the condition, the field is retained. Fields that do not match are deleted. You must configure at least one field.
  regex	Boolean	No	If this parameter is set to False, regular expressions are not used for matching. If this parameter is not configured, the default value is True.

• Response

  The retained logs are returned.

• Examples

  If the value of the content field is 123, retain the content and age fields.

  • Raw log

    age: 18
    content: 123
    name: twiss

  • Transformation rule

    e_if(e_search("content==123"), e_keep_fields("content", "age"))

  • Result

    age: 18
    content: 123

You can call the e_pack_fields function to pack log fields and output them to a new field.

• Syntax

  e_pack_fields(output_fields,include=".*",exclude=None,drop_packed=True)

• Parameters

  Parameter	Type	Required	Description
  output_field	String	Yes	The name of the output field after packaging. Its value is in JSON format.
  include	String	No	The whitelist configuration. Fields that match the regular expression are packed. The default value is ".*", which indicates that all fields are matched. For more information, see Regular expressions.
  exclude	String	No	The blacklist configuration. Fields that match the regular expression are not packed. The default value is None, which indicates that no matching is performed. For more information, see Regular expressions.
  drop_packed	Boolean	No	Specifies whether to delete the original packed data. The default value is True. True (default): The original packed data is deleted from the output. False: The original packed data is not deleted from the output.

• Response

  Returns the packed log data.

• Examples

  • Example 1: Pack all log fields into the test field. By default, the original packed fields are deleted.

    • Raw log

      test1:123
      test2:456
      test3:789

    • Transformation rule

      e_pack_fields("test")

    • Result

      test:{"test1": "123", "test2": "456", "test3": "789"}

  • Example 2: Pack all log fields into the test field. The original packed fields are not deleted.

    • Raw log

      test1:123
      test2:456
      test3:789

    • Transformation rule

      e_pack_fields("test",drop_packed=False)

    • Result

      test:{"test1": "123", "test2": "456", "test3": "789"}
      test1:123
      test2:456
      test3:789

  • Example 3: Pack the test and abcd fields into the content field. The original packed fields are not deleted.

    • Raw log

      abcd@#%:123
      test:456
      abcd:789

    • Transformation rule

      e_pack_fields("content", include="\w+", drop_packed=False)

    • Result

      abcd:789
      abcd@#%:123
      content:{"test": "456", "abcd": "789"}
      test:456

  • Example 4: Pack all fields except test and abcd into the content field. The original packed fields are deleted.

    • Raw log

      abcd@#%:123
      test:456
      abcd:789

    • Transformation rule

      e_pack_fields("content", exclude="\w+", drop_packed=True)

    • Result

      abcd:789
      content:{"abcd@#%": "123"}
      test:456

You can call the e_rename function to rename log fields that meet specified conditions.

• Syntax

  e_rename("key1", "new key1", "key2", "new key2", ..., regex=False)

  Note

  The key and new key parameters must be specified in pairs. If the new key already exists in the source log, no operation is performed.

• Parameters

  Parameter	Parameter type	Required	Description
  key	String	Yes	The log field name. It can be a regular expression. If the field name fully matches the condition, the field is renamed. For more information about regular expressions, see Regular expressions. You must configure at least one field.
  new key	String	Yes	The new name of the field.
  regex	Boolean	No	If this parameter is set to False, regular expressions are not used for matching. If this parameter is not configured, the default value is True.

• Response

  The renamed field is returned.

• Examples

  • Example 1: Rename the host field to client_host.

    • Raw log

      host: 1006

    • Transformation rule

      e_rename("host","client_host")

    • Result

      client_host: 1006

  • Example 2: If the field does not exist, it is not renamed.

    • Raw log

      host: 1006

    • Transformation rule

      e_rename("url","rename_url")

    • Result

      host: 1006

  • References

    This function can be used with other functions. For more information, see Cleanse data by calling functions.