This topic describes the syntax of event processing functions and provides parameter description and function examples.

Functions

Type Function Description
Event processing e_drop Discards an event based on a condition.
e_keep Retains an event based on a condition.
Event splitting e_split Splits an event to multiple events based on the value of a field. You can also use JMESPath to extract the value of the field before splitting the event.
Event output e_output or e_coutput Writes an event to a specified target and configures information such as the topic, source, and tags.
  • e_output: deletes the original event after it is written to the target.
  • e_coutput: retains the original event after it is written to the target. The original event continues to be processed in subsequent steps.

e_drop

  • Syntax
    e_drop(condition=True)
    You can use the identifier DROP, which is equivalent to e_drop().
  • Parameters
    Parameter Type Required? Description
    condition Boolean No Default value: True. This parameter is typically set to a condition-based judgment function.
  • Response

    If the condition is met, the event is deleted and None is returned. Otherwise, the event is retained and returned.

  • Examples
    • Example 1: If the value of the __programe__ field is access, the log is deleted. Otherwise, the log is retained.
      Raw logs:
      __programe__: access
      age:  18
      content:  123
      name:  maki
      __programe__: error
      age:  18
      content:  123
      name:  maki
      Processing rule:
      e_if(e_search("__programe__==access"), DROP)
      Processing result:
      __programe__: error
      age:  18
      content:  123
      name:  maki
    • Example 2: Call the e_drop function. The condition is True, and the log is deleted.
      Raw log:
      k1: v1
      k2: v2
      k3: k1
      Processing rule:
      e_drop(e_search("k1==v1"))
      Processing result:
      # The log is deleted because the condition is True.
    • Example 3: Call the e_drop function. The condition is False, and the log is retained.
      Raw log:
      k1: v1
      k2: v2
      k3: k1
      Processing rule:
      e_drop(e_search("not k1==v1"))
      Processing result:
      k1: v1
      k2: v2
      k3: k1
    • Example 4: Call the e_drop function without specifying a condition. In this case, the default condition is used, which is True.
      Raw log:
      k1: v1
      k2: v2
      k3: k1
      Processing rule:
      e_drop()    # Delete all logs.
      Processing result:
      # The log is deleted.

e_keep

Both the e_keep and e_drop functions can discard events. The e_keep function discards an event when the condition is not met, while the e_drop function discards an event when the condition is met.
# The following four functions are equivalent.
e_if_else(e_search("f1==v1"), KEEP, DROP)
e_if_else(e_search("not f1==v1"), DROP) 
e_keep(e_search("f1==v1"))
e_drop(e_search("not f1==v1"))

# The following code is useless.
e_if(e_search("..."), KEEP)    # It is useless to retain a log after searching for it.
e_keep()
  • Syntax
    e_keep(condition=True)
    You can use the identifier KEEP, which is equivalent to e_keep().
  • Parameters
    Parameter Type Required? Description
    condition Boolean No Default value: True. This parameter is typically set to a condition-based judgment function.
  • Response

    If the condition is met, the event is retained and returned. Otherwise, the event is deleted and None is returned.

  • Examples
    • Example 1: If the value of the __programe__ field is access, the log is retained. Otherwise, the log is deleted.
      Raw logs:
      __programe__: access
      age:  18
      content:  123
      name:  maki
      __programe__: error
      age:  18
      content:  123
      name:  maki
      Processing rule:
      e_keep(e_search("__programe__==access"))
      # Equivalent to:
      e_if(e_search("not __programe__==access"), DROP) 
      # Equivalent to:
      e_if_else(e_search("__programe__==access"), KEEP, DROP)  
      Processing result: The log in which the value of the __programe__ field isaccess is retained.
      __programe__: access
      age:  18
      content:  123
      name:  maki
    • Example 2: Call the e_keep function. The condition is True, and the log is retained.
      Raw log:
      k1: v1
      k2: v2
      k3: k1
      Processing rule:
      e_keep(e_search("k1==v1"))
      Processing result:
      k1: v1
      k2: v2
      k3: k1
    • Example 3: Call the e_keep function. The condition is False, and the log is deleted.
      Raw log:
      k1: v1
      k2: v2
      k3: k1
      Processing rule:
      e_keep(e_search("not k1==v1"))
      Processing result:
      # The log is deleted because the condition is False.
    • Example 4: Call the e_keep function by directly setting the condition to False.
      Raw log:
      k1: v1
      k2: v2
      k3: k1
      Processing rule:
      e_keep(False)
      Processing result:
      # The log is deleted.

e_split

  • Syntax
    e_split(Field name, sep=',', quote='"', lstrip=True, jmes=None, output=None)
  • Splitting rules
    1. If the jmes parameter is set, the system converts the value of the field to a JSON list, and uses the JMESPath expression to extract values from the JSON list, which will be used in the next step. If the jmes parameter is not set, the system directly uses the value of the field in the next step.
    2. If the value obtained from the previous step is a list or a string that represents a JSON list, the system splits the event based on this value. Otherwise, the system parses the value to multiple delimited values based on the sep, quote, and lstrip parameters. Then, the system splits the event based on these values.
  • Parameters
    Parameter Type Required? Description
    Field name String Yes The name of the field used to split the event. For more information about how to set special field names, see Event structure and fields.
    sep String No The delimiter used to separate values.
    quote String No The character used to enclose a value.
    lstrip String No Specifies whether to trim the leading space characters from each value. Default value: True.
    jmes String No The JMESPath string used to convert the value of the field to a JSON object and extract values from the JSON object.
    output String No The new field name, which overwrites the existing field name by default.
  • Response

    A log list is returned. The values of fields in the list are all those from the source log.

  • Examples
    Raw log:
    __topic__:   
    age:  18
    content:  123
    name:  maki
    
    __topic__:   
    age:  18
    content:  123
    name:  maki
    Processing rule:
    e_set("__topic__", "V_SENT,V_RECV,A_SENT,A_RECV")
    e_split("__topic__")
    Processing result:
    __topic__:  A_SENT
    age:  18
    content:  123
    name:  maki
    
    __topic__:  V_RECV
    age:  18
    content:  123
    name:  maki
    
    ...

e_output or e_coutput

  • Syntax
    e_output(name=None, project=None, logstore=None, topic=None, source=None, tags=None)
    e_coutput(name=None, project=None, logstore=None, topic=None, source=None, tags=None)
  • Parameters
    Parameter Type Required? Description
    name String No The name of the target that you set when saving a data processing task in the Log Service console. Default value: None, specifying that the first target configured for the data processing task is used.
    project String No The existing project to which the log is written. If you do not set this parameter, the log is written to the project configured for the data processing task by default.
    logstore String No The existing Logstore to which the log is written. If you do not set this parameter, the log is written to the Logstore configured for the data processing task by default.
    topic String No The new topic of the log.
    source String No The new source of the log.
    tags Dict No The new tags of the log, in the dictionary format.
    Note You do not need to prefix keywords with __tag__:.
  • Response
    • e_output: The original log is deleted after it is written to the target.
    • e_coutput: The original log is retained after it is written to the target. The original log continues to be processed in subsequent steps.
    Note
    • By default, all the logs that are not deleted at last are written to the first target configured for the data processing task. If only one target is configured for the data processing task, you do not need to call the e_output function.
    • The first target configured for the data processing task is used by default. If you call the e_output function when only one target is configured, we recommend that you do not set the name parameter.
  • Examples
    Raw log:
    __topic__:  
    k1: v1
    k2: v2
    x1: v3
    x5: v4
    Processing rule:
    e_if(e_match("k2", r"\w+"), e_output(name="target2", source="source1", topic="topic1"))
    Processing result:
    __topic__:  topic1
    k1: v1
    k2: v2
    x1: v3
    x5: v4
    During preview, the log is not written to the destination Logstore, but to the Logstore named internal-etl-log.
    Note The internal-etl-log Logstore is a dedicated Logstore created by the system in the current project when you preview a data processing task for the first time. You cannot modify the configuration of this Logstore or write other data to this Logstore. It is free of charge.