Event processing functions - Data Transformation| Alibaba Cloud Documentation Center

Functions


Type	Function	Description
Event processing	e_drop	Discards an event based on a condition.
Event processing	e_keep	Retains an event based on a condition.
Event splitting	e_split	Splits an event to multiple events based on the value of a field. You can also use JMESPath to extract the value of the field before splitting the event.
Event output	e_output or e_coutput	Writes an event to a specified target and configures information such as the topic, source, and tags. e_output: deletes the original event after it is written to the target. e_coutput: retains the original event after it is written to the target. The original event continues to be processed in subsequent steps.

e_drop

Syntax
```
e_drop(condition=True)
```
You can use the identifier DROP, which is equivalent to e_drop().

Parameters


Parameter	Type	Required?	Description
condition	Boolean	No	Default value: True. This parameter is typically set to a condition-based judgment function.

Response
If the condition is met, the event is deleted and None is returned. Otherwise, the event is retained and returned.
Examples
- Example 1: If the value of the __programe__ field is access, the log is deleted. Otherwise, the log is retained.
  Raw logs:
```
__programe__: access
age:  18
content:  123
name:  maki
```
```
__programe__: error
age:  18
content:  123
name:  maki
```
  Processing rule:
```
e_if(e_search("__programe__==access"), DROP)
```
  Processing result:
```
__programe__: error
age:  18
content:  123
name:  maki
```
- Example 2: Call the e_drop function. The condition is True, and the log is deleted.
  Raw log:
```
k1: v1
k2: v2
k3: k1
```
  Processing rule:
```
e_drop(e_search("k1==v1"))
```
  Processing result:
```
# The log is deleted because the condition is True.
```
- Example 3: Call the e_drop function. The condition is False, and the log is retained.
  Raw log:
```
k1: v1
k2: v2
k3: k1
```
  Processing rule:
```
e_drop(e_search("not k1==v1"))
```
  Processing result:
```
k1: v1
k2: v2
k3: k1
```
- Example 4: Call the e_drop function without specifying a condition. In this case, the default condition is used, which is True.
  Raw log:
```
k1: v1
k2: v2
k3: k1
```
  Processing rule:
```
e_drop()    # Delete all logs.
```
  Processing result:
```
# The log is deleted.
```

e_keep

Both the e_keep and e_drop functions can discard events. The e_keep function discards an event when the condition is not met, while the e_drop function discards an event when the condition is met.

# The following four functions are equivalent.
e_if_else(e_search("f1==v1"), KEEP, DROP)
e_if_else(e_search("not f1==v1"), DROP) 
e_keep(e_search("f1==v1"))
e_drop(e_search("not f1==v1"))

# The following code is useless.
e_if(e_search("..."), KEEP)    # It is useless to retain a log after searching for it.
e_keep()

Syntax
```
e_keep(condition=True)
```
You can use the identifier KEEP, which is equivalent to e_keep().

Parameters


Parameter	Type	Required?	Description
condition	Boolean	No	Default value: True. This parameter is typically set to a condition-based judgment function.

Response
If the condition is met, the event is retained and returned. Otherwise, the event is deleted and None is returned.
Examples
- Example 1: If the value of the __programe__ field is access, the log is retained. Otherwise, the log is deleted.
  Raw logs:
```
__programe__: access
age:  18
content:  123
name:  maki
```
```
__programe__: error
age:  18
content:  123
name:  maki
```
  Processing rule:
```
e_keep(e_search("__programe__==access"))
# Equivalent to:
e_if(e_search("not __programe__==access"), DROP) 
# Equivalent to:
e_if_else(e_search("__programe__==access"), KEEP, DROP)  
```
  Processing result: The log in which the value of the __programe__ field isaccess is retained.
```
__programe__: access
age:  18
content:  123
name:  maki
```
- Example 2: Call the e_keep function. The condition is True, and the log is retained.
  Raw log:
```
k1: v1
k2: v2
k3: k1
```
  Processing rule:
```
e_keep(e_search("k1==v1"))
```
  Processing result:
```
k1: v1
k2: v2
k3: k1
```
- Example 3: Call the e_keep function. The condition is False, and the log is deleted.
  Raw log:
```
k1: v1
k2: v2
k3: k1
```
  Processing rule:
```
e_keep(e_search("not k1==v1"))
```
  Processing result:
```
# The log is deleted because the condition is False.
```
- Example 4: Call the e_keep function by directly setting the condition to False.
  Raw log:
```
k1: v1
k2: v2
k3: k1
```
  Processing rule:
```
e_keep(False)
```
  Processing result:
```
# The log is deleted.
```

e_split

Syntax

e_split(Field name, sep=',', quote='"', lstrip=True, jmes=None, output=None)

Splitting rules
1. If the jmes parameter is set, the system converts the value of the field to a JSON list, and uses the JMESPath expression to extract values from the JSON list, which will be used in the next step. If the jmes parameter is not set, the system directly uses the value of the field in the next step.
2. If the value obtained from the previous step is a list or a string that represents a JSON list, the system splits the event based on this value. Otherwise, the system parses the value to multiple delimited values based on the sep, quote, and lstrip parameters. Then, the system splits the event based on these values.

Parameters


Parameter	Type	Required?	Description
Field name	String	Yes	The name of the field used to split the event. For more information about how to set special field names, see Event structure and fields.
sep	String	No	The delimiter used to separate values.
quote	String	No	The character used to enclose a value.
lstrip	String	No	Specifies whether to trim the leading space characters from each value. Default value: True.
jmes	String	No	The JMESPath string used to convert the value of the field to a JSON object and extract values from the JSON object.
output	String	No	The new field name, which overwrites the existing field name by default.

Response
A log list is returned. The values of fields in the list are all those from the source log.

Examples

Raw log:

__topic__:   
age:  18
content:  123
name:  maki

__topic__:   
age:  18
content:  123
name:  maki

Processing rule:

e_set("__topic__", "V_SENT,V_RECV,A_SENT,A_RECV")
e_split("__topic__")

Processing result:

__topic__:  A_SENT
age:  18
content:  123
name:  maki

__topic__:  V_RECV
age:  18
content:  123
name:  maki

...

e_output or e_coutput

Syntax

e_output(name=None, project=None, logstore=None, topic=None, source=None, tags=None)
e_coutput(name=None, project=None, logstore=None, topic=None, source=None, tags=None)

Parameters


Parameter	Type	Required?	Description
name	String	No	The name of the target that you set when saving a data processing task in the Log Service console. Default value: None, specifying that the first target configured for the data processing task is used.
project	String	No	The existing project to which the log is written. If you do not set this parameter, the log is written to the project configured for the data processing task by default.
logstore	String	No	The existing Logstore to which the log is written. If you do not set this parameter, the log is written to the Logstore configured for the data processing task by default.
topic	String	No	The new topic of the log.
source	String	No	The new source of the log.
tags	Dict	No	The new tags of the log, in the dictionary format. Note You do not need to prefix keywords with `__tag__:`.

Response
- e_output: The original log is deleted after it is written to the target.
- e_coutput: The original log is retained after it is written to the target. The original log continues to be processed in subsequent steps.
Note
- By default, all the logs that are not deleted at last are written to the first target configured for the data processing task. If only one target is configured for the data processing task, you do not need to call the e_output function.
- The first target configured for the data processing task is used by default. If you call the e_output function when only one target is configured, we recommend that you do not set the name parameter.
Examples
Raw log:
```
__topic__:  
k1: v1
k2: v2
x1: v3
x5: v4
```
Processing rule:
```
e_if(e_match("k2", r"\w+"), e_output(name="target2", source="source1", topic="topic1"))
```
Processing result:
```
__topic__:  topic1
k1: v1
k2: v2
x1: v3
x5: v4
```
During preview, the log is not written to the destination Logstore, but to the Logstore named internal-etl-log.

Note The internal-etl-log Logstore is a dedicated Logstore created by the system in the current project when you preview a data processing task for the first time. You cannot modify the configuration of this Logstore or write other data to this Logstore. It is free of charge.