This topic describes the syntax of event processing functions and provides parameter description and function examples.
Functions
Type | Function | Description |
---|---|---|
Event processing | e_drop | Drops log entries if the specified condition is met. |
e_keep | Retains log entries if the specified condition is met.
Both the e_keep and e_drop functions drop log entries. The difference is that the
e_keep function drops log entries if the specified condition is not met while the
e_drop function drops log entries if the specified condition is met.
|
|
Event splitting | e_split | Splits a log entry to multiple log entries based on the value of a field. You can specify a JMESPath expression to extract fields from a log entry and then split the log entry into multiple log entries. |
Event output | e_output, e_coutput | Sends log entries to a specified Logstore. You can specify the topic, source, and
tags fields of log entries sent to the specified Logstore.
|
e_drop
- Syntax
e_drop(condition=True)
You can use the identifier DROP to drop a log entry. This identifier is equivalent to the e_drop function.
- Parameters
Parameter Data type Required Description condition Bool No Default value: True. This parameter is typically set to a conditional expression. - Result
If the specified condition is met, log entries are dropped and None is returned. Otherwise, the raw log entries are returned.
- Examples
- Example 1: Evaluate log entries against the __programe__==access expression. If the
value of the __programe__ field in a log entry is access, the log entry is dropped. Otherwise, the log entry is retained.
- Raw log entry
__programe__: access age: 18 content: 123 name: maki __programe__: error age: 18 content: 123 name: maki
- Transformation rule
e_if(e_search("__programe__==access"), DROP)
- Result:
The log entry whose __programe__ field value is access is dropped while the log entry whose __programe__ field value is error is retained.
__programe__: error age: 18 content: 123 name: maki
- Raw log entry
- Example 2: Evaluate a log entry against the k1==v1 expression. If the evaluation result
of the conditional expression is True, the log entry is dropped.
- Raw log entry
k1: v1 k2: v2 k3: k1
- Transformation rule
e_drop(e_search("k1==v1"))
- Result:
This log entry is dropped because the evaluation result of the k1==v1 condition is True.
- Raw log entry
- Example 3: Evaluate a log entry against the not k1==v1 expression. If the evaluation
result of the conditional expression is False, the log entry is retained.
- Raw log entry
k1: v1 k2: v2 k3: k1
- Transformation rule
e_drop(e_search("not k1==v1"))
- Result:
k1: v1 k2: v2 k3: k1
- Raw log entry
- Example 4: Specify no conditional expression in the e_drop function and use the default
value True to drop a log entry.
- Raw log entry
k1: v1 k2: v2 k3: k1
- Transformation rule
e_drop()
- Result:
The log entry is dropped.
- Raw log entry
- Example 1: Evaluate log entries against the __programe__==access expression. If the
value of the __programe__ field in a log entry is access, the log entry is dropped. Otherwise, the log entry is retained.
e_keep
- Syntax
e_keep(condition=True)
You can use the identifier KEEP to retain a log entry. This identifier is equivalent to the e_keep function.
- Parameters
Parameter Data type Required Description condition Bool No Default value: True. This parameter is typically set to a conditional expression. - Response
If the specified condition is met, the raw log entries are returned. If the specified condition is not met, log entries are dropped.
- Examples
- Example 1: Evaluate log entries against the __programe__==access expression. If the
value of the
__programe__
field in a log entry is access, the log entry is retained. Otherwise, the log entry is dropped.- Raw log entry
__programe__: access age: 18 content: 123 name: maki __programe__: error age: 18 content: 123 name: maki
- Transformation rule
e_keep(e_search("__programe__==access")) # Equivalent to: e_if(e_search("not __programe__==access"), DROP) # Equivalent to: e_if_else(e_search("__programe__==access"), KEEP, DROP)
- Result:
The log entry whose __programe__ field value is access is retained while the log entry whose __programe__ field value is error is dropped.
__programe__: access age: 18 content: 123 name: maki
- Raw log entry
- Example: Evaluate a log entry against the k1==v1 expression. If the evaluation result
of the conditional expression is True, the log entry is retained.
- Raw log entry
k1: v1 k2: v2 k3: k1
- Transformation rule
e_keep(e_search("k1==v1"))
- Result:
k1: v1 k2: v2 k3: k1
- Raw log entry
- Example 3: Evaluate a log entry against the not k1==v1 expression. If the evaluation
result of the conditional expression is False, the log entry is dropped.
- Raw log entry
k1: v1 k2: v2 k3: k1
- Transformation rule
e_keep(e_search("not k1==v1"))
- Result:
The log entry is dropped.
- Raw log entry
- Example 4: Specify the False expression in the e_keep function to drop a log entry.
- Raw log entry
k1: v1 k2: v2 k3: k1
- Transformation rule
e_keep(False)
- Result:
The log entry is dropped.
- Raw log entry
- Example 1: Evaluate log entries against the __programe__==access expression. If the
value of the
e_split
- Syntax
e_split(Field name, sep=',', quote='"', lstrip=True, jmes=None, output=None)
- Splitting rules
- If you specify the jmes parameter, the system converts the value of the field to a JSON list, and uses the JMESPath expression to extract values from the JSON list, which will be used in the next step. If you do not specify the jmes parameter, the system directly uses the value of the field in the next step.
- If the value obtained from the previous step is a list or a string that represents a JSON list, the system splits the event based on this value. Otherwise, the system parses the value to multiple delimited values based on the sep, quote, and lstrip parameters. Then, the system splits the event based on these values.
- Parameters
Parameter Data type Required Description Log field String Yes The name of the field used to split the event. For more information about how to set special field names, see Event structure and fields. sep String No The delimiter used to separate values. quote String No The character used to enclose a value. lstrip String No Specifies whether to remove the spaces before a value. Default value: True. jmes String No The JMESPath string used to convert the value of the field to a JSON object and extract values from the JSON object. output String No The new field name, which overwrites the existing field name by default. - Result
A log list is returned. The values of fields in the list are all those from the source log.
- Examples
- Raw log entry
__topic__: age: 18 content: 123 name: maki __topic__: age: 18 content: 123 name: maki
- Transformation rule
e_set("__topic__", "V_SENT,V_RECV,A_SENT,A_RECV") e_split("__topic__")
- Result:
__topic__: A_SENT age: 18 content: 123 name: maki __topic__: V_RECV age: 18 content: 123 name: maki ...
- Raw log entry
e_output, e_coutput
- Syntax
e_output(name=None, project=None, logstore=None, topic=None, source=None, tags=None) e_coutput(name=None, project=None, logstore=None, topic=None, source=None, tags=None)
During preview, log entries are not written to the destination Logstore but to the Logstore named internal-etl-log. The internal-etl-log Logstore is a dedicated Logstore created by the system in the current project when you preview a data transformation task for the first time. You cannot modify the configuration of this Logstore or write other data to this Logstore. It is free of charge.
- Description
Note If you specify the name, project, and logstore parameters in the e_output function or e_coutput function and specify the destination project and Logstore on the Create Data Transformation Rule pane, the configurations in the e_output and e_coutput functions prevail. The following list shows the differences of the configurations:
- If you specify only the name parameter in the e_output or e_coutput function, the transformed data is stored in the destination Logstore that stores the storage target.
- If you specify only the project and logstore parameters in the e_output function,
the transformed data is stored in the Logstore specified in the function.
If you use an AccessKey pair to authorize data transformation, the AccessKey pair of the current logon account is used for data transformation.
- If you specify the name, project, and logstore parameters in the e_output function,
the transformed data is stored in the Logstore specified in the function.
If you use an AccessKey pair to authorize data transformation, the AccessKey pair that you specify when you configure the storage target is used for data transformation.
Parameter Data type Required Description name String No The name of the storage target. Default value: None. project String No The existing project to which the log entries are written. logstore String No The existing Logstore to which the log entries are written. topic String No The new topic of the log entries. source String No The source of the log entries. tags Dict No The new tags of the log entries, in the dictionary format. Note You do not need to prefix keywords with__tag__:
. - Set default storage target
To use the e_output or e_coutput function, you must configure a default storage target on the Create Data Transformation Rule pane. By default, Log Service uses the storage target labelled 1 as the default storage target. For example, transformed data is respectively sent to destination Logstores that store target_01, target_02, and target_03. Data that is not dropped during transformation is stored in the Logstore that stores the default storage target (target_00), as shown in the following figure.
- Advanced parameter settings
If the project or Logstore that you specify in the e_output or e_coutput function does not exist, you can set key-value pairs in the Advanced Parameter Settings section of the Create Data Transformation Rule pane. You can set a key to config.sls_output.failure_strategy and a value to {"drop_when_not_exists":"true"} to skip a log entry. The skipped log entry is dropped and reported as a WARNING-level log entry. If you do not set key-value pairs in the Advanced Parameter Settings section, the data transformation tasks are suspended until the specified project or Logstore is created.Warning If the specified project or Logstore does not exist and you set key-value pairs in the Advanced Parameter Settings section to skip a log entry, the log entry will be dropped. Proceed with caution.
- Result:
- e_output: sends log entries to the specified Logstore. After a log entry is sent, the following transformation rules are not executed on the log entry.
- e_coutput: sends log entries to the specified Logstore. After a log entry is sent, the following transformation rules are still executed on the log entry.
- Examples
- Raw log entry
__topic__: k1: v1 k2: v2 x1: v3 x5: v4
- Transformation rule
e_if(e_match("k2", r"\w+"), e_output(name="target2", source="source1", topic="topic1"))
- Result:
__topic__: topic1 k1: v1 k2: v2 x1: v3 x5: v4
- Raw log entry