The domain-specific language (DSL) for Log Service provides different types of functions to meet the data transformation and processing needs in different scenarios.
Language modes
Category | Python syntax | Standard mode |
---|---|---|
Data structure | Number, string, and Boolean | Supported, except for strings in the """ form.
|
Tuple, list, set, and dictionary | Supported, except for the set structure, such as {1,2,3} .
|
|
Object | Only built-in extended data structures such as table and datetime objects are supported. | |
Basic syntax | Operators, such as + (addition), - (subtraction), * (multiplication), and / (division) | Not supported directly. You must use the operators in functions. |
Comments | Supported. | |
Variable assignment | Not supported. You must call functions to pass values to variables. | |
Condition evaluation and loops | Not supported directly. You must use built-in functions to implement condition evaluation and loops. | |
Function | Standard built-in functions of Python | Not supported. You can use more than 200 built-in functions provided by the DSL for Log Service. |
Function calls | Supported, except for function calls with parameter unpacking. | |
UDFs, such as def or lambda | Not supported. You can use more than 200 global processing and expression functions provided by the DSL for Log Service. You can also combine these functions to suite your needs. | |
Module | Import and use of the Python standard library | Not supported. |
Creation of threads and processes | Not supported. | |
Import of third-party libraries | Not supported. | |
External network connection or external command call | Supported. The DSL for Log Service provides built-in resource connectors. |
Function categories
- Global processing functions
Global processing functions receive, process, and return events. Only global processing functions can be used to construct each step of a transformation rule.
- Expression functions
Expression functions are general functions that receive specific parameters and return specific values. Expression functions can be combined and passed to global processing functions as parameters to define more flexible logic.
Category | Construct a step | Receive an event | Return results | Modify an event | Combine functions |
---|---|---|---|---|---|
Global processing function | Supported. | Events are automatically received. | Zero to multiple events are returned. | Supported. Events can be modified in most cases. | Supported. |
Expression function | Not supported. | Supported only by a few expression functions. Most expression functions do not directly process events. | Specific data structures are returned. | Not supported. | Supported. |
Global processing functions
Global Processing Function 1(...Parameters...)
Global Processing Function 2(...Parameters...)
Global Processing Function 3(...Parameters...)
Global Processing Function 4(...Parameters...)
Global processing functions can be further categorized into flow control functions
and event processing functions. Category | Description | Example |
---|---|---|
Flow control function | This type of function controls processes, receives events, and calls event processing functions based on conditions. | e_if , e_switch , and e_if_else
|
Event processing function | This type of function processes events. Zero to multiple events are returned. | Examples:
|
- Basic processing
The data transformation feature reads streaming data from a source Logstore and sends each log event in a dictionary structure to a transformation rule. Then, the feature runs the event processing functions defined in the transformation rule in sequence to process the events and writes the processing results to the destination Logstores.Note All the fields and values of an event are transmitted as strings. For example, the log event
{"__time__": "1234567", "__topic__": "", "k1": "test"}
is processed by the functione_set("f1", 200)
, which adds the fieldf1
with a value of 200. After processing, the event{"__time__": "1234567", "__topic__": "", "k1": "test", "f1": "200"}
is returned.f1
and 200 are both strings.Event processing functions defined in a transformation rule are called in sequence. Each function receives and processes an event and returns a processed event.
For example, the function
e_set("type", "test")
adds the fieldtype
to each event and sets the value to test. The next function receives the processed event. - Condition evaluation
- e_if: You can call the e_if function to add conditional expressions to process events.
If an event does not meet the specified condition, the operation is skipped. The e_if
function implements the
if
logic.For example, the function
e_if(e_match("status", "200"), e_regex("data", "ret: \d +", "result"))
first determines whether the value of thestatus
field is 200. If the value is 200, the function extracts a new fieldresult
from thedata
field by using the regular expression. If the value is not 200, the extract operation is not performed. e_if_else
: This function works in a similar way to theif_else
function.
- e_if: You can call the e_if function to add conditional expressions to process events.
If an event does not meet the specified condition, the operation is skipped. The e_if
function implements the
- Processing termination
- Some steps of a transformation rule may return no events. This indicates that the
event is deleted.
For example, if the function
e_if(str_islower(v("result")), e_drop())
is used and the value of eachresult
field in an event is a string consisting of only lowercase characters, the function deletes this event. Then, the subsequent steps are not performed on this event. The system automatically processes the next event. - If an event is written to a destination Logstore, the processing is terminated. For
example, the
e_output
function writes an event to a destination Logstore and deletes the event, and the subsequent steps are not performed on this event.Note However, thee_coutput
function writes an event to a destination Logstore but retains the original event, and the subsequent steps are performed on this event.
- Some steps of a transformation rule may return no events. This indicates that the
event is deleted.
- Event splitting for parallel processing
Some steps of a transformation rule may return multiple events. This indicates that the original event is split.
For example, the function
e_split(data)
splits an event into two events based on the value of thedata
field. If the value of thedata
field in the original event is"abc, xyz"
, the values of thedata
field in the events generated after splitting are abc and xyz.All the events generated after splitting are processed in the subsequent steps.
Expression functions
Global Processing Function 1(Expression Function 1(...), ...)
Global Processing Function 2(..., Expression Function 2(...), Expression Function 3(...), ...)
Expression functions can be categorized into event check functions, resource functions,
control functions, and other expression functions. Category | Description | Example |
---|---|---|
Event check function | This type of function receives events, extracts specific information, and returns the information. Events are not modified. | v : returns the values of fields. e_search and e_match : check whether events meet the specified conditions.
|
Resource function | This type of function accesses on-premises or external resources, receives specific parameters, and returns specific values. The data types of return values include dictionary and table. | res_oss_file, res_rds_mysql, and res_log_logstore_pull. |
Control function | This type of function receives specific parameters and performs logical operations on expressions or condition-based control. This type of function also calls other expression functions to return results. | op_and , op_or , op_not , op_if , and op_coalesce .
|
Other expression function | This type of function receives specific parameters or the results of other functions and returns specific values. | String functions, date and time functions, and conversion functions. |