LOG domain specific language (DSL) provides different types of functions to meet the data processing needs in different scenarios.
Language modes
- Standard mode: This is the default mode. In this mode, LOG DSL can be considered as a subset of Python. Except for basic data structures and expressions, other syntax rules are defined through functions.
- Advanced mode: In this mode, LOG DSL is fully compatible with Python. If you want to use LOG DSL in advanced mode, submit a ticket to request Alibaba Cloud to add you to the whitelist.
Category | Python syntax | Standard mode | Advanced mode |
---|---|---|---|
Data structure | Number, string, and Boolean | Supported, except for strings in """ form.
|
Supported. |
Tuple, list, set, and dictionary | Supported, except for the set structure, such as {1,2,3} .
|
Supported. | |
Object definition | Only the built-in extended data structures are supported, such as table and datetime object. | Supported. | |
Basic syntax | Operators, such as +, -, *, and /. | Not supported directly. You can implement the corresponding operations through functions. | Supported. |
Comments | Supported. | Supported. | |
Variable assignment | Not supported. You need to pass values through function calls. | Supported. | |
Condition-based judgment and loops | Not supported directly. You can implement condition-based judgment and loops through built-in functions. | - | |
Functions | Standard built-in functions of Python | Not supported. LOG DSL provides over 200 built-in functions for you to use. | Supported. |
Function calls | Supported, except for function calls with parameter unpacking. | Supported. | |
User-defined functions such as def and lambda functions | Not supported. LOG DSL provides over 200 global processing and expression functions for you to use. You can also combine these functions as needed. | Supported. | |
Modules | Import and use of the Python standard library | Not supported. | Supported. |
Thread and process creation | Not supported. | Supported. | |
Third-party library import | Not supported. | You can submit a ticket for importing third-party libraries. | |
External network access or external command calls | Supported. LOG DSL provides built-in resource connectors. | You can submit a ticket for accessing external networks or calling external commands. |
Function categories
- Global processing functions
Global processing functions receive, process, and return events. Only global processing functions can be used to construct the steps of processing rules.
- Expression functions
Expression functions are general functions that receive specific parameters and return specific values. Expression functions can be combined and passed to global processing functions as parameters to define more flexible logic.
Category | Step construction | Event receiving | Return value | Event modification | Function combination |
---|---|---|---|---|---|
Global processing functions | Supported. | Events are received automatically. | Zero to multiple events. | Supported. Events are modified in most cases. | Supported. |
Expression functions | Not supported. | Supported only by a few functions. Most functions do not process events directly. | Specific data structures. | Not supported. | Supported. |
Global processing functions
Global processing function 1 (... Parameters...)
Global processing function 2 (... Parameters...)
Global processing function 3 (... Parameters...)
Global processing function 4 (... Parameters...)
Global processing functions can be divided into two categories, as described in the
following table.
Category | Description | Example |
---|---|---|
Process control functions | The functions that control processes, receive events, and call event processing functions based on conditions. | e_if , e_switch , and e_if_else |
Event processing functions | The functions that process events. They return zero to multiple events. | Examples:
|
- Basic processing
The data processing feature reads data from the source Logstore in streaming mode, and transmits each event in a dictionary structure to a processing rule. The system runs the event processing functions defined in the processing rule in sequence to process the event, and writes the final data processing result to the default destination Logstore.Note All fields and values of an event are transmitted as strings. Assume that the raw event is
{"__time__": "1234567", "__topic__": "", "k1": "test"}
. Thee_set("f1", 200)
function adds thef1
field to the event and sets its value to 200. The event becomes{"__time__": "1234567", "__topic__": "", "k1": "test", "f1": "200"}
. In the event, both thef1
and 200 fields are strings.Event processing functions defined in a processing rule are called in sequence. Each function receives and processes each event and returns a processed event.
For example, the
e_set("type", "test")
function adds thetype
field to each event and sets its value to test. The subsequent function receives the modified event. - Condition-based judgment
- e_if: You can call the e_if function to add condition-based judgment for some events.
If an event does not meet the condition, the operation is skipped. The e_if function
implements the
if
logic.Take the
e_if(e_match("status", "200"), e_regex("data", "ret: \d+", "result"))
function for example. This function checks whether the value of thestatus
field is 200 in an event. If the result is true, a value is extracted from thedata
field based on the regular expression and assigned to a new field namedresult
. If the result is false, no operation is performed. e_if_else
: This function implements theif_else
logic.
- e_if: You can call the e_if function to add condition-based judgment for some events.
If an event does not meet the condition, the operation is skipped. The e_if function
implements the
- Process termination
- Some steps may return no event. This indicates that the event is deleted.
Take the
e_if(str_islower(v("result")), e_drop())
function for example. If the value of theresult
field is a string consisting of only lowercase characters in an event, this event is deleted. In this case, the subsequent steps are not performed on this event. The system starts processing the next event automatically. - Writing an event to a Logstore may also indicate that the process ends. For example,
the
e_output
function writes an event to a Logstore and deletes the event. In this case, the subsequent steps are not performed on this event.Note However, thee_coutput
function writes an event to a Logstore but retains the original event. In this case, the event continues to be processed in subsequent steps.
- Some steps may return no event. This indicates that the event is deleted.
- Event splitting for concurrent processing
Some steps may return multiple events. This indicates that the original event is split.
For example, the
e_split(data)
function splits an event with thedata
field to multiple events. Assume that the value of thedata
field is"abc, xyz"
. The event is split to two events, in which the values of thedata
field are abc and xyz, respectively.All events generated after the splitting are processed in the subsequent steps.
Expression functions
Global processing function 1(Expression function 1(...), ...)
Global processing function 2(..., Expression function 2(...), Expression function 3(...), ...)
Expression functions can be divided into four categories, as described in the following
table.
Category | Description | Example |
---|---|---|
Event check functions | The functions that receive events, extract specific information, and return the information. Event check functions do not modify events. | v : returns the values of fields. e_search and e_match : check whether events meet the specified conditions.
|
Resource functions | The functions that connect to local or external resources and return data based on specified parameters. The returned data is generally dictionaries and tables. | res_oss_file, res_rds_mysql, and res_log_logstore_pull |
Control functions | The functions that receive specific parameters and perform logical operations on expressions or condition-based control. They can call other expression functions and return the results. | op_and , op_or , op_not , op_if , and op_coalesce |
Other expression functions | The functions that receive specific parameters or the results of other functions and return specific values. | String functions, date and time functions, and conversion functions |