LOG domain specific language (DSL) provides different types of functions to meet the data processing needs in different scenarios.

Language modes

LOG DSL is compatible with Python. LOG DSL supports the following modes:
  • Standard mode: This is the default mode. In this mode, LOG DSL can be considered as a subset of Python. Except for basic data structures and expressions, other syntax rules are defined through functions.
  • Advanced mode: In this mode, LOG DSL is fully compatible with Python. If you want to use LOG DSL in advanced mode, submit a ticket to request Alibaba Cloud to add you to the whitelist.
The following table compares LOG DSL syntax in the preceding two modes with Python syntax.
Category Python syntax Standard mode Advanced mode
Data structure Number, string, and Boolean Supported, except for strings in """ form. Supported.
Tuple, list, set, and dictionary Supported, except for the set structure, such as {1,2,3}. Supported.
Object definition Only the built-in extended data structures are supported, such as table and datetime object. Supported.
Basic syntax Operators, such as +, -, *, and /. Not supported directly. You can implement the corresponding operations through functions. Supported.
Comments Supported. Supported.
Variable assignment Not supported. You need to pass values through function calls. Supported.
Condition-based judgment and loops Not supported directly. You can implement condition-based judgment and loops through built-in functions. -
Functions Standard built-in functions of Python Not supported. LOG DSL provides over 200 built-in functions for you to use. Supported.
Function calls Supported, except for function calls with parameter unpacking. Supported.
User-defined functions such as def and lambda functions Not supported. LOG DSL provides over 200 global processing and expression functions for you to use. You can also combine these functions as needed. Supported.
Modules Import and use of the Python standard library Not supported. Supported.
Thread and process creation Not supported. Supported.
Third-party library import Not supported. You can submit a ticket for importing third-party libraries.
External network access or external command calls Supported. LOG DSL provides built-in resource connectors. You can submit a ticket for accessing external networks or calling external commands.

Function categories

In standard mode of LOG DSL, all operations are implemented through function calls. LOG DSL provides over 200 built-in functions, which are divided into two categories as follows:
  • Global processing functions

    Global processing functions receive, process, and return events. Only global processing functions can be used to construct the steps of processing rules.

  • Expression functions

    Expression functions are general functions that receive specific parameters and return specific values. Expression functions can be combined and passed to global processing functions as parameters to define more flexible logic.

The following table compares global processing functions with expression functions.
Category Step construction Event receiving Return value Event modification Function combination
Global processing functions Supported. Events are received automatically. Zero to multiple events. Supported. Events are modified in most cases. Supported.
Expression functions Not supported. Supported only by a few functions. Most functions do not process events directly. Specific data structures. Not supported. Supported.

Global processing functions

Global processing functions receive, process, and return events.
Note Only a global operation function can be placed in the first line of each step.
The syntax of a LOG DSL processing rule is as follows:
Global processing function 1 (... Parameters...)
Global processing function 2 (... Parameters...)
Global processing function 3 (... Parameters...)
Global processing function 4 (... Parameters...)
Global processing functions can be divided into two categories, as described in the following table.
Category Description Example
Process control functions The functions that control processes, receive events, and call event processing functions based on conditions. e_if, e_switch, and e_if_else
Event processing functions The functions that process events. They return zero to multiple events. Examples:
  • e_drop_fields: deletes fields from events.
  • e_kv: extracts and adds key-value pairs.
  • e_dict_map: enriches events.
Processing logic:
  • Basic processing
    The data processing feature reads data from the source Logstore in streaming mode, and transmits each event in a dictionary structure to a processing rule. The system runs the event processing functions defined in the processing rule in sequence to process the event, and writes the final data processing result to the default destination Logstore.
    Note All fields and values of an event are transmitted as strings. Assume that the raw event is {"__time__": "1234567", "__topic__": "", "k1": "test"}. The e_set("f1", 200) function adds the f1 field to the event and sets its value to 200. The event becomes {"__time__": "1234567", "__topic__": "", "k1": "test", "f1": "200"}. In the event, both the f1 and 200 fields are strings.

    Event processing functions defined in a processing rule are called in sequence. Each function receives and processes each event and returns a processed event.

    For example, the e_set("type", "test") function adds the type field to each event and sets its value to test. The subsequent function receives the modified event.

  • Condition-based judgment
    • e_if: You can call the e_if function to add condition-based judgment for some events. If an event does not meet the condition, the operation is skipped. The e_if function implements the if logic.

      Take the e_if(e_match("status", "200"), e_regex("data", "ret: \d+", "result")) function for example. This function checks whether the value of the status field is 200 in an event. If the result is true, a value is extracted from the data field based on the regular expression and assigned to a new field named result. If the result is false, no operation is performed.

    • e_if_else: This function implements the if_else logic.
  • Process termination
    • Some steps may return no event. This indicates that the event is deleted.

      Take the e_if(str_islower(v("result")), e_drop()) function for example. If the value of the result field is a string consisting of only lowercase characters in an event, this event is deleted. In this case, the subsequent steps are not performed on this event. The system starts processing the next event automatically.

    • Writing an event to a Logstore may also indicate that the process ends. For example, the e_output function writes an event to a Logstore and deletes the event. In this case, the subsequent steps are not performed on this event.
      Note However, the e_coutput function writes an event to a Logstore but retains the original event. In this case, the event continues to be processed in subsequent steps.
  • Event splitting for concurrent processing

    Some steps may return multiple events. This indicates that the original event is split.

    For example, the e_split(data) function splits an event with the data field to multiple events. Assume that the value of the data field is "abc, xyz". The event is split to two events, in which the values of the data field are abc and xyz, respectively.

    All events generated after the splitting are processed in the subsequent steps.

Expression functions

In addition to global processing functions, LOG DSL provides over 200 expression functions that receive specific parameters and return specific values. You can call a single expression function or a combination of expression functions in a global processing function. The syntax is as follows:
Global processing function 1(Expression function 1(...), ...)
Global processing function 2(..., Expression function 2(...), Expression function 3(...), ...)
Expression functions can be divided into four categories, as described in the following table.
Category Description Example
Event check functions The functions that receive events, extract specific information, and return the information. Event check functions do not modify events. v: returns the values of fields. e_search and e_match: check whether events meet the specified conditions.
Resource functions The functions that connect to local or external resources and return data based on specified parameters. The returned data is generally dictionaries and tables. res_oss_file, res_rds_mysql, and res_log_logstore_pull
Control functions The functions that receive specific parameters and perform logical operations on expressions or condition-based control. They can call other expression functions and return the results. op_and, op_or, op_not, op_if, and op_coalesce
Other expression functions The functions that receive specific parameters or the results of other functions and return specific values. String functions, date and time functions, and conversion functions