Syntax introduction

The DSL for Log Service is compatible with Python. In standard mode, the DSL can be regarded as a subset of Python. Except for basic data structures and expressions, other syntax rules are orchestrated by using functions. If you want to use user-defined functions (UDFs), submit a ticket.

Category	Python syntax	Standard mode
Data structure	Number, string, and Boolean	Supported, except for strings in the """ form.
	Tuple, list, set, and dictionary	Supported, except for the set structure, such as {1,2,3}.
	Object	Only built-in extended data structures such as table and datetime objects are supported.
Basic syntax	Operators, such as + (addition), - (subtraction), * (multiplication), and / (division)	Not supported directly. You must use the operators in functions.
	Comments	Supported.
	Variable assignment	Not supported. You must call functions to pass values to variables.
	Condition evaluation and loops	Not supported directly. You must use built-in functions to implement condition evaluation and loops.
Function	Standard built-in functions of Python	Not supported. You can use more than 200 built-in functions provided by the DSL for Log Service.
	Function calls	Supported, except for function calls with parameter unpacking.
	UDFs, such as def or lambda	Not supported. You can use more than 200 global processing and expression functions provided by the DSL for Log Service. You can also combine these functions to suite your needs.
Module	Import and use of the Python standard library	Not supported.
	Creation of threads and processes	Not supported.
	Import of third-party libraries	Not supported.
	External network connection or external command call	Supported. The DSL for Log Service provides built-in resource connectors.

In standard mode of the DSL for Log Service, all operations are implemented by calling functions. The DSL provides more than 200 built-in functions, which are categorized into global processing functions and expression functions.

• Global processing functions

  Global processing functions receive, process, and return events. Only global processing functions can be used to construct each step of a transformation rule.

• Expression functions

  Expression functions are general functions that receive specific parameters and return specific values. Expression functions can be combined and passed to global processing functions as parameters to define more flexible logic.

The following table compares global processing functions with expression functions.

Category	Construct a step	Receive an event	Return results	Modify an event	Combine functions
Global processing function	Supported.	Events are automatically received.	Zero to multiple events are returned.	Supported. Events can be modified in most cases.	Supported.
Expression function	Not supported.	Supported only by a few expression functions. Most expression functions do not directly process events.	Specific data structures are returned.	Not supported.	Supported.

Global processing functions receive, process, and return events. The following syntax is used:

Global Processing Function 1(...Parameters...)
Global Processing Function 2(...Parameters...)
Global Processing Function 3(...Parameters...)
Global Processing Function 4(...Parameters...)

Global processing functions can be further categorized into flow control functions and event processing functions.

Category	Description	Example
Flow control function	This type of function controls processes, receives events, and calls event processing functions based on conditions.	e_if, e_switch, and e_if_else
Event processing function	This type of function processes events. Zero to multiple events are returned.	Examples: e_drop_fields: discards fields of events. e_kv: extracts and adds key-value pairs of events. e_dict_map: enriches events.

Transformation logic:

• Basic processing
  The data transformation feature reads streaming data from a source Logstore and sends each log event in a dictionary structure to a transformation rule. Then, the feature runs the event processing functions defined in the transformation rule in sequence to process the events and writes the processing results to the destination Logstores.

  Note All the fields and values of an event are transmitted as strings. For example, the log event {"__time__": "1234567", "__topic__": "", "k1": "test"} is processed by the function e_set("f1", 200), which adds the field f1 with a value of 200. After processing, the event {"__time__": "1234567", "__topic__": "", "k1": "test", "f1": "200"} is returned. f1 and 200 are both strings.

  Event processing functions defined in a transformation rule are called in sequence. Each function receives and processes an event and returns a processed event.

  For example, the function e_set("type", "test") adds the field type to each event and sets the value to test. The next function receives the processed event.

• Condition evaluation
  • e_if: You can call the e_if function to add conditional expressions to process events. If an event does not meet the specified condition, the operation is skipped. The e_if function implements the if logic.

    For example, the function e_if(e_match("status", "200"), e_regex("data", "ret: \d +", "result")) first determines whether the value of the status field is 200. If the value is 200, the function extracts a new field result from the data field by using the regular expression. If the value is not 200, the extract operation is not performed.

  • e_if_else: This function works in a similar way to the if_else function.
• Processing termination
  • Some steps of a transformation rule may return no events. This indicates that the event is deleted.

    For example, if the function e_if(str_islower(v("result")), e_drop()) is used and the value of each result field in an event is a string consisting of only lowercase characters, the function deletes this event. Then, the subsequent steps are not performed on this event. The system automatically processes the next event.

  • If an event is written to a destination Logstore, the processing is terminated. For example, the e_output function writes an event to a destination Logstore and deletes the event, and the subsequent steps are not performed on this event.
    Note However, the e_coutput function writes an event to a destination Logstore but retains the original event, and the subsequent steps are performed on this event.
• Event splitting for parallel processing

  Some steps of a transformation rule may return multiple events. This indicates that the original event is split.

  For example, the function e_split(data) splits an event into two events based on the value of the data field. If the value of the data field in the original event is "abc, xyz", the values of the data field in the events generated after splitting are abc and xyz.

  All the events generated after splitting are processed in the subsequent steps.

In addition to global processing functions, the DSL for Log Service provides 200 expression functions that receive specific parameters and return specific values. You can call a single expression function or a combination of expression functions in a global processing function. The following syntax is used:

Global Processing Function 1(Expression Function 1(...), ...)
Global Processing Function 2(..., Expression Function 2(...), Expression Function 3(...), ...)

Expression functions can be categorized into event check functions, resource functions, control functions, and other expression functions.

Category	Description	Example
Event check function	This type of function receives events, extracts specific information, and returns the information. Events are not modified.	v: returns the values of fields. e_search and e_match: check whether events meet the specified conditions.
Resource function	This type of function accesses on-premises or external resources, receives specific parameters, and returns specific values. The data types of return values include dictionary and table.	res_oss_file, res_rds_mysql, and res_log_logstore_pull.
Control function	This type of function receives specific parameters and performs logical operations on expressions or condition-based control. This type of function also calls other expression functions to return results.	op_and, op_or, op_not, op_if, and op_coalesce.
Other expression function	This type of function receives specific parameters or the results of other functions and returns specific values.	String functions, date and time functions, and conversion functions.