Log Service:Syntax overview

The DSL for Log Service is compatible with Python. In standard mode, the DSL can be regarded as a subset of Python. Except for basic data structures and expressions, other syntax rules are orchestrated by using functions. If you want to use user-defined functions (UDFs), submit a ticket.

Category	Python syntax	Standard mode
Data structure	Number, string, and Boolean	Supported. Strings that start or end with """ are not supported.
	Tuple, list, set, and dictionary	Supported. The set structure is not supported. Example: {1,2,3}.
	Object	Only built-in extended data structures such as table and datetime objects are supported.
Basic syntax	Operators such as the plus sign (+), the subtraction sign (-), the multiplication sign (×), and the divide operator (/)	Only comparison operators such as ==, !=, and >, and logical operators such as AND, OR, and NOT can be directly used in code. You must call functions to use the functionality of other operators.
	Comments	Supported.
	Variable assignment	Not supported. You must call functions to pass values to variables.
	Condition evaluation	Supported. Functions: e_if, e_if_else, and e_switch.
	Loops	Indirectly supported. You must use nested built-in functions to implement loops. The following sample functions show how to traverse the elements in an array: e_if(op_ge(op_len(json_parse(v("x"))), 1), e_set("x0", lst_get(v("x"), 0))) e_if(op_ge(op_len(json_parse(v("x"))), 2), e_set("x1", lst_get(v("x"), 1)))
Function	Standard built-in functions of Python	Not supported. You can use more than 200 built-in functions provided by the DSL for Log Service.
	Function calls	Supported. Function calls that use parameter unpacking are not supported.
	UDFs, such as def or lambda	Not supported. You can use more than 200 global processing and expression functions provided by the DSL for Log Service. You can also combine these functions based on your business requirements.
Module	Import and use of the Python standard library	Not supported.
	Creation of threads and processes	Not supported.
	Import of third-party libraries	Not supported.
	External network connection or external command call	Supported. The DSL for Log Service provides built-in resource connectors.

In standard mode of the DSL for Log Service, all operations are performed by calling functions. The DSL provides more than 200 built-in functions, which are categorized into global processing functions and expression functions.

• Global processing functions

  Global processing functions are used to receive, process, and return logs. Only global processing functions can be used to construct each step in a transformation rule.

• Expression functions

  Expression functions are commonly used to receive specific parameters and return specific values. Expression functions can be combined and passed to global processing functions as parameters to define more flexible logic.

The following table describes global processing functions and expression functions.

Category	Construct a step	Receive a log	Return results	Modify a log	Combine functions
Global processing functions	Supported.	Logs are automatically received.	Zero to multiple logs are returned.	Supported. In most cases, logs can be modified.	Supported.
Expression functions	Not supported.	Supported by only a few expression functions. Most expression functions do not directly process logs.	Specific data structures are returned.	Not supported.	Supported.

Global processing functions are used to receive, process, and return logs. The following syntax is used:

Global Processing Function 1(..Parameters....)
Global Processing Function 2(..Parameters....)
Global Processing Function 3(..Parameters....)
Global Processing Function 4(..Parameters....)

Global processing functions can be further categorized into flow control functions and event processing functions. The following table describes the functions.

Category	Description	Example
Flow control functions	The functions are used to manage processes, receive logs, and call other functions to process logs based on specific conditions.	e_if, e_switch, and e_if_else.
Event processing functions	The functions are used to transform logs. Zero to multiple logs are returned.	Examples: e_drop_fields: discards log fields. e_kv: extracts the key-value pairs of logs. e_dict_map: enriches logs.

Transformation logic:

• Basic processing
  The data transformation feature reads streaming data from a source Logstore and sends each log in a dictionary structure to specific functions. Then, the feature runs the functions that are specified in the transformation rule in sequence to process the events and writes the transformation results to specified destination Logstores.

  Note All fields and values of a log are sent as strings. For example, the raw log {"__time__": "1234567", "__topic__": "", "k1": "test"} is processed by the e_set("f1", 200) function. The function adds the f1 field whose value is set to 200 to the raw log. Then, the raw log is transformed into {"__time__": "1234567", "__topic__": "", "k1": "test", "f1": "200"}. In this log, the f1 field and the value 200 are strings.

  The event processing functions specified in a transformation rule are called in sequence. Each function receives and processes a log, and then returns a processed log.

  For example, the e_set("type", "test") function adds the type field whose value is set to test to a log. The next function receives and processes the processed log.

• Condition evaluation
  • e_if: You can call the e_if function to add conditional expressions to process logs. If a log does not meet the specified condition, the corresponding operation is skipped. The e_if function implements the if logic.

    For example, the e_if(e_match("status", "200"), e_regex("data", "ret: \d+", "result")) function checks whether the value of the status field is 200. If the value is 200, the function extracts the result field from the data field by using the specified regular expression. Otherwise, no operation is performed.

  • e_if_else: This function works in a manner similar to the if_else function.
• Processing termination
  • A step in a transformation rule may return no log. This indicates that the related log is deleted.

    For example, the e_if(str_islower(v("result")), e_drop()) function is used to check whether the value of the result field in a log is a string that consists of only lowercase characters. If the condition is evaluated to true, the log is discarded and the subsequent steps are not performed on this log. The system automatically processes the next log.

  • If a log is written to a destination Logstore, the processing is terminated. For example, if the e_output function is used to write a log to a destination Logstore and delete the log, the subsequent steps are not performed on this log.
    Note The e_coutput function copies the output log and the subsequent steps are performed on this log.
• Log splitting for parallel processing

  A step in a transformation rule may return multiple logs. This indicates that the related log is split.

  For example, the e_split(data) function splits a log into two logs based on the value of the data field. If the value of the data field in the log is "abc, xyz", the log is split into two logs. In one log, the value of thedata field is abc. In the other log, the value of the data field is xyz.

  The logs that are generated after splitting are processed in the subsequent steps.

In addition to global processing functions, the DSL for Log Service provides 200 expression functions that are used to receive specific parameters and return specific values. You can call an expression function or a combination of expression functions in a global processing function. The following syntax is used:

Global Processing Function 1(Expression Function 1(...), ...)
Global Processing Function 2(..., Expression Function 2(...), Expression Function 3(...), ...)

Expression functions can be categorized into event check functions, resource functions, control functions, and other expression functions. The following table describes the functions.

Category	Description	Example
Event check functions	The functions are used to receive logs, extract specific information, and then return the information without modifying the logs.	v: returns the value of a log field. e_search and e_match: checks whether the value of a field in a log meets a specified condition.
Resource functions	The functions are used to access on-premises or external resources, receive specific parameters, and return specific values. The data types of the return values include dictionary and table.	res_oss_file, res_rds_mysql, and res_log_logstore_pull.
Control functions	The functions are used to receive specific parameters and perform logical operations on expressions or condition-based control. The functions are also used to call other expression functions to return results.	op_and, op_or, op_not, op_if, and op_coalesce.
Other expression functions	The functions are used to receive specific parameters or the results of other functions and return specific values.	String functions, date and time functions, and conversion functions.