All Products
Search
Document Center

Log Service:Syntax overview

Last Updated:Aug 08, 2023

The Python-based domain-specific language (DSL) for Simple Log Service provides more than 200 built-in functions that you can use to efficiently transform data. This topic describes the language modes, function categories, and implementation of the DSL for Simple Log Service.

Language modes

The DSL for Simple Log Service is compatible with Python. In standard mode, the DSL can be regarded as a subset of Python. Except for basic data structures and expressions, other syntax rules are orchestrated by using functions. If you want to use user-defined functions (UDFs), submit a ticket.

Category

Python syntax

Standard mode

Data structure

Number, string, and Boolean

Supported.

Strings that start or end with """ are not supported.

Tuple, list, set, and dictionary

Supported.

The set structure is not supported. Example: {1,2,3}.

Object

Only built-in extended data structures such as table and datetime objects are supported.

Basic syntax

Operators such as the plus sign (+), the subtraction sign (-), the multiplication sign (×), and the divide operator (/)

Only comparison operators such as ==, !=, and >, and logical operators such as AND, OR, and NOT can be directly used in code. You must call functions to use the functionality of other operators.

Comments

Supported.

Variable assignment

Not supported. You must call functions to pass values to variables.

Condition evaluation

Supported. Functions: e_if, e_if_else, and e_switch.

Loops

Indirectly supported. You must use nested built-in functions to implement loops. The following sample functions show how to traverse the elements in an array:

e_if(op_ge(op_len(json_parse(v("x"))), 1), e_set("x0", lst_get(v("x"), 0)))
e_if(op_ge(op_len(json_parse(v("x"))), 2), e_set("x1", lst_get(v("x"), 1)))

Function

Standard built-in functions of Python

Not supported. You can use more than 200 built-in functions provided by the DSL for Simple Log Service.

Function calls

Supported. Function calls that use parameter unpacking are not supported.

UDFs, such as def or lambda

Not supported. You can use more than 200 global processing and expression functions provided by the DSL for Simple Log Service. You can also combine these functions based on your business requirements.

Module

Import and use of the Python standard library

Not supported.

Creation of threads and processes

Not supported.

Import of third-party libraries

Not supported.

External network connection or external command call

Supported. The DSL for Simple Log Service provides built-in resource connectors.

Function categories

In standard mode of the DSL for Simple Log Service, all operations are performed by calling functions. The DSL provides more than 200 built-in functions, which are categorized into global processing functions and expression functions.

  • Global processing functions

    Global processing functions are used to receive, process, and return logs. Only global processing functions can be used to construct each step in a transformation rule.

  • Expression functions

    Expression functions are commonly used to receive specific parameters and return specific values. Expression functions can be combined and passed to global processing functions as parameters to define more flexible logic.

The following table describes global processing functions and expression functions.

Category

Construct a step

Receive a log

Return results

Modify a log

Combine functions

Global processing functions

Supported.

Logs are automatically received.

Zero to multiple logs are returned.

Supported. In most cases, logs can be modified.

Supported.

Expression functions

Not supported.

Supported by only a few expression functions. Most expression functions do not directly process logs.

Specific data structures are returned.

Not supported.

Supported.

Global processing functions

Global processing functions are used to receive, process, and return logs.

Note

Only a global processing function can be placed in the first line of each step.

The following syntax is used:

Global Processing Function 1(..Parameters....)
Global Processing Function 2(..Parameters....)
Global Processing Function 3(..Parameters....)
Global Processing Function 4(..Parameters....)

Global processing functions can be further categorized into flow control functions and event processing functions. The following table describes the functions.

Category

Description

Example

Flow control functions

The functions are used to manage processes, receive logs, and call other functions to process logs based on specific conditions.

e_if, e_switch, and e_if_else.

Event processing functions

The functions are used to transform logs. Zero to multiple logs are returned.

Examples:

  • e_drop_fields: discards log fields.

  • e_kv: extracts the key-value pairs of logs.

  • e_dict_map: enriches logs.

Transformation logic:

  • Basic processing

    The data transformation feature reads streaming data from a source Logstore and sends each log in a dictionary structure to specific functions. Then, the feature runs the functions that are specified in the transformation rule in sequence to process the events and writes the transformation results to specified destination Logstores.

    Note

    All fields and values of a log are sent as strings. For example, the raw log {"__time__": "1234567", "__topic__": "", "k1": "test"} is processed by the e_set("f1", 200) function. The function adds the f1 field whose value is set to 200 to the raw log. Then, the raw log is transformed into {"__time__": "1234567", "__topic__": "", "k1": "test", "f1": "200"}. In this log, the f1 field and the value 200 are strings.

    The event processing functions specified in a transformation rule are called in sequence. Each function receives and processes a log, and then returns a processed log.

    For example, the e_set("type", "test") function adds the type field whose value is set to test to a log. The next function receives and processes the processed log.

  • Condition evaluation

    • e_if: You can call the e_if function to add conditional expressions to process logs. If a log does not meet the specified condition, the corresponding operation is skipped. The e_if function implements the if logic.

      For example, the e_if(e_match("status", "200"), e_regex("data", "ret: \d+", "result")) function checks whether the value of the status field is 200. If the value is 200, the function extracts the result field from the data field by using the specified regular expression. Otherwise, no operation is performed.

    • e_if_else: This function works in a manner similar to the if_else function.

  • Processing termination

    • A step in a transformation rule may return no log. This indicates that the related log is deleted.

      For example, the e_if(str_islower(v("result")), e_drop()) function is used to check whether the value of the result field in a log is a string that consists of only lowercase characters. If the condition is evaluated to true, the log is discarded and the subsequent steps are not performed on this log. The system automatically processes the next log.

    • If a log is written to a destination Logstore, the processing is terminated. For example, if the e_output function is used to write a log to a destination Logstore and delete the log, the subsequent steps are not performed on this log.

      Note

      The e_coutput function copies the output log and the subsequent steps are performed on this log.

  • Log splitting for parallel processing

    A step in a transformation rule may return multiple logs. This indicates that the related log is split.

    For example, the e_split(data) function splits a log into two logs based on the value of the data field. If the value of the data field in the log is "abc, xyz", the log is split into two logs. In one log, the value of thedata field is abc. In the other log, the value of the data field is xyz.

    The logs that are generated after splitting are processed in the subsequent steps.

Expression functions

In addition to global processing functions, the DSL for Simple Log Service provides 200 expression functions that are used to receive specific parameters and return specific values. You can call an expression function or a combination of expression functions in a global processing function. The following syntax is used:

Global Processing Function 1(Expression Function 1(...), ...)
Global Processing Function 2(..., Expression Function 2(...), Expression Function 3(...), ...)

Expression functions can be categorized into event check functions, resource functions, control functions, and other expression functions. The following table describes the functions.

Category

Description

Example

Event check functions

The functions are used to receive logs, extract specific information, and then return the information without modifying the logs.

v: returns the value of a log field. e_search and e_match: checks whether the value of a field in a log meets a specified condition.

Resource functions

The functions are used to access on-premises or external resources, receive specific parameters, and return specific values. The data types of the return values include dictionary and table.

res_oss_file, res_rds_mysql, and res_log_logstore_pull.

Control functions

The functions are used to receive specific parameters and perform logical operations on expressions or condition-based control. The functions are also used to call other expression functions to return results.

op_and, op_or, op_not, op_if, and op_coalesce.

Other expression functions

The functions are used to receive specific parameters or the results of other functions and return specific values.

String functions, date and time functions, and conversion functions.