Syntax introduction

• Standard mode: This is the default mode. In this mode, LOG DSL can be considered as a subset of Python. Except for basic data structures and expressions, other syntax rules are defined through functions.
• Advanced mode: In this mode, LOG DSL is fully compatible with Python. If you want to use LOG DSL in advanced mode, submit a ticket to request Alibaba Cloud to add you to the whitelist.

• Global processing functions

  Global processing functions receive, process, and return events. Only global processing functions can be used to construct the steps of processing rules.

• Expression functions

  Expression functions are general functions that receive specific parameters and return specific values. Expression functions can be combined and passed to global processing functions as parameters to define more flexible logic.

Global processing function 1 (... Parameters...)
Global processing function 2 (... Parameters...)
Global processing function 3 (... Parameters...)
Global processing function 4 (... Parameters...)

• Basic processing
  The data processing feature reads data from the source Logstore in streaming mode, and transmits each event in a dictionary structure to a processing rule. The system runs the event processing functions defined in the processing rule in sequence to process the event, and writes the final data processing result to the default destination Logstore.

  Note All fields and values of an event are transmitted as strings. Assume that the raw event is {"__time__": "1234567", "__topic__": "", "k1": "test"}. The e_set("f1", 200) function adds the f1 field to the event and sets its value to 200. The event becomes {"__time__": "1234567", "__topic__": "", "k1": "test", "f1": "200"}. In the event, both the f1 and 200 fields are strings.

  Event processing functions defined in a processing rule are called in sequence. Each function receives and processes each event and returns a processed event.

  For example, the e_set("type", "test") function adds the type field to each event and sets its value to test. The subsequent function receives the modified event.

• Condition-based judgment
  • e_if: You can call the e_if function to add condition-based judgment for some events. If an event does not meet the condition, the operation is skipped. The e_if function implements the if logic.

    Take the e_if(e_match("status", "200"), e_regex("data", "ret: \d+", "result")) function for example. This function checks whether the value of the status field is 200 in an event. If the result is true, a value is extracted from the data field based on the regular expression and assigned to a new field named result. If the result is false, no operation is performed.

  • e_if_else: This function implements the if_else logic.
• Process termination
  • Some steps may return no event. This indicates that the event is deleted.

    Take the e_if(str_islower(v("result")), e_drop()) function for example. If the value of the result field is a string consisting of only lowercase characters in an event, this event is deleted. In this case, the subsequent steps are not performed on this event. The system starts processing the next event automatically.

  • Writing an event to a Logstore may also indicate that the process ends. For example, the e_output function writes an event to a Logstore and deletes the event. In this case, the subsequent steps are not performed on this event.
    Note However, the e_coutput function writes an event to a Logstore but retains the original event. In this case, the event continues to be processed in subsequent steps.
• Event splitting for concurrent processing

  Some steps may return multiple events. This indicates that the original event is split.

  For example, the e_split(data) function splits an event with the data field to multiple events. Assume that the value of the data field is "abc, xyz". The event is split to two events, in which the values of the data field are abc and xyz, respectively.

  All events generated after the splitting are processed in the subsequent steps.

Global processing function 1(Expression function 1(...), ...)
Global processing function 2(..., Expression function 2(...), Expression function 3(...), ...)