LOG domain specific language (DSL) is a Python-compatible script language that is used for the data processing feature of Log Service. Based on Python, LOG DSL provides over 200 built-in functions to simplify data processing.

Flexible orchestration

LOG DSL allows you to flexibly combine functions to implement complex logic.

Global processing functions

LOG DSL provides about 30 global processing functions. You can configure the parameters of global processing functions to control operations in data processing steps. Global processing functions can use expression functions as parameters. As a special type of global processing functions, process control functions can also use other global processing functions as parameters.
  • Process control functions
    • Allow you to control processes through condition-based judgment such as if-else, if, switch, and compose.
    • Allow you to call simple search functions such as e_search to process logs of different types flexibly.
  • Event processing functions

    Allow you to discard, retain, split, write, and replicate events.

  • Field processing functions

    Allow you to retain, delete, and rename fields.

  • Value assignment function

    Allows you to assign the results of any expression functions or their combinations as values to fields.

  • Value extraction functions
    • Allow you to extract values or key-value pairs from fields based on regular expressions, Grok patterns, Syslog standard, key-value pair delimiters, and value delimiters such as commas (,), vertical bars (|), and tabs (\t).
    • Allow you to extract and enrich JSON data.
  • Mapping and enrichment functions
    • Allow you to map fields to or search for data in dictionaries and tables.
    • Allow you to obtain dimension tables for enrichment from resources such as rule configuration data, Object Storage Service (OSS) buckets, Relational Database Service (RDS) databases, and Logstores.
    • Allow you to refresh external resources based on full or incremental logs.

Expression functions

LOG DSL provides over 200 built-in expression functions for you to convert events or control global processing functions. The expression functions cover most data processing scenarios. LOG DSL provides the following expression functions:
  • Event search function: allows you to search for events based on regular expressions, strings, wildcard characters, value comparison, and logical operations such as AND, OR, and NOT. The syntax is similar to that of Lucene.
  • Basic processing functions: support field value extraction, control, comparison, containment judgment, and multi-field operations.
  • Conversion functions: convert values among different types, such as numbers, dictionaries, and lists.
  • Arithmetic functions: support basic calculation, multi-value calculation, mathematical calculation, and mathematical parameters.
  • String functions: support multi-field operations and other operations such as encoding, decoding, sorting, reversing, replacing, normalizing, searching, judging, slicing, and formatting strings.
  • Date and time functions: convert date and time strings, obtain date and time attributes, obtain date and time, obtain UNIX timestamps, obtain date and time as strings, and modify and compare date and time.
  • Regular expression functions: extract, match, judge, replace, and split fields based on regular expressions.
  • Grok function: supports over 400 built-in Grok patterns and pattern replacement.
  • JSON, Protobuf, and XML functions: extract and filter data.
  • Encoding and decoding functions: encode and decode text in the SHA1, SHA256, SHA512, MD5, HTML, URL, and Base64 formats.

Dynamic distribution

LOG DSL can distribute data to different destination Logstores based on the specified logic. The names of the destination Logstores can be obtained through dynamic computing or from external resources.

Flexible data enrichment

  • Allows you to obtain data from local or external resources such as OSS buckets, RDS databases, and Logstores, and use the data to enrich logs.
  • Supports common mapping for tables and dictionaries and advanced mapping based on table search.
  • Automatically refreshes external resources that are loaded.