Domain-specific language (DSL) for Log Service is a Python-compatible scripting language that is used for data transformation in Log Service. DSL for Log Service is developed based on Python and provides more than 200 built-in functions that can be used to simplify data transformation jobs.

Flexible orchestration

You can use DSL for Log Service to edit functions in a flexible manner and combine functions to implement complex logic in most data transformation scenarios.

Dynamic distribution

You can use DSL for Log Service to distribute data to different Logstores based on specific logic and your business requirements. The names of the Logstores can be obtained by using dynamic computing or from external resources such as Object Storage Service (OSS) buckets.

Data enrichment

  • You can use DSL for Log Service to obtain data for enrichment from local or external resources, such as OSS buckets and ApsaraDB RDS for MySQL instances.
  • You can use DSL for Log Service to perform regular mapping for dictionaries and tables and advanced mapping for tables.
  • You can use DSL for Log Service to automatically refresh external resources that are loaded.

Global processing functions

DSL for Log Service provides approximately 30 global processing functions. You can configure the parameters of global processing functions to control processing operations. Global processing functions accept the results of expression functions as parameters. Flow control functions are a type of global processing function and can be used together with expression functions and the following types of global processing functions:
  • Flow control functions
    • You can control processes based on conditions by using functions such as e_if_else, e_if, e_switch, and e_compose.
    • You can use simple search functions, such as e_search, to process different types of logs in a flexible manner.
  • Event processing functions

    You can discard, retain, split, write, and replicate events.

  • Field processing functions

    You can retain, delete, and rename fields.

  • Value extraction functions
    • You can extract values or key-value pairs from fields based on regular expressions, Grok patterns, syslog protocols, quotes, key-value pair delimiters, and delimiters such as commas (,), vertical bars (|), and tabs (\t).
    • You can extract and enrich JSON data.
  • Mapping and enrichment functions
    • You can map or search for data based on a dictionary or a table.
    • You can obtain information about a dimension table that is used to enrich data from resources such as rule configurations, external OSS buckets, and ApsaraDB RDS for MySQL instances.
    • You can use a function to automatically refresh external resources based on full or incremental change logs.
  • Value-added content functions

    You can enrich the information about some log fields. For example, you can obtain threat intelligence for an IP address and store the threat intelligence to log fields for log analysis.

Expression functions

DSL for Log Service provides more than 200 built-in expression functions to convert events or affect the results of the global processing functions. The expression functions are suitable for most data transformation scenarios. DSL for Log Service provides the following expression functions:
  • Event check functions

    DSL for Log Service provides a condition-based filtering mechanism that uses Lucene-like syntax, complete regular expressions, strings, generic characters, numeric value comparison, and logical operators such as AND, OR, and NOT.

  • Operator functions

    You can extract, control, and compare field values. You can also perform container evaluation and operations on multiple fields.

  • Conversion functions

    You can convert the values of basic data types. You can also convert numbers, dictionaries, and lists.

  • Arithmetic functions

    You can perform basic, multi-value, and mathematical calculations. You can also perform operations based on mathematical parameters.

  • String functions

    You can encode, decode, sort, reverse, replace, normalize, search, evaluate, truncate, and format multiple fields. You can also perform evaluation based on character sets.

  • Date and time functions

    You can convert date and time values. You can obtain date and time attributes, date and time values, UNIX timestamps, and date and time strings. You can also modify and compare date and time values.

  • Regular expression functions

    You can extract, match, evaluate, replace, and truncate fields.

  • Grok function

    DSL for Log Service provides more than 400 built-in Grok patterns. Grok patterns can be replaced.

  • Structured data functions

    You can extract and filter JSON, Protobuf, and XML data.

  • IP address parsing functions

    You can parse IP addresses and convert data.

  • Encoding and decoding functions

    You can encode and decode text in the SHA1, SHA256, SHA512, MD5, HTML, URL, or Base64 format.