The Simple Log Service (SLS) domain-specific language (DSL) is a Python-compatible script for data transformation. The SLS DSL is built on Python and provides over 200 built-in functions to simplify data transformation tasks.
Flexible orchestration
You can use the SLS DSL to edit functions and combine various logic to address most data transformation scenarios.
Dynamic distribution
You can distribute data to different destination Logstores based on specific logic. The names of the destination Logstores can be dynamically calculated or retrieved from external sources, such as OSS buckets.
Flexible enrichment
Obtain enrichment data from local or external resources, such as OSS buckets and ApsaraDB RDS for MySQL instances.
Perform standard mapping for dictionaries and tables, and advanced mapping for tables.
Automatically refresh loaded external resources.
Global operation functions
The SLS DSL provides approximately 30 global operation functions that use parameters to control operations. These functions accept the results of expression functions as parameters. Control functions can be combined with expression functions and other step functions. The main types of functions include the following:
Create conditional branches using functions such as
if-else,if,switch, andcompose.Use simple search functions, such as
e_search, to process different types of logs.
Drop, keep, split, output, and copy events.
Keep, delete, and rename fields.
Field value extraction functions
Extract multiple values or key-value pairs from fields using regular expressions, GROK, KV, KV separators, CSV, TSV, PSV, or Syslog.
Extract and enrich JSON data.
Mapping and enrichment functions
Map or search based on dictionaries and tables.
Obtain dimension table information for enrichment from resources such as rule configurations, external OSS buckets, and ApsaraDB RDS for MySQL instances.
Automatically refresh external resources based on full or incremental change logs.
Enrich log field information. For example, you can obtain threat intelligence for an IP address and save this information to a log field to facilitate log analysis.
Expression functions
The SLS DSL provides over 200 built-in expression functions to transform events or control the behavior of global functions. These functions cover most data transformation needs. The main types of functions include the following:
These functions provide a conditional filtering mechanism that uses Lucene-like syntax, complete regular expressions, strings, wildcards, numeric comparisons, and logical combinations of AND, OR, and NOT.
These functions support field value retrieval, control, comparison, container checks, and multi-field operations.
These functions provide operations for basic type conversion, number conversion, and dictionary and list conversion.
These functions provide support for basic calculations, multi-value calculations, mathematical calculations, and mathematical parameters.
These functions provide multi-field operations, including encoding, decoding, sorting, reversing, replacing, standard normalization, searching, splitting, formatting, and character set checking.
These functions provide support for converting dates and times, retrieving date and time properties, retrieving the current date and time, retrieving a UNIX timestamp, retrieving a date and time string, and modifying and comparing date and time values.
These functions provide operations to extract, match, evaluate, replace, and split fields.
These functions provide over 400 built-in GROK patterns and support GROK pattern replacement.
Specific structured data functions
These functions support extracting and filtering data from specific structures such as JSON, Protobuf, and XML.
These functions provide support for IP address resolution and conversion.
Encoding and decoding functions
These functions support encoding and decoding text in formats such as SHA1, SHA256, SHA512, MD5, HTML, URL, and Base64.