Simple Log Service SPL - Simple Log Service - Alibaba Cloud Documentation Center

This topic describes Simple Log Service Processing Language (SPL), including its implementation, syntax, and instruction expressions.

SPL overview

Simple Log Service provides SPL (SLS Processing Language) statements for structured information extraction, field operations, and data filtering of raw data. Additionally, Simple Log Service offers multi-level pipeline cascade functionality, where the first level pipeline is the index filter condition, followed by multiple levels of SPL instruction pipelines, ultimately outputting the result data after processing. If you are familiar with SQL, you can refer to Comparison of SPL and SQL usage scenarios when using Simple Log Service SPL in different data processing scenarios.

How it works

Simple Log Service supports SPL in the following features: Collection by Logtail, ingest processor, rule-based data consumption, data transformation (new version), scan-based query and log analysis. The following figure shows how SPL works:

Note

For more information about the capabilities that are supported by SPL in different scenarios, see SPL in different scenarios.

SPL-SPL Doc - ch.jpeg

Limits

Category	Item	Logtail collection	Ingest processor	Real-time consumption	Data transformation (new version)	Scan query
SPL complexity	Number of script pipeline levels	16	16	16	16	16
SPL complexity	Script length	64 KB	64 KB	10 KB	10 KB	64 KB
SPL runtime	Memory size Important For more information, see Handle errors.	50 MB	1 GB	1 GB	1 GB	2 GB
SPL runtime	Timeout period Important For more information, see Handle errors.	1 second	5 seconds	5 seconds	5 seconds	2 seconds

SPL syntax

SPL statements

An SPL statement supports multi-level data processing, uses a vertical bar (|) as the pipeline symbol for connection, and ends with a semicolon (;). The following section describes the SPL syntax:

Syntax

<data-source> | <spl-expr> | <spl-expr> ;

Parameters

Parameter	Description	Example
data-source	The data source, including a Logstore and an SPL-defined dataset. For more information about the data sources that are supported by SPL in different scenarios, see SPL in different scenarios.	`* \| project status, body;`
spl-expr	The data processing expression. For more information, see SPL instruction expressions.	`* \| project status, body;`

Syntax symbols

The following table describes the syntax symbols used in SPL.

Symbol	Description	Example
*	The placeholder that allows you to specify a Logstore as the data source of your SPL statement.	Filter and categorize access logs based on status codes, and then output the results. SPL statement -- Define the SPL processing result as a named dataset src, which serves as the input for subsequent SPL expressions .let src = * \| where status=cast(status as BIGINT); -- Use the named dataset src as the input for an SPL expression to obtain the data whose status field is 5xx and define the SPL processing result as a dataset named err. Do not export the dataset. .let err = $src \| where status >= 500 \| extend msg='ERR'; -- Use the named dataset src as the input for an SPL expression to obtain the data whose status field is 2xx and define the SPL processing result as a dataset named ok. Do not export the dataset. .let ok = $src \| where status >= 200 and status < 300 \| extend msg='OK'; -- Export the named datasets err and ok $err; $ok; Input data `# Entry 1 status: '200' body: 'this is a test' # Entry 2 status: '500' body: 'internal error' # Entry 3 status: '404' body: 'not found'` Result `# Entry 1: Dataset err status: '500' body: 'internal error' msg: 'ERR' # Entry 2: Dataset ok status: '200' body: 'this is a test' msg: 'OK'`
.	The keyword of the SPL syntax if an SPL statement starts with a period (.).
\|	The SPL pipeline symbol, which is used to introduce an SPL instruction expression. Format: `\| <spl-cmd> ...`.
;	The end identifier of an SPL statement. This symbol is optional in a single statement or in the last statement among multiple statements.
'...'	The quotes that are used to enclose a string constant.
"..."	The quotes that are used to enclose a field name or a field name pattern.
--	The comment symbol for single-line content.
/.../	You can comment multiple lines of content.
$	The symbol for named dataset references. Format: `$<dataset-name>`.

SPL data types

The following table describes the log field data types supported by SPL:

Category	Data type	Description
Basic numeric data types	BOOLEAN	Boolean type.
	TINYINT	An integer with a width of 8 bits.
	SMALLINT	An integer with a width of 16 bits.
	INTEGER	An integer with a width of 32 bits.
	BIGINT	An integer with a width of 64 bits.
	HUGEINT	An integer with a width of 128 bits.
	REAL	A variable-precision floating-point number with a width of 32 bits.
	DOUBLE	A variable-precision floating-point number with a width of 64 bits.
	TIMESTAMP	A UNIX timestamp that is accurate to the nanosecond.
	DATE	The date data type. Format: YYYY-MM-DD.
	VARCHAR	The variable-length character data type.
	VARBINARY	The variable-length binary data type.
Structured numeric data types	ARRAY	The array type. Brackets (`[]`) are used to access an element in an array based on the corresponding index. The index starts from 1. Example: `* \| extend a = ARRAY[0, 1, 2] \| extend b = a[1]`.
Structured numeric data types	MAP	The dictionary type. A dictionary key must be of a basic numeric data type. A dictionary value can be of an arbitrary data type. Brackets (`[]`) are used to access an element in a dictionary. Example: `* \| extend a = MAP(ARRAY['foo', 'bar'], ARRAY[0, 1]) \| extend b = a['foo']`.
JSON data type	JSON	The JSON type.

For more information about how to convert data types during SPL-based data processing, see General references.

SPL instruction expressions

Instruction expression syntax

cmd -option=<option> -option ... <expression>, ... as <output>, ...

Parameters

Parameter	Description
cmd	The name of the instruction.
option	Optional. The execution parameter of the instruction. You must specify a parameter name and start the name with a hyphen `(-)`. You can specify multiple execution parameters in an arbitrary order. The following parameter formats are supported: `-option=<option>:` Specify the parameter in the key-value pair format to execute the instruction. -option: Specify whether to turn on the switch of a feature. By default, all switches are turned off. If you configure this parameter, the switch of a feature is turned on.
expression	Required. The processing logic on the data source. You do not need to specify a parameter name. The position of the <expression> parameter must comply with the definition of the instruction. The following types of expressions are supported: SPL expression An expression that is used to match a field name or a field name pattern, such as content. If you specify characters other than [a-zA-Z_], enclose the characters in double quotation marks (""). Example: `"__tag__:__path__"`. An expression that is used to assign a value to a field, such as `level='INFO'`. You can assign the extraction result of a regular expression to another field as a value. Example: `msg=regex_extract(content, '\S+')`. SQL expression An expression that is used to return the result of data calculation. Example: `cast(status as int)>=500`.
output	The output field that contains the processing result. Example: `\| parse-csv content as a, b, c`.

Instructions

SPL supports the following instructions. For more information, see SPL instructions.

Category	Name	Description
Control instruction	.let	Defines named datasets.
Field processing instructions	project	Retains the fields that match the specified pattern, and renames the specified fields. During instruction execution, all retain-related expressions are executed before rename-related ones.
	project-away	Removes the fields that match the specified pattern, and retains all other fields as they are.
	project-rename	Renames the specified fields, and retains all other fields as they are.
	expand-values	Expands a first-level JSON object for the specified field, and returns multiple result entries.
SQL instructions for structured data computation	extend	Creates fields based on the result of SQL expression-based data calculation. For more information, see SPL-supported SQL functions.
SQL instructions for structured data computation	where	Filters data based on the result of SQL expression-based data calculation, and retains the data that matches the specified SQL expression. For more information, see SPL-supported SQL functions.
Instructions for extracting semi-structured data	parse-regexp	Extracts the information that matches groups in the specified regular expression from the specified field.
	parse-csv	Extracts information in the CSV format from the specified field.
	parse-json	Extracts the first-layer JSON information from the specified field.
	parse-kv	Extracts the key-value pair information from the specified field.
Data transformation of the new version Data	pack-fields	Encapsulates log fields, serializes the fields in the JSON format, and exports the fields to a new field. This instruction is applicable to scenarios in which structured transmission is required, such as API request body construction.
	log-to-metric	The e_to_metric function converts logs to metrics that can be stored in a Metricstore.
	metric-to-metric	Processes existing time series data, such as adding, modifying, or removing tags.
Aggregation instructions	stats	Used for statistical analysis of logs, similar to aggregate functions in SQL (such as `COUNT`, `SUM`, and `AVG`). Performs statistical, grouping, and aggregation operations on specific fields in log data.
	sort	Sorts query results, supporting ascending (`asc`) or descending (`desc`) sorting of field values or statistical results. It is an important tool for quickly locating key data and generating ordered reports in log analysis.
	limit	Used to limit the number of log rows returned in query results. It is one of the core instructions for controlling data volume. Through `limit`, you can effectively prevent performance issues or resource waste caused by excessively large query results. Applicable to various scenarios such as log analysis and real-time monitoring.