This topic describes Simple Log Service Processing Language (SPL), including its implementation, syntax, and instruction expressions.
Implementation
Simple Log Service supports SPL in the following scenarios: real-time consumption, scan-based query, and Logtail collection. The following figure shows how SPL works.
For more information about the capabilities that are supported by SPL in different scenarios, see SPL in different scenarios. For more information about the Alibaba Cloud regions where SPL is supported, see Supported regions.
Limits
Category | Item | Real-time consumption | Scan-based query | Logtail collection |
SPL complexity | Number of script pipeline levels | 16 | 16 | 16 |
Script length | 64 KB | 64 KB | 64 KB | |
SPL runtime | Memory size Important For more information, see Handle errors. | 1 GB | 2 GB | 50 MB |
Timeout period Important For more information, see Handle errors. | 1 second | 2 seconds | 1 second |
SPL syntax
SPL statements
An SPL statement supports multi-level data processing, uses a vertical bar (|) as the pipeline symbol for connection, and ends with a semicolon (;).
Syntax
<data-source> | <spl-expr> | <spl-expr> ;
Parameters
Parameter
Description
data-source
The data source, including a Logstore and an SPL-defined dataset.
Example:
* | project status, body
.For more information about the data sources that are supported by SPL in different scenarios, see SPL in different scenarios.
spl-expr
The data processing expression. For more information, see SPL instruction expressions.
Syntax symbols
Symbol | Description |
* | The placeholder that allows you to specify a Logstore as the data source of your SPL statement. |
. | The keyword of the SPL syntax if an SPL statement starts with a period (.). |
| | The SPL pipeline symbol, which is used to introduce an SPL instruction expression. Format: |
; | The end identifier of an SPL statement. This symbol is optional in a single statement or in the last statement among multiple statements. |
'...' | The quotes that are used to enclose a string constant. |
"..." | The quotes that are used to enclose a field name or a field name pattern. |
-- | The comment symbol for single-line content. |
/*...*/ | The comment symbol for multi-line content. |
SPL data types
The following table describes the data types that are supported by SPL.
Category | Data type | Description |
Basic numeric data types | BOOLEAN | The BOOLEAN data type. |
TINYINT | An integer with a width of 8 bits. | |
SMALLINT | An integer with a width of 16 bits. | |
INTEGER | An integer with a width of 32 bits. | |
BIGINT | An integer with a width of 64 bits. | |
HUGEINT | An integer with a width of 128 bits. | |
REAL | A variable-precision floating-point number with a width of 32 bits. | |
DOUBLE | A variable-precision floating-point number with a width of 64 bits. | |
TIMESTAMP | A UNIX timestamp that is accurate to the nanosecond. | |
DATE | The date data type. Format: YYYY-MM-DD. | |
VARCHAR | The variable-length character data type. | |
VARBINARY | The variable-length binary data type. | |
Structured numeric data types | ARRAY | The array data type. Brackets ( Example: |
MAP | The dictionary data type. A dictionary key must be of a basic numeric data type. A dictionary value can be of an arbitrary data type. Brackets ( Example: | |
JSON data type | JSON | The JSON data type. |
For more information about how to convert data types during SPL-based data processing, see General references.
SPL instruction expressions
Instruction expression syntax
cmd -option=<option> -option ... <expression>, ... as <output>, ...
Parameters
Parameter | Description |
cmd | The name of the instruction. |
option | The following parameter formats are supported:
|
expression | Required. The processing logic on the data source. You do not need to specify a parameter name. The position of the <expression> parameter must comply with the definition of the instruction. The following types of expressions are supported:
|
output | The output field that contains the processing result. Example: |
Instructions
The following table describes the instructions that are supported by SPL.
Category | Instruction | Description |
Field processing instructions | project | Retains the fields that match the specified pattern and renames the specified fields. |
project-away | Removes the fields that match the specified pattern and retains all other fields as they are. | |
project-rename | Renames the specified fields and retains all other fields as they are. | |
SQL calculation instructions on structured data | extend | Creates fields based on the result of SQL expression-based data calculation. For more information about the SQL functions that are supported by SPL, see SPL-supported SQL functions. |
where | Filters data based on the result of SQL expression-based data calculation. For more information about the SQL functions that are supported by SPL, see SPL-supported SQL functions. | |
Extraction instructions on semi-structured data | parse-csv | Extracts information in the CSV format from the specified field. |
parse-json | Extracts the first-layer JSON information from the specified field. | |
parse-regexp | Extracts the information that matches groups in the specified regular expression from the specified field. |