All Products
Search
Document Center

Simple Log Service:SPL overview

Last Updated:May 31, 2024

This topic describes Simple Log Service Processing Language (SPL), including its implementation, syntax, and instruction expressions.

Implementation

Simple Log Service supports SPL in the following scenarios: real-time consumption, scan-based query, and Logtail collection. The following figure shows how SPL works.

image.jpeg

For more information about the capabilities that are supported by SPL in different scenarios, see SPL in different scenarios. For more information about the Alibaba Cloud regions where SPL is supported, see Supported regions.

Limits

Category

Item

Real-time consumption

Scan-based query

Logtail collection

SPL complexity

Number of script pipeline levels

16

16

16

Script length

64 KB

64 KB

64 KB

SPL runtime

Memory size

Important

For more information, see Handle errors.

1 GB

2 GB

50 MB

Timeout period

Important

For more information, see Handle errors.

1 second

2 seconds

1 second

SPL syntax

SPL statements

An SPL statement supports multi-level data processing, uses a vertical bar (|) as the pipeline symbol for connection, and ends with a semicolon (;).

  • Syntax

    <data-source> | <spl-expr> | <spl-expr> ;
  • Parameters

    Parameter

    Description

    data-source

    The data source, including a Logstore and an SPL-defined dataset.

    Example: * | project status, body.

    For more information about the data sources that are supported by SPL in different scenarios, see SPL in different scenarios.

    spl-expr

    The data processing expression. For more information, see SPL instruction expressions.

Syntax symbols

Symbol

Description

*

The placeholder that allows you to specify a Logstore as the data source of your SPL statement.

.

The keyword of the SPL syntax if an SPL statement starts with a period (.).

|

The SPL pipeline symbol, which is used to introduce an SPL instruction expression. Format: | <spl-cmd> ....

;

The end identifier of an SPL statement. This symbol is optional in a single statement or in the last statement among multiple statements.

'...'

The quotes that are used to enclose a string constant.

"..."

The quotes that are used to enclose a field name or a field name pattern.

--

The comment symbol for single-line content.

/*...*/

The comment symbol for multi-line content.

SPL data types

The following table describes the data types that are supported by SPL.

Category

Data type

Description

Basic numeric data types

BOOLEAN

The BOOLEAN data type.

TINYINT

An integer with a width of 8 bits.

SMALLINT

An integer with a width of 16 bits.

INTEGER

An integer with a width of 32 bits.

BIGINT

An integer with a width of 64 bits.

HUGEINT

An integer with a width of 128 bits.

REAL

A variable-precision floating-point number with a width of 32 bits.

DOUBLE

A variable-precision floating-point number with a width of 64 bits.

TIMESTAMP

A UNIX timestamp that is accurate to the nanosecond.

DATE

The date data type. Format: YYYY-MM-DD.

VARCHAR

The variable-length character data type.

VARBINARY

The variable-length binary data type.

Structured numeric data types

ARRAY

The array data type. Brackets ([]) are used to access an element in an array based on the corresponding index. The index starts from 1.

Example: * | extend a = ARRAY[0, 1, 2] | extend b = a[1].

MAP

The dictionary data type. A dictionary key must be of a basic numeric data type. A dictionary value can be of an arbitrary data type. Brackets ([]) are used to access an element in a dictionary.

Example: * | extend a = MAP(ARRAY['foo', 'bar'], ARRAY[0, 1]) | extend b = a['foo'].

JSON data type

JSON

The JSON data type.

For more information about how to convert data types during SPL-based data processing, see General references.

SPL instruction expressions

Instruction expression syntax

cmd -option=<option> -option ... <expression>, ... as <output>, ...

Parameters

Parameter

Description

cmd

The name of the instruction.

option

Optional. The execution parameter of the instruction. You must specify a parameter name and start the name with a hyphen (-). You can specify multiple execution parameters in an arbitrary order.

The following parameter formats are supported:

  • -option=<option>: Specify the parameter in the key-value pair format to execute the instruction.

  • -option: Specify whether to turn on the switch of a feature. By default, all switches are turned off. If you configure this parameter, the switch of a feature is turned on.

expression

Required. The processing logic on the data source. You do not need to specify a parameter name. The position of the <expression> parameter must comply with the definition of the instruction.

The following types of expressions are supported:

  • SPL expression

    • An expression that is used to match a field name or a field name pattern, such as content. If you specify characters other than [a-zA-Z_], enclose the characters in double quotation marks (""). Example: "__tag__:__path__".

    • An expression that is used to assign a value to a field, such as level='INFO'. You can assign the extraction result of a regular expression to another field as a value. Example: msg=regex_extract(content, '\S+').

  • SQL expression

    • An expression that is used to return the result of data calculation. Example: cast(status as int)>=500.

output

The output field that contains the processing result. Example: | parse-csv content as a, b, c.

Instructions

The following table describes the instructions that are supported by SPL.

Category

Instruction

Description

Field processing instructions

project

Retains the fields that match the specified pattern and renames the specified fields.

project-away

Removes the fields that match the specified pattern and retains all other fields as they are.

project-rename

Renames the specified fields and retains all other fields as they are.

SQL calculation instructions on structured data

extend

Creates fields based on the result of SQL expression-based data calculation.

For more information about the SQL functions that are supported by SPL, see SPL-supported SQL functions.

where

Filters data based on the result of SQL expression-based data calculation.

For more information about the SQL functions that are supported by SPL, see SPL-supported SQL functions.

Extraction instructions on semi-structured data

parse-csv

Extracts information in the CSV format from the specified field.

parse-json

Extracts the first-layer JSON information from the specified field.

parse-regexp

Extracts the information that matches groups in the specified regular expression from the specified field.