Use SPL to query and analyze logs in SLS - Simple Log Service

Use SPL (SLS Processing Language) to extract structured data, process fields, and filter logs. Simple Log Service supports multi-level pipeline cascading to chain SPL operations and export processed results.

SPL execution modes

SPL runs in index mode when all queried fields are indexed with analytics enabled. Otherwise, it falls back to scan mode. Scan query syntax.

Basic syntax

The scan-based query supports SPL syntax. SPL statements allow you to extract structured information, conduct field operations, and filter data from the raw data read. It also supports multi-level pipeline cascades. The first-level pipeline is the index filter condition, followed by SPL instructions in subsequent levels. The final results are data processed by SPL.

Index-based search statement | <spl-cmd> ... | <spl-cmd> ...

Log sample

Raw fields: labeled [R], applicable to scan search.
Index fields: labeled [I], applicable to index search.

[I] __topic__: nginx-access-log
[I] Status: 200
[I] Host: api.abc.com
[R] Method: PUT
[R] ClientIp: 192.168.1.1
[R] Payload: {"Item": "1122", "UserId": "112233", "Operation": "AddCart"}
[R] BeginTime: 1705029260
[R] EndTime: 1705028561
[R] RT: 87
[R] Uri: /request/path-3/file-1
[R] UserAgent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; ar) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4

Examples

Note

In SPL statements, constant strings are enclosed in single quotes ('), such as * | where ClientIp = '192.168.1.1'.
If a field name includes special characters, enclose it in double quotes ("), such as * | project-away "user-agent".

Filter by different conditions

Equality comparison

Status: 200 | where ClientIp = '192.168.1.1'

Case-insensitive search

__topic__: nginx-access-log | where lower(Method) != 'put'

Fuzzy match

Status: 200 | where UserAgent like '%Macintosh%'

Numerical comparison
Note that the default field type is varchar. Convert the type to bigint before performing numerical comparisons.
```
Status: 200 |  where cast(RT as bigint) > 50
```

Regular expression match

# Find URIs that contain "path-number"
Status: 200 | where regexp_like(Uri, 'path-\d+')

Calculate new fields

Calculate new fields from existing ones using the extend instruction.

Extract fields using regular expressions

# Extract the file number from the Uri field
* not Status: 200 | extend fileNumber=regexp_extract(Uri, 'file-(\d+)', 1)

Extract fields from JSON

Status:200 | extend Item = json_extract_scalar(Payload, '$.Item')

Extract fields based on a separator

Status:200 | extend urlParam=split_part(Uri, '/', 3)

Calculate new fields from multiple field values

# Calculate the time difference based on BeginTime and EndTime
Status:200 | extend timeRange = cast(BeginTime as bigint) - cast(EndTime as bigint)

Retain, remove, and rename fields

Keep only specific fields and remove the rest
```
Status:200 | project Status, Uri
```
Remove specific fields and keep the rest
```
Status:200 | project-away UserAgent
```
Rename fields
```
Status:200 | project-rename Latency=RT
```

Expand unstructured data

Expand all fields in JSON

# Filter non-empty Payloads and expand all JSON fields
__topic__: nginx-access-log | where Payload is not null | parse-json Payload

Expand JSON fields and discard the original ones

status:200 
| parse-json body 
| project-away body

Extract multiple fields using regular expressions

Status:200 | parse-regexp Uri, 'path-(\d+)/file-(\d+)' as pathIndex, fileIndex

Multi-level pipeline cascade

You can perform all the operations mentioned above in the same search statement using multi-level pipeline cascades, executed in sequence.

Status:200 
| where Payload is not null 
| parse-json Payload 
| project-away Payload 
| where Host='api.qzzw.com' and cast(RT as bigint) > 80 
| extend timeRange=cast(BeginTime as bigint) - cast(EndTime as bigint)
| where timeRange > 500
| project UserId, Uri

Index mode vs. scan mode

SPL automatically selects the execution mode based on whether the queried fields are indexed with analytics enabled.

Item	Index mode SPL	Scan mode SPL
Is index configuration required?	Yes. All queried fields must have field indexes with analytics enabled.	Not required. Important The query statement before the first pipe (\|) still requires an index.
Performance	Strong	Moderate
Can I use random pagination?	Supported.	Not supported.
Log histogram	Based on query results.	Based on query results and scan progress.
Operators and functions	SPL instructions and functions and SPL-supported SQL functions.	SPL instructions and functions and SPL-supported SQL functions.
Field types	All fields are treated as text. Data type conversion.	All fields are treated as text. Data type conversion.
Result set size	Configurable in the console or via SDK. Maximum: 100 logs.	The scan stops and returns results when any of these conditions is met: Returned log count reaches the specified limit. Configurable in the console or via SDK. Scanned log count exceeds the single-scan maximum (default: 100,000 from the query result set). The scan runs for more than 45 seconds.
Fees	Charged for index traffic and index storage. Billable items of the pay-by-feature billing method. SPL query execution incurs no additional fees.	Charged for the scanned data volume, which is the data matched by the scan after the initial index query.

Procedure

Important

Before querying logs, ensure you have collected logs and created an index. An index sorts one or more columns of log data for efficient retrieval. Create an index.

Console

Log on to the Simple Log Service console. On the query and analysis page of your Logstore, run a query. Quick start for log query and analysis.

Example

Assume 10 million raw logs exist. The query Status:200 | where Category like '%xx%' returns 1,000 logs matching both Status:200 and where Category like '%xx%'. The histogram shows the distribution of these logs over time.

API

Call the GetLogs or GetLogsV2 operation to query logs. GetLogs returns uncompressed results; GetLogsV2 returns compressed results.