Use SPL statements to collect text logs - Simple Log Service

This topic describes how to use Simple Log Service Processing Language (SPL) statements to collect logs. In this case, SPL statements function similar to processing plug-ins.

Background information

For more information about how to install processing plug-ins, see Overview of Logtail plug-ins for data processing.
For more information about how to use SPL statements to parse logs, see Use Logtail SPL to parse logs.

Comparison between SPL statements and native plug-ins

Comparison between SPL statements and the Data Parsing (Regex Mode) plug-in

Sample log:

127.0.0.1 - - [07/Jul/2022:10:43:30 +0800] "POST /PutData?Category=YunOsAccountOpLog" 0.024 18204 200 37 "-" "aliyun-sdk-java"

Data Parsing (Regex Mode) plug-in

SPL statement

Specify the ([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\"]*)\" \"([^\"]*)\" regular expression to parse data in regex mode and extract the following fields from the log: ip, time, method, url, request_time, request_length, status, length, ref_url, and browser. For more information, see Collect text logs in regex mode.

Specify the following SPL statement: * | parse-regexp content, '([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\"]*)\" \"([^\"]*)\"' as ip, time, method, url, request_time, request_length, status, length, ref_url, browser | project-away content. In the statement, the project-away instruction discards the content field and the parse-regexp instruction extracts the specified fields.

Output preview

{
    "ip": "127.0.0.1",
    "time": "07/Jul/2022:10:43:30",
    "method": "POST",
    "url": "/PutData?Category=YunOsAccountOpLog",
    "request_time": "0.024",
    "request_length": "18204",
    "status": "200",
    "length": "37",
    "ref_url": "-",
    "browser": "aliyun-sdk-java",
    "__time__": "1713184059"
}

Comparison between SPL statements and the Data Parsing (Delimiter Mode) plug-in

Sample log:

127.0.0.1,07/Jul/2022:10:43:30 +0800,POST,PutData Category=YunOsAccountOpLog,0.024,18204,200,37,-,aliyun-sdk-java

Data Parsing (Delimiter Mode) plug-in

SPL statement

Set the Processor Type parameter to Data Parsing (Delimiter Mode). Select Custom from the Delimiter drop-down list and specify commas (,) to separate fields. For more information, see Collect text logs from servers.

Specify the following SPL statement: *| parse-csv content as ip, time, method, url, request_time, request_length, status, length, ref_url, browser | project-away content. In the statement, the project-away instruction discards the content field and the parse-csv instruction extracts the specified fields.

Output preview

{
    "ip": "127.0.0.1",
    "time": "07/Jul/2022:10:43:30 +0800",
    "method": "POST",
    "url": "PutData?Category=YunOsAccountOpLog",
    "request_time": "0.024",
    "request_length": "18204",
    "status": "200",
    "length": "37",
    "ref_url": "-",
    "browser": "aliyun-sdk-java",
    "__time__": "1713231487"
}

Comparison between SPL statements and the Data Parsing (JSON Mode) plug-in

Sample log:

{"url": "POST /PutData?Category=YunOsAccountOpLog HTTP/1.1","ip": "10.200.98.220", "user-agent": "aliyun-sdk-java","request": "{\"status\":\"200\",\"latency\":\"18204\"}","time": "07/Jul/2022:10:30:28"}

Data Parsing (JSON Mode) plug-in

SPL statement

For more information, see Collect text logs in JSON mode.

Specify the following SPL statement: *| parse-json content| project-away content. In the statement, the project-away instruction discards the content field and the parse-json instruction extracts the specified fields.

Output preview

{
    "url": "POST /PutData?Category=YunOsAccountOpLog HTTP/1.1",
    "ip": "10.200.98.220",
    "user-agent": "aliyun-sdk-java",
    "request": "{"status":"200","latency":"18204"}",
    "time": "07/Jul/2022:10:30:28"
}

Comparison between SPL statements and the combination of the Data Parsing (Regex Mode) and Time Parsing plug-ins

Sample log:

127.0.0.1 - - [2024-11-05T15:47:05 +0800] "POST /PutData?Category=YunOsAccountOpLog" 0.024 18204 200 37 "-" "aliyun-sdk-java"

Combination of the Data Parsing (Regex Mode) and Time Parsing plug-ins

SPL statement

Specify the ([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\"]*)\" \"([^\"]*)\" regular expression to parse data in regex mode and extract the following fields from the log: ip, time, method,url, request_time, request_length, status, length, ref_url, and browser. For more information, see Collect text logs in regex mode.
Specify time as the original field and %Y-%m-%dT%H:%M:%S as the time format in the Time Parsing plug-in.

Specify the following SPL statement: * | parse-regexp content, '([\d\.]+) \S+ \S+ \[(\S+)\] \"(\w+) ([^\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\"]*)\" \"([^\"]*)\"' as ip, time, method, url, request_time, request_length, status, length, ref_url, browser| extend ts=date_parse(time, '%Y-%m-%dT%H:%i:%S')| extend __time__=cast(to_unixtime(ts) as INTEGER)-28800| project-away ts| project-away content. In the statement, the project-away instruction discards the content field, the parse-regexp instruction extracts the specified fields, and the date_parse instruction parses the log time.

Comparison between SPL statements and the combination of the Data Parsing (Regex Mode) and Data Filtering plug-ins

Sample log:

127.0.0.1 - - [2024-11-05T15:47:05 +0800] "POST /PutData?Category=YunOsAccountOpLog" 0.024 18204 200 37 "-" "aliyun-sdk-java"

Combination of the Data Parsing (Regex Mode) and Data Filtering plug-ins

SPL statement

Specify the ([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\"]*)\" \"([^\"]*)\" regular expression to parse data in regex mode and extract the following fields from the log: ip, time, method,url, request_time, request_length, status, length, ref_url, and browser. For more information, see Collect text logs in regex mode.
Add the status and method fields to the whitelist of the Data Filtering plug-in.

Specify the following SPL statement: *| parse-regexp content, '([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\"]*)\" \"([^\"]*)\"' as ip, time, method, url, request_time, request_length, status, length, ref_url, browser| project-away content| where regexp_like(method, '^(POST|PUT)$') and regexp_like(status, '^200$'). In the statement, the project-away instruction discards the content field, the parse-regexp instruction extracts the specified fields, and the regexp_like function returns data that matches the specified regular expression.

Output preview

{
    "ip": "127.0.0.1",
    "time": "2024-11-05T15:47:05",
    "method": "POST",
    "url": "/PutData?Category=YunOsAccountOpLog",
    "request_time": "0.024",
    "request_length": "18204",
    "status": "200",
    "length": "37",
    "ref_url": "-",
    "browser": "aliyun-sdk-java",
    "__time__": "1713238839"
}

Comparison between SPL statements and the Data Masking plug-in

Sample log:

{"account":"1812213231432969","password":"04a23f38"}

Data Masking plug-in

SPL statement

Specify password as the field to mask in the Data Masking plug-in.

Specify the following SPL statement: *| parse-regexp content, 'password":"(\S+)"' as password| extend content=replace(content, password, '******'). In the statement, the project-away instruction discards the content field, the parse-regexp instruction extracts the specified fields, and the replace function returns masked data.

Output preview

{
    "content": "{"account":"1812213231432969","password":"******"}"
}

Comparison between SPL statements and extended plug-ins

Comparison between SPL statements and the Add Field plug-in

Sample log:

this is a test log

Add Field plug-in

SPL statement

By default, the log is stored in the content field. Use the Add Field plug-in to add the service:A field to the log.

Specify the following SPL statement: * | extend service='A'. In the statement, the extend instruction adds the service:A field.

Output preview

{
    "content": "this is a test log",
    "service": "A"
}

Comparison between SPL statements and the combination of the Data Parsing (JSON Mode) and Drop Field plug-ins

Sample log:

{"key1": 123456, "key2": "abcd"}

Combination of the Data Parsing (JSON Mode) and Drop Field plug-ins

SPL statement

For more information, see Collect text logs in JSON mode.
For more information, see Drop fields.

Specify the following SPL statement: *| parse-json content| project-away content| project-away key1. In the statement, the project-away instruction discards the content and key1 fields and the parse-json instruction extracts the specified fields.

Output preview

{
    "key2": "abcd"
}

Comparison between SPL statements and the combination of the Data Parsing (JSON Mode) and Rename Fields plug-ins

Sample log:

{"key1": 123456, "key2": "abcd"}

Combination of the Data Parsing (JSON Mode) and Rename Fields plug-ins

SPL statement

For more information, see Collect text logs in JSON mode.
For more information, see Rename fields.

Specify the following SPL statement: *| parse-json content| project-away content| project-rename new_key1=key1. In the statement, the project-away instruction discards the content field, the parse-json instruction extracts the specified fields, and the project-rename instruction renames the key1 field new_key1.

Output preview

{
    "new_key1": "123456",
    "key2": "abcd"
}

Comparison between SPL statements and the combination of the Data Parsing (JSON Mode) and Filter Logs with Regular Expression (Match Log Field Values) plug-ins

Sample log:

{"ip": "10.**.**.**", "method": "POST", "browser": "aliyun-sdk-java"}
{"ip": "10.**.**.**", "method": "POST", "browser": "chrome"}
{"ip": "192.168.**.**", "method": "POST", "browser": "aliyun-sls-ilogtail"}

Combination of the Data Parsing (JSON Mode) and Filter Logs with Regular Expression (Match Log Field Values) plug-ins

SPL statement

For more information, see Collect text logs in JSON mode.
For more information, see Filter logs.

Specify the following SPL statement: *| parse-json content| project-away content| where regexp_like(ip, '10\..*') and regexp_like(method, 'POST') and not regexp_like(browser, 'aliyun.*'). In the statement, the project-away instruction discards the content field, the parse-json instruction extracts the specified fields, and the regexp_like function returns data that matches the specified regular expression.

Output preview

{
    "ip": "10.**.**.**",
    "method": "POST",
    "browser": "chrome"
}

Comparison between SPL statements and the combination of the Data Parsing (JSON Mode) and Field Value Mapping plug-ins

Sample log:

{"_ip_":"192.168.*.*","Index":"900000003"}
{"_ip_":"255.255.**.**","Index":"3"}

Combination of the Data Parsing (JSON Mode) and Field Value Mapping plug-ins

SPL statement

For more information, see Collect text logs in JSON mode.
For more information, see Map field values.

Specify the following SPL statement: *| parse-json content| project-away content| extend _processed_ip_= CASE WHEN _ip_ = '127.0.*.*' THEN 'LocalHost-LocalHost' WHEN _ip_ = '192.168.*.*' THEN 'default login' ELSE 'Not Detected' END. In the statement, the project-away instruction discards the content field, the parse-json instruction extracts the specified fields, and the extend instruction adds a field.

Output preview

{
    "_ip_": "192.168.*.*",
    "Index": "900000003",
    "_processed_ip_": "default login"
}

Comparison between SPL statements and the String Replacement plug-in

Sample log:

hello,how old are you?  nice to meet you

String Replacement plug-in

SPL statement

Use the String Replacement plug-in to replace how old are you? with NULL.

Specify the following SPL statement: *| extend content=replace(content, 'how old are you?', ''). In the statement, the extend instruction replaces how old are you? with NULL.

Output preview

{
    "content": "hello, nice to meet you"
}

Comparison between SPL statements and data encoding plug-ins

Sample log:

this is a test log

Base64 encoding

BASE64 (Encoding) plug-in

SPL statement

For more information, see Base64 encoding.

Specify the following statement: *| extend content1=to_base64(cast(content as varbinary)). In the statement, the extend instruction adds the content1 field and the to_base64 function encodes data in Base64.

Output preview

{
    "content": "this is a test log",
    "content1": "dGhpcyBpcyBhIHRlc3QgbG9n"
}

MD5 encoding

MD5 plug-in

SPL statement

For more information, see MD5 encoding.

Specify the following SPL statement: *| extend test=lower(to_hex(md5(cast(content as varbinary)))). In the statement, the extend instruction adds the test field and the md5 function computes the MD5 hash value for data.

Output preview

{
    "content": "this is a test log",
    "content1": "4f3c93e010f366eca78e00dc1ed08984"
}

New capabilities

Mathematical calculation

Sample log
```
4
```
SPL statement
The cast function converts the values of a field to a specified data type. For more information about the power, round, and sqrt functions, see Mathematical calculation functions.
```
*
| extend val = cast(content as double)
| extend power_test = power(val, 2)
| extend round_test = round(val)
| extend sqrt_test = sqrt(val)
```

Output preview

{
    "content": "4",
    "power_test": 16.0,
    "round_test": 4.0,
    "sqrt_test": 2.0,
    "val": 4.0
}

URL-based calculation

URL encoding and decoding

Sample log

https://homenew.console.aliyun.com/home/dashboard/ProductAndService

SPL statement
For more information about the url_encode and url_decode functions, see URL functions.
```
*
| extend encoded = url_encode(content)
| extend decoded = url_decode(encoded)
```

Output preview

{
    "content": "https://homenew.console.aliyun.com/home/dashboard/ProductAndService",
    "decoded": "https://homenew.console.aliyun.com/home/dashboard/ProductAndService",
    "encoded": "https%3A%2F%2Fhomenew.console.aliyun.com%2Fhome%2Fdashboard%2FProductAndService"
}

URL extraction

Sample log

https://sls.console.aliyun.com:443/lognext/project/dashboard-all/logsearch/nginx-demo?accounttraceid=d6241a173f88471c91d3405cda010ff5ghdw

SPL statement

For more information about the functions used in the following SPL statement, see URL functions.

*
| extend host = url_extract_host(content)
| extend query = url_extract_query(content)
| extend path = url_extract_path(content) 
| extend protocol = url_extract_protocol(content) 
| extend port = url_extract_port(content) 
| extend param = url_extract_parameter(content, 'accounttraceid')

Output preview

{
    "content": "https://sls.console.aliyun.com:443/lognext/project/dashboard-all/logsearch/nginx-demo?accounttraceid=d6241a173f88471c91d3405cda010ff5ghdw",
    "host": "sls.console.aliyun.com",
    "param": "d6241a173f88471c91d3405cda010ff5ghdw",
    "path": "/lognext/project/dashboard-all/logsearch/nginx-demo",
    "port": "443",
    "protocol": "https",
    "query": "accounttraceid=d6241a173f88471c91d3405cda010ff5ghdw"
}

Comparison and logical operators

Sample log
```
{"num1": 199, "num2": 10, "num3": 9}
```
SPL statement
The cast function converts the values of a field to a specified data type and the parse-json instruction extracts the specified fields.
```
*
| parse-json content
| extend compare_result = cast(num1 as double) > cast(num2 as double) AND cast(num2 as double) > cast(num3 as double
```

Output preview

{
    "compare_result": "true",
    "content": "{"num1": 199, "num2": 10, "num3": 9}",
    "num1": "199",
    "num2": "10",
    "num3": "9"
}