Collecting text logs using SPL - Simple Log Service - Alibaba Cloud Documentation Center

This topic explains how to use SPL to replicate the functionality of processing plugins.

Background information

SPL vs. native plugins

Regex parsing

Sample log:

127.0.0.1 - - [07/Jul/2022:10:43:30 +0800] "POST /PutData?Category=YunOsAccountOpLog" 0.024 18204 200 37 "-" "aliyun-sdk-java"

Regex parsing

SPL

The Data Parsing (Regex Mode) plugin uses the regular expression ([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\"]*)\" \"([^\"]*)\" to extract the following fields: ip,time,method,url,request_time,request_length,status,length,ref_url,browser. For more information, see Collect text logs in regex mode.

The SPL statement is: * | parse-regexp content, '([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\"]*)\" \"([^\"]*)\"' as ip, time, method, url, request_time, request_length, status, length, ref_url, browser | project-away content. In this statement, the SPL instructions and functions discard the original content field, and the SPL instructions and functions extract fields.

Output preview

{
    "ip": "127.0.0.1",
    "time": "07/Jul/2022:10:43:30",
    "method": "POST",
    "url": "/PutData?Category=YunOsAccountOpLog",
    "request_time": "0.024",
    "request_length": "18204",
    "status": "200",
    "length": "37",
    "ref_url": "-",
    "browser": "aliyun-sdk-java",
    "__time__": "1713184059"
}

Delimiter parsing

Sample log:

127.0.0.1,07/Jul/2022:10:43:30 +0800,POST,PutData Category=YunOsAccountOpLog,0.024,18204,200,37,-,aliyun-sdk-java

Delimiter parsing

SPL

The Data Parsing (Delimiter Mode) plugin parses the log by using a comma (,) as the delimiter. For more information, see Collect text logs from servers.

The SPL statement is: *| parse-csv content as ip, time, method, url, request_time, request_length, status, length, ref_url, browser | project-away content. In this statement, the SPL instructions and functions discard the original content field, and the SPL instructions and functions extract fields.

Output preview

{
    "ip": "127.0.0.1",
    "time": "07/Jul/2022:10:43:30 +0800",
    "method": "POST",
    "url": "PutData?Category=YunOsAccountOpLog",
    "request_time": "0.024",
    "request_length": "18204",
    "status": "200",
    "length": "37",
    "ref_url": "-",
    "browser": "aliyun-sdk-java",
    "__time__": "1713231487"
}

JSON parsing

Sample log:

{"url": "POST /PutData?Category=YunOsAccountOpLog HTTP/1.1","ip": "10.200.98.220", "user-agent": "aliyun-sdk-java","request": "{\"status\":\"200\",\"latency\":\"18204\"}","time": "07/Jul/2022:10:30:28"}

JSON parsing

SPL

For more information, see Collect text logs in JSON mode.

Select Data Parsing (JSON Mode) plugin for the processing plugin type, and set the original field to content.

The SPL statement is: *| parse-json content| project-away content. In this statement, SPL instructions and functions discard the original content field, and SPL instructions and functions extract fields.

Output preview

{
    "url": "POST /PutData?Category=YunOsAccountOpLog HTTP/1.1",
    "ip": "10.200.98.220",
    "user-agent": "aliyun-sdk-java",
    "request": "{\"status\":\"200\",\"latency\":\"18204\"}",
    "time": "07/Jul/2022:10:30:28"
}

Regex and time parsing

Sample log:

127.0.0.1 - - [2024-11-05T15:47:05 +0800] "POST /PutData?Category=YunOsAccountOpLog" 0.024 18204 200 37 "-" "aliyun-sdk-java"

Regex and time parsing

SPL

The Data Parsing (Regex Mode) plugin uses the regular expression ([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\"]*)\" \"([^\"]*)\" to extract the following fields: ip,time,method,url,request_time,request_length,status,length,ref_url,browser. For more information, see Collect text logs in regex mode.
Use the Time-processing plugins. Set the Original Field to time and the Time Format to %Y-%m-%dT%H:%M:%S.

The SPL statement is: * | parse-regexp content, '([\d\.]+) \S+ \S+ \[(\S+)\] \"(\w+) ([^\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\"]*)\" \"([^\"]*)\"' as ip, time, method, url, request_time, request_length, status, length, ref_url, browser| extend ts=date_parse(time, '%Y-%m-%dT%H:%i:%S')| extend __time__=cast(to_unixtime(ts) as INTEGER)-28800| project-away ts| project-away content. In this statement, the SPL instructions and functions discard the original content field, the SPL instructions and functions extract fields, and date_parse parses the time in the log.

Regex parsing and data filtering

Sample log:

127.0.0.1 - - [2024-11-05T15:47:05 +0800] "POST /PutData?Category=YunOsAccountOpLog" 0.024 18204 200 37 "-" "aliyun-sdk-java"

Regex parsing and data filtering

SPL

The Data Parsing (Regex Mode) plugin uses the regular expression ([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\"]*)\" \"([^\"]*)\" to extract the following fields: ip,time,method,url,request_time,request_length,status,length,ref_url,browser. For more information, see Collect text logs in regex mode.
Use the Data filtering plugin and add the status and method fields to the whitelist.
The value of the method field in the whitelist matches ^(POST|PUT)$, and the value of the status field matches ^200$. These conditions are evaluated with a logical AND, so a log entry is collected only if all conditions are met.

The SPL statement is:*| parse-regexp content, '([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\"]*)\" \"([^\"]*)\"' as ip, time, method, url, request_time, request_length, status, length, ref_url, browser| project-away content| where regexp_like(method, '^(POST|PUT)$') and regexp_like(status, '^200$')。Among them, SPL instructions and functionsdiscard the original contentfield，SPL instructions and functions extract fields，and the regexp_like functionmatches data that conforms to regular expressions.

Output preview

{
    "ip": "127.0.0.1",
    "time": "2024-11-05T15:47:05",
    "method": "POST",
    "url": "/PutData?Category=YunOsAccountOpLog",
    "request_time": "0.024",
    "request_length": "18204",
    "status": "200",
    "length": "37",
    "ref_url": "-",
    "browser": "aliyun-sdk-java",
    "__time__": "1713238839"
}

Data masking

Sample log:

{"account":"1812213231432969","password":"04a23f38"}

Data masking

SPL

Use the Data masking and encryption plugins to mask the password field.

The original field is content, the desensitization method is const, the replacement string is **, the expression for preceding content is password".", the expression for the content to be replaced is [^"]+, and the Replace all matches option is selected.

The SPL statement is: *| parse-regexp content, 'password":"(\S+)"' as password| extend content=replace(content, password, '******'). In this statement, SPL instructions and functions replace the original content field, SPL instructions and functions extract a field, and the replace function replaces the value with masked data.

Output preview

{
    "content": "{"account":"1812213231432969","password":"******"}"
}

SPL vs. extended plugins

Add field

Sample log:

this is a test log

Add field

SPL

By default, the log is stored in the content field. Use the Field processing plugins to add the field service: A.

The SPL statement is: * | extend service='A'. This statement uses SPL instructions and functions to add the service: A field.

Output preview

{
    "content": "this is a test log",
    "service": "A"
}

JSON parsing and drop field

Sample log:

{"key1": 123456, "key2": "abcd"}

JSON parsing and drop field

SPL

Collect text logs in JSON mode.
Set the processing plugin type to Data Parsing (JSON Mode) plugin and the original field to content.
Field processing plugins.
Select Drop Field plugin for the processing plugin type, and set the drop field name to key1.

The SPL statement is: *| parse-json content| project-away content| project-away key1. In this statement, the SPL instructions and functions discard the original fields content and key1, and the SPL instructions and functions extract fields.

Output preview

{
    "key2": "abcd"
}

JSON parsing and rename fields

Sample log:

{"key1": 123456, "key2": "abcd"}

JSON parsing and rename fields

SPL

Collect text logs in JSON mode.
For the processing plugin type, select Data Parsing (JSON Mode) plugin, and set the original field to content.
Field processing plugins
For the processing plugin, select the Rename Fields plugin, set the original field to key1, and set the result field to new_key1.

The SPL statement is: *| parse-json content| project-away content| project-rename new_key1=key1. In this statement, the SPL instructions and functions discard the original field content, the SPL instructions and functions extract fields, and the SPL instructions and functions rename the field key1 to new_key1.

Output preview

{
    "new_key1": "123456",
    "key2": "abcd"
}

JSON parsing and log filtering

Sample log:

{"ip": "10.**.**.**", "method": "POST", "browser": "aliyun-sdk-java"}
{"ip": "10.**.**.**", "method": "POST", "browser": "chrome"}
{"ip": "192.168.**.**", "method": "POST", "browser": "aliyun-sls-ilogtail"}

JSON parsing and log filtering

SPL

Collect text logs in JSON mode.
For the processing plugin type, select Data Parsing (JSON Mode) plugin, and set the original field to content.
Data filtering plugin.
Select Filter Logs with Regular Expression (Match Log Field Values) plugin as the processing plugin type. In the log collection rule, the method field matches POST and the ip field matches 10\.. In the log discarding rule, the browser field matches aliyun.*.

The SPL statement is: *| parse-json content| project-away content| where regexp_like(ip, '10\..*') and regexp_like(method, 'POST') and not regexp_like(browser, 'aliyun.*'). In this statement, SPL instructions and functions discard the original content field, SPL instructions and functions extract fields, and the regexp_like function determines whether the conditions are met.

Output preview

{
    "ip": "10.**.**.**",
    "method": "POST",
    "browser": "chrome"
}

JSON parsing and field value mapping

Sample log:

{"_ip_":"192.168.*.*","Index":"900000003"}
{"_ip_":"255.255.**.**","Index":"3"}

JSON parsing and field value mapping

SPL

Collect text logs in JSON mode.
Select Data Parsing (JSON Mode) plugin for the processing plugin type, and set the original field to content.
Field processing plugins.
Set the processing plugin type to Field Value Mapping plugin, the original field to ip, and the result field to processed_ip. In the mapping dictionary, 192.168.*.* is mapped to default login, and 127.0.*.1 is mapped to *LocalHost-LocalHost.

The SPL statement is: *| parse-json content| project-away content| extend _processed_ip_= CASE WHEN _ip_ = '127.0.*.*' THEN 'LocalHost-LocalHost' WHEN _ip_ = '192.168.*.*' THEN 'default login' ELSE 'Not Detected' END. In this statement, SPL instructions and functions extract fields, SPL instructions and functions discard the original field content, and SPL instructions and functions create a new field.

Output preview

{
    "_ip_": "192.168.*.*",
    "Index": "900000003",
    "_processed_ip_": "default login"
}

String replacement

Sample log:

hello,how old are you? nice to meet you

String replacement

SPL

Use the Field processing plugins to replace "how old are you?" with an empty value.

The SPL statement is: *| extend content=replace(content, 'how old are you?', ''). This statement uses SPL instructions and functions to replace how old are you? with an empty value.

Output preview

{
    "content": "hello, nice to meet you"
}

Data encoding

Sample log:

this is a test log

BASE64 encoding

BASE64 encoding

SPL

For more information, see Data masking and encryption plugins.

For Processing Plugin Type, select BASE64 (encode), set Original Field to content, and set Result Field to content1.

The SPL statement is: *| extend content1=to_base64(cast(content as varbinary)). In this statement, SPL instructions and functions are used to add the content1 field, and the to_base64 function is used to encode the data in BASE64 format.

Output preview

{
    "content": "this is a test log",
    "content1": "dGhpcyBpcyBhIHRlc3QgbG9n"
}

MD5 encoding

MD5

SPL

For more information, see Data masking and encryption plugins.

The processing plugin type is MD5, the original field is content, and the result field is test.

The SPL statement is: *| extend test=lower(to_hex(md5(cast(content as varbinary)))). This statement uses SPL instructions and functions to add the test field, and uses the md5 function to generate the MD5 hash of the original content.

Output preview

{
    "content": "this is a test log",
    "test": "e7d7063444dd5593855de31741584288"
}

Additional SPL capabilities

Mathematical calculations

Sample log
```
4
```

SPL statement

The cast function converts data types. For more information about the power, round, and sqrt functions, see Mathematical functions.

*
| extend val = cast(content as double)
| extend power_test = power(val, 2)
| extend round_test = round(val)
| extend sqrt_test = sqrt(val)

Output preview

{
    "content": "4",
    "power_test": 16.0,
    "round_test": 4.0,
    "sqrt_test": 2.0,
    "val": 4.0
}

URL operations

URL encoding and decoding

Sample log

https://home.console.alibabacloud.com/home/dashboard/ProductAndService

SPL statement
For more information about the url_encode and url_decode functions, see URL functions.
```
*
| extend encoded = url_encode(content)
| extend decoded = url_decode(encoded)
```

Output preview

{
    "content": "https://home.console.alibabacloud.com/home/dashboard/ProductAndService",
    "decoded": "https://home.console.alibabacloud.com/home/dashboard/ProductAndService",
    "encoded": "https%3A%2F%2Fhome.console.alibabacloud.com%2Fhome%2Fdashboard%2FProductAndService"
}

URL extraction

Sample log

https://sls.console.alibabacloud.com:443/lognext/project/dashboard-all/logsearch/nginx-demo?accounttraceid=d6241a173f88471c91d3405cda010ff5ghdw

SPL statement

For more information about the functions used in this SPL statement, see URL functions.

*
| extend host = url_extract_host(content)
| extend query = url_extract_query(content)
| extend path = url_extract_path(content) 
| extend protocol = url_extract_protocol(content) 
| extend port = url_extract_port(content) 
| extend param = url_extract_parameter(content, 'accounttraceid')

Output preview

{
    "content": "https://sls.console.alibabacloud.com:443/lognext/project/dashboard-all/logsearch/nginx-demo?accounttraceid=d6241a173f88471c91d3405cda010ff5ghdw",
    "host": "sls.console.alibabacloud.com",
    "param": "d6241a173f88471c91d3405cda010ff5ghdw",
    "path": "/lognext/project/dashboard-all/logsearch/nginx-demo",
    "port": "443",
    "protocol": "https",
    "query": "accounttraceid=d6241a173f88471c91d3405cda010ff5ghdw"
}

Comparison and logical operators

Sample log
```
{"num1": 199, "num2": 10, "num3": 9}
```

SPL statement

The cast function converts data types, and SPL instructions and functions extract fields.

*
| parse-json content
| extend compare_result = cast(num1 as double) > cast(num2 as double) AND cast(num2 as double) > cast(num3 as double)

Output preview

{
    "compare_result": "true",
    "content": "{\"num1\": 199, \"num2\": 10, \"num3\": 9}",
    "num1": "199",
    "num2": "10",
    "num3": "9"
}