使用SPL採集文本日誌 - Simple Log Service

本文主要介紹如何使用SPL實作類別似於處理外掛程式採集日誌的功能。

背景資訊

SPL與原生外掛程式對比

正則解析對比

日誌範例：

127.0.0.1 - - [07/Jul/2022:10:43:30 +0800] "POST /PutData?Category=YunOsAccountOpLog" 0.024 18204 200 37 "-" "aliyun-sdk-java"

正則解析

SPL

正則解析的Regex為：([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\"]*)\" \"([^\"]*)\"，日誌提取欄位為：ip,time,method,url,request_time,request_length,status,length,ref_url,browser。具體操作請參見使用正則模式採集文本日誌。

SPL語句為：* | parse-regexp content, '([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\"]*)\" \"([^\"]*)\"' as ip, time, method, url, request_time, request_length, status, length, ref_url, browser | project-away content。此語句中project-away捨棄原有欄位content，parse-regexp提取欄位。

輸出結果預覽

{
    "ip": "127.0.0.1",
    "time": "07/Jul/2022:10:43:30",
    "method": "POST",
    "url": "/PutData?Category=YunOsAccountOpLog",
    "request_time": "0.024",
    "request_length": "18204",
    "status": "200",
    "length": "37",
    "ref_url": "-",
    "browser": "aliyun-sdk-java",
    "__time__": "1713184059"
}

分隔字元解析對比

日誌範例：

127.0.0.1,07/Jul/2022:10:43:30 +0800,POST,PutData Category=YunOsAccountOpLog,0.024,18204,200,37,-,aliyun-sdk-java

分隔字元解析

SPL

分隔字元解析使用半形逗號（,）分割，處理外掛程式選擇分隔字元解析，具體操作請參見採集主機文本日誌。

SPL語句為：*| parse-csv content as ip, time, method, url, request_time, request_length, status, length, ref_url, browser | project-away content。此語句中project-away捨棄原有欄位content，parse-csv提取欄位。

輸出結果預覽

{
    "ip": "127.0.0.1",
    "time": "07/Jul/2022:10:43:30 +0800",
    "method": "POST",
    "url": "PutData?Category=YunOsAccountOpLog",
    "request_time": "0.024",
    "request_length": "18204",
    "status": "200",
    "length": "37",
    "ref_url": "-",
    "browser": "aliyun-sdk-java",
    "__time__": "1713231487"
}

JSON解析對比

日誌範例：

{"url": "POST /PutData?Category=YunOsAccountOpLog HTTP/1.1","ip": "10.200.98.220", "user-agent": "aliyun-sdk-java","request": "{\"status\":\"200\",\"latency\":\"18204\"}","time": "07/Jul/2022:10:30:28"}

JSON解析

SPL

具體操作請參見採集JSON格式文本日誌。

SPL語句為：*| parse-json content| project-away content。此語句中project-away捨棄原有欄位content，parse-json提取欄位。

輸出結果預覽

{
    "url": "POST /PutData?Category=YunOsAccountOpLog HTTP/1.1",
    "ip": "10.200.98.220",
    "user-agent": "aliyun-sdk-java",
    "request": "{"status":"200","latency":"18204"}",
    "time": "07/Jul/2022:10:30:28"
}

正則解析+時間解析對比

日誌範例：

127.0.0.1 - - [2024-11-05T15:47:05 +0800] "POST /PutData?Category=YunOsAccountOpLog" 0.024 18204 200 37 "-" "aliyun-sdk-java"

正則解析+時間解析

SPL

正則解析的Regex為：([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\"]*)\" \"([^\"]*)\"，日誌提取欄位為：ip,time,method,url,request_time,request_length,status,length,ref_url,browser。具體操作請參見使用正則模式採集文本日誌。
時間解析外掛程式原始欄位為time，時間格式為%Y-%m-%dT%H:%M:%S。

SPL語句為：* | parse-regexp content, '([\d\.]+) \S+ \S+ \[(\S+)\] \"(\w+) ([^\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\"]*)\" \"([^\"]*)\"' as ip, time, method, url, request_time, request_length, status, length, ref_url, browser| extend ts=date_parse(time, '%Y-%m-%dT%H:%i:%S')| extend __time__=cast(to_unixtime(ts) as INTEGER)-28800| project-away ts| project-away content。此語句中project-away捨棄原有欄位content，parse-regexp提取欄位，date_parse解析日誌中的時間。

正則解析+過濾處理對比

日誌範例：

127.0.0.1 - - [2024-11-05T15:47:05 +0800] "POST /PutData?Category=YunOsAccountOpLog" 0.024 18204 200 37 "-" "aliyun-sdk-java"

正則解析+過濾解析

SPL

正則解析的Regex為：([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\"]*)\" \"([^\"]*)\"，日誌提取欄位為：ip,time,method,url,request_time,request_length,status,length,ref_url,browser。具體操作請參見使用正則模式採集文本日誌。
過濾處理添加白名單status和method。

SPL語句為：*| parse-regexp content, '([\d\.]+) \S+ \S+ \[(\S+) \S+\] \"(\w+) ([^\"]*)\" ([\d\.]+) (\d+) (\d+) (\d+|-) \"([^\"]*)\" \"([^\"]*)\"' as ip, time, method, url, request_time, request_length, status, length, ref_url, browser| project-away content| where regexp_like(method, '^(POST|PUT)$') and regexp_like(status, '^200$')。此語句中project-away捨棄原有欄位content，parse-regexp提取欄位，regexp_like函數匹配符合正則的資料。

輸出結果預覽

{
    "ip": "127.0.0.1",
    "time": "2024-11-05T15:47:05",
    "method": "POST",
    "url": "/PutData?Category=YunOsAccountOpLog",
    "request_time": "0.024",
    "request_length": "18204",
    "status": "200",
    "length": "37",
    "ref_url": "-",
    "browser": "aliyun-sdk-java",
    "__time__": "1713238839"
}

脫敏處理對比

日誌範例：

{"account":"1812213231432969","password":"04a23f38"}

脫敏處理

SPL

脫敏處理將password欄位脫敏。

SPL語句為：*| parse-regexp content, 'password":"(\S+)"' as password| extend content=replace(content, password, '******')。此語句中project-away捨棄原有欄位content，parse-regexp提取欄位，replace替換成脫敏資料。

輸出結果預覽

{
    "content": "{"account":"1812213231432969","password":"******"}"
}

SPL與擴充外掛程式對比

添加欄位對比

日誌範例：

this is a test log

添加欄位

SPL

日誌預設存在content欄位中。添加欄位添加欄位service：A。

SPL語句為：* | extend service='A'。此語句中extend添加欄位service：A。

輸出結果預覽

{
    "content": "this is a test log",
    "service": "A"
}

Json解析+丟棄欄位對比

日誌範例：

{"key1": 123456, "key2": "abcd"}

Json解析+丟棄欄位

SPL

SPL語句為：*| parse-json content| project-away content| project-away key1。此語句中project-away捨棄原有欄位content和key1，parse-json提取欄位。

輸出結果預覽

{
    "key2": "abcd"
}

Json解析+重新命名欄位對比

日誌範例：

{"key1": 123456, "key2": "abcd"}

Json解析+重新命名欄位

SPL

SPL語句為：*| parse-json content| project-away content| project-rename new_key1=key1。此語句中project-away捨棄原有欄位content，parse-json提取欄位，project-rename重新命名欄位key1為new_key1。

輸出結果預覽

{
    "new_key1": "123456",
    "key2": "abcd"
}

Json解析+過濾欄位對比

日誌範例：

{"ip": "10.**.**.**", "method": "POST", "browser": "aliyun-sdk-java"}
{"ip": "10.**.**.**", "method": "POST", "browser": "chrome"}
{"ip": "192.168.**.**", "method": "POST", "browser": "aliyun-sls-ilogtail"}

Json解析+過濾欄位

SPL

SPL語句為：*| parse-json content| project-away content| where regexp_like(ip, '10\..*') and regexp_like(method, 'POST') and not regexp_like(browser, 'aliyun.*')。此語句中project-away捨棄原有欄位content，parse-json提取欄位，regexp_like判斷符合條件。

輸出結果預覽

{
    "ip": "10.**.**.**",
    "method": "POST",
    "browser": "chrome"
}

Json解析+欄位值對應處理對比

日誌範例：

{"_ip_":"192.168.*.*","Index":"900000003"}
{"_ip_":"255.255.**.**","Index":"3"}

Json解析+欄位值對應處理

SPL

SPL語句為：*| parse-json content| project-away content| extend _processed_ip_= CASE WHEN _ip_ = '127.0.*.*' THEN 'LocalHost-LocalHost' WHEN _ip_ = '192.168.*.*' THEN 'default login' ELSE 'Not Detected' END。此語句中project-away捨棄原有欄位content，parse-json提取欄位，extend構建新欄位。

輸出結果預覽

{
    "_ip_": "192.168.*.*",
    "Index": "900000003",
    "_processed_ip_": "default login"
}

字串替換對比

日誌範例：

hello,how old are you? nice to meet you

字串替換

SPL

字串替換用空值代替how old are you?。

SPL語句為：*| extend content=replace(content, 'how old are you?', '')。此語句中extend用空值代替how old are you?。

輸出結果預覽

{
    "content": "hello, nice to meet you"
}

資料編碼對比

日誌範例：

this is a test log

BASE64編碼

BASE64編碼

SPL

具體操作請參見BASE64編碼。

SPL語句為：*| extend content1=to_base64(cast(content as varbinary))。此語句中extend添加欄位content1，to_base64函數對資料進行BASE64編碼。

輸出結果預覽

{
    "content": "this is a test log",
    "content1": "dGhpcyBpcyBhIHRlc3QgbG9n"
}

MD5編碼

MD5

SPL

具體操作請參見MD5編碼。

SPL語句為：*| extend test=lower(to_hex(md5(cast(content as varbinary))))。此語句中extend添加欄位test，md5函數對資料進行MD5編碼。

輸出結果預覽

{
    "content": "this is a test log",
    "content1": "4f3c93e010f366eca78e00dc1ed08984"
}

新增能力項

數學計算

日誌範例
```
4
```

SPL語句

cast函數轉換資料類型，power函數、round函數和sqrt函數請參見數學計算函數。

*
| extend val = cast(content as double)
| extend power_test = power(val, 2)
| extend round_test = round(val)
| extend sqrt_test = sqrt(val)

輸出結果預覽

{
    "content": "4",
    "power_test": 16.0,
    "round_test": 4.0,
    "sqrt_test": 2.0,
    "val": 4.0
}

URL 計算

URL 編碼解碼

日誌範例

https://home.console.alibabacloud.com/home/dashboard/ProductAndService

SPL語句

url_encode函數和url_decode函數請參見URL函數。

*
| extend encoded = url_encode(content)
| extend decoded = url_decode(encoded)

輸出結果預覽

{
    "content": "https://home.console.alibabacloud.com/home/dashboard/ProductAndService",
    "decoded": "https://home.console.alibabacloud.com/home/dashboard/ProductAndService",
    "encoded": "https%3A%2F%2Fhome.console.alibabacloud.com%2Fhome%2Fdashboard%2FProductAndService"
}

URL 提取

日誌範例

https://sls.console.alibabacloud.com:443/lognext/project/dashboard-all/logsearch/nginx-demo?accounttraceid=d6241a173f88471c91d3405cda010ff5ghdw

SPL語句

SPL語句中涉及函數請參見URL函數。

*
| extend host = url_extract_host(content)
| extend query = url_extract_query(content)
| extend path = url_extract_path(content) 
| extend protocol = url_extract_protocol(content) 
| extend port = url_extract_port(content) 
| extend param = url_extract_parameter(content, 'accounttraceid')

輸出結果預覽

{
    "content": "https://sls.console.alibabacloud.com:443/lognext/project/dashboard-all/logsearch/nginx-demo?accounttraceid=d6241a173f88471c91d3405cda010ff5ghdw",
    "host": "sls.console.alibabacloud.com",
    "param": "d6241a173f88471c91d3405cda010ff5ghdw",
    "path": "/lognext/project/dashboard-all/logsearch/nginx-demo",
    "port": "443",
    "protocol": "https",
    "query": "accounttraceid=d6241a173f88471c91d3405cda010ff5ghdw"
}

比較&邏輯運算子

日誌範例
```
{"num1": 199, "num2": 10, "num3": 9}
```

SPL語句

cast函數轉換資料類型，parse-json提取欄位。

*
| parse-json content
| extend compare_result = cast(num1 as double) > cast(num2 as double) AND cast(num2 as double) > cast(num3 as double

輸出結果預覽

{
    "compare_result": "true",
    "content": "{"num1": 199, "num2": 10, "num3": 9}",
    "num1": "199",
    "num2": "10",
    "num3": "9"
}