Fungsi Grok menyederhanakan penguraian log dengan menggantikan ekspresi reguler yang kompleks menggunakan pola bernama yang telah ditentukan sebelumnya.
Ikhtisar
Fungsi Grok merupakan alternatif yang lebih sederhana dibandingkan fungsi ekspresi reguler. Anda dapat menggabungkan Grok dengan fungsi ekspresi reguler:
e_match("content", grok(r"\w+: (%{IP})")) # Mencocokkan pola seperti abc: 192.0.2.0 atau xyz: 192.0.2.2.
e_match("content", grok(r"\w+: (%{IP})", escape=True)) # Tidak mencocokkan abc: 192.0.2.0, tetapi mencocokkan \w+: 192.0.2.0.Fungsi Grok mengekstraksi nilai menggunakan pola ekspresi reguler yang telah ditentukan sebelumnya.
-
Sintaksis fungsi
grok(pattern, escape=False, extend=None) -
Sintaksis Grok
%{SYNTAX} %{SYNTAX:NAME}SYNTAX adalah pola ekspresi reguler yang telah ditentukan sebelumnya. NAME adalah kelompok bernama.
"%{IP}" # Setara dengan r"(?:\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" "%{IP:source_id}" # Setara dengan r"(?P<source_id>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" ("%{IP}") # Setara dengan r"(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"Grok memiliki dua mode pengelompokan:
-
Mode capturing group
Beberapa pola Grok memiliki named capturing group bawaan. Untuk pola-pola ini, gunakan hanya sintaksis %{SYNTAX}. Pola-pola ini umumnya digunakan untuk mengurai pernyataan log. Format log yang didukung tercantum dalam Pola Grok.
"%{SYSLOGBASE}" "%{COMMONAPACHELOG}" "%{COMBINEDAPACHELOG}" "%{HTTPD20_ERRORLOG}" "%{HTTPD24_ERRORLOG}" "%{HTTPD_ERRORLOG}" ... -
Mode non-capturing group
Beberapa pola Grok menggunakan mode non-capturing group. Contohnya:
"%{INT}" "%{YEAR}" "%{HOUR}" ...
-
-
Parameter
Parameter
Type
Wajib
Deskripsi
pattern
String
Ya
Pola Grok yang akan dicocokkan. Pola yang tersedia tercantum dalam Pola Grok.
escape
Bool
Tidak
Jika diatur ke True, karakter khusus regex di luar pola Grok akan di-escape. Nilai default: False.
extend
Dict
Tidak
Ekspresi Grok yang ditentukan pengguna.
Contoh fungsi
-
Contoh 1: Ekstrak tanggal dan konten dalam tanda kutip.
-
Log mentah
content: 2019 June 24 "I am iron man" -
Aturan transformasi
e_regex('content',grok('%{YEAR:year} %{MONTH:month} %{MONTHDAY:day} %{QUOTEDSTRING:motto}')) -
Hasil
content: 2019 June 24 "I am iron man" year: 2019 month: June day: 24 motto: "I am iron man"
-
-
Contoh 2: Ekstrak log permintaan HTTP.
-
Log mentah
content: 10.0.0.0 GET /index.html 15824 0.043 -
Aturan transformasi
e_regex('content',grok('%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}')) -
Hasil
content: 10.0.0.0 GET /index.html 15824 0.043 client: 10.0.0.0 method: GET request: /index.html bytes: 15824 duration: 0.043
-
-
Contoh 3: Ekstrak log Apache.
-
Log mentah
content: 127.0.0.1 - - [13/Apr/2015:17:22:03 +0800] "GET /router.php HTTP/1.1" 404 285 "-" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2" -
Aturan transformasi
e_regex('content',grok('%{COMBINEDAPACHELOG}')) -
Hasil
content: 127.0.0.1 - - [13/Apr/2015:17:22:03 +0800] "GET /router.php HTTP/1.1" 404 285 "-" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2" clientip: 127.0.0.1 ident: - auth: - timestamp: 13/Apr/2015:17:22:03 +0800 verb: GET request: /router.php httpversion: 1.1 response: 404 bytes: 285 referrer: "-" agent: "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2"
-
-
Contoh 4: Uraikan entri syslog.
-
Log mentah
content: May 29 16:37:11 sadness logger: hello world -
Aturan transformasi
e_regex('content',grok('%{SYSLOGBASE} %{DATA:message}')) -
Hasil
content: May 29 16:37:11 sadness logger: hello world timestamp: May 29 16:37:11 logsource: sadness program: logger message: hello world
-
-
Contoh 5: Escape karakter khusus.
-
Log mentah
content: Nov 1 21:14:23 scorn kernel: pid 84558 (expect), uid 30206: exited on signal 3 -
Aturan transformasi
e_regex('content',grok(r'%{SYSLOGBASE} pid %{NUMBER:pid} \(%{WORD:program}\), uid %{NUMBER:uid}: exited on signal %{NUMBER:signal}'))Aturan ini berisi tanda kurung, yang merupakan karakter khusus regex. Untuk melakukan auto-escape, atur escape ke True:
e_regex('content',grok('%{SYSLOGBASE} pid %{NUMBER:pid} (%{WORD:program}), uid %{NUMBER:uid}: exited on signal %{NUMBER:signal}', escape=True)) -
Hasil
content: Nov 1 21:14:23 scorn kernel: pid 84558 (expect), uid 30206: exited on signal 3 timestamp: Nov 1 21:14:23 logsource: scorn program: expect pid: 84558 uid: 30206 signal: 3
-
-
Contoh 6: Gunakan ekspresi Grok kustom.
-
Log mentah
content: Beijing-1104,gary 25 "never quit" -
Aturan transformasi
e_regex('content',grok('%{ID:user_id},%{WORD:name} %{INT:age} %{QUOTEDSTRING:motto}',extend={'ID': '%{WORD}-%{INT}'})) -
Hasil
content: Beijing-1104,gary 25 "never quit" user_id: Beijing-1104 name: gary age: 25 motto: "never quit"
-
-
Contoh 7: Cocokkan data JSON.
-
Log mentah
content: 2019-10-29 16:41:39,218 - INFO: owt.AudioFrameConstructor - McsStats: {"event":"mediaStats","connectionId":"331578616547393100","durationMs":"5000","rtpPackets":"250","rtpBytes":"36945","nackPackets":"0","nackBytes":"0","rtpIntervalAvg":"20","rtpIntervalMax":"104","rtpIntervalVar":"4","rtcpRecvPackets":"0","rtcpRecvBytes":"0","rtcpSendPackets":"1","rtcpSendBytes":"32","frame":"250","frameBytes":"36945","timeStampOutOfOrder":"0","frameIntervalAvg":"20","frameIntervalMax":"104","frameIntervalVar":"4","timeStampIntervalAvg":"960","timeStampIntervalMax":"960","timeStampIntervalVar":"0"} -
Aturan transformasi
e_regex('content',grok('%{EXTRACTJSON}')) -
Hasil
content: 2019-10-29 16:41:39,218 - INFO: owt.AudioFrameConstructor - McsStats: {"event":"mediaStats","connectionId":"331578616547393100","durationMs":"5000","rtpPackets":"250","rtpBytes":"36945","nackPackets":"0","nackBytes":"0","rtpIntervalAvg":"20","rtpIntervalMax":"104","rtpIntervalVar":"4","rtcpRecvPackets":"0","rtcpRecvBytes":"0","rtcpSendPackets":"1","rtcpSendBytes":"32","frame":"250","frameBytes":"36945","timeStampOutOfOrder":"0","frameIntervalAvg":"20","frameIntervalMax":"104","frameIntervalVar":"4","timeStampIntervalAvg":"960","timeStampIntervalMax":"960","timeStampIntervalVar":"0"} json:{"event":"mediaStats","connectionId":"331578616547393100","durationMs":"5000","rtpPackets":"250","rtpBytes":"36945","nackPackets":"0","nackBytes":"0","rtpIntervalAvg":"20","rtpIntervalMax":"104","rtpIntervalVar":"4","rtcpRecvPackets":"0","rtcpRecvBytes":"0","rtcpSendPackets":"1","rtcpSendBytes":"32","frame":"250","frameBytes":"36945","timeStampOutOfOrder":"0","frameIntervalAvg":"20","frameIntervalMax":"104","frameIntervalVar":"4","timeStampIntervalAvg":"960","timeStampIntervalMax":"960","timeStampIntervalVar":"0"}
-
-
Contoh 8: Uraikan log W3C.
-
Log mentah
content: 2018-12-26 00:00:00 W3SVC2 application001 192.168.0.0 HEAD / - 8000 - 10.0.0.0 HTTP/1.0 - - - - 404 0 64 0 19 0 -
Aturan transformasi
Bidang W3C yang tidak didukung muncul sebagai tanda hubung (-). Gunakan tanda hubung dalam pola Grok untuk mencocokkan bidang-bidang tersebut.
e_regex("content",grok('%{DATE:data} %{TIME:time} %{WORD:s_sitename} %{WORD:s_computername} %{IP:s_ip} %{WORD:cs_method} %{NOTSPACE:cs_uri_stem} - %{NUMBER:s_port} - %{IP:c_ip} %{NOTSPACE:cs_version} - - - - %{NUMBER:sc_status} %{NUMBER:sc_substatus} %{NUMBER:sc_win32_status} %{NUMBER:sc_bytes} %{NUMBER:cs_bytes} %{NUMBER:time_taken}')) -
Hasil
content: 2018-12-26 00:00:00 W3SVC2 application001 192.168.0.0 HEAD / - 8000 - 10.0.0.0 HTTP/1.0 - - - - 404 0 64 0 19 0 data: 18-12-26 time: 00:00:00 s_sitename: W3SVC2 s_computername: application001 s_ip: 192.168.0.0 cs_method: HEAD cs_uri_stem: / s_port: 8000 c_ip: 10.0.0.0 cs_version: HTTP/1.0 sc_status: 404 sc_substatus: 0 sc_win32_status: 64 sc_bytes: 0 cs_bytes: 19 time_taken: 0
-