This topic describes how to use the data transformation feature of Simple Log Service (SLS) to parse JSON objects from complex strings.
Use case 1: Parse a JSON object from a complex string and expand specific fields
Raw log
content: [v1] [2024-07-12 14:45:15.497+0800] [INFO ] [XXID-1 task-10] WBERVIE [UID: N/A] m_point|{\"extra\":\"{\\\"items\\\":[{\\\"path\\\":\\\"Vdsjbxk.Cbsj.EV.FPDD\\\",\\\"id\\\":17,\\\"value\\\":1}]}\",\"vin\":\"WQ21497492\",\"source\":\"clo_is\",\"event\":\"clo_edd_received_rop\",\"tid\":\"Rcslcml\",\"ts\":1720680315491}Parsing requirements
Extract the JSON object that follows m_point from the content field and expand it into key-value pairs.
Assign values to the new fields newid, pid, pid_type, cid, and cid_type based on the value of the event field.
Set the __time__ field to the value of the ts field, which is parsed from the field a.
Structured Process Language (SPL) statement
* | project content | where regexp_like(content, '.*m_point\|.*') | parse-regexp content, '.*m_point\|(.*)' as a | parse-json a | project-away content , a | extend newid = if(event like 'clo_edd_received_rop', 'T989092', '') , pid = if(event like 'clo_edd_received_rop', 'clo_edd_received_rop', '') , pid_type = if(event like 'clo_edd_received_rop', 'clo_edd_received_rop', '') , cid = if(event like 'clo_edd_received_rop', 'clo_edd_received_rop', '') , cid_type = if(event like 'clo_edd_received_rop', 'clo_edd_received_rop', '') |extend "__time__" = cast(ts as bigint)/1000Result
cid:clo_edd_received_rop cid_type:clo_edd_received_rop event:clo_edd_received_rop extra:{"items":[{"path":"Vdsjbxk.Cbsj.EV.FPDD","id":17,"value":1}]} newid:T989092 pid:clo_edd_received_rop pid_type:clo_edd_received_rop source:clo_is tid:Rcslcml ts:1720680315491 vin:WQ21497492
Use case 2: Parse a JSON object from a string that contains XML characters and perform a two-level expansion
Raw log
{"__log_order_key__":11,"traceId":"dnohdohwhwiqnd923010hem2e","duration":"342424","spanId":"cr334c44","parentSpanId":"0","startTime":"1713765089425444896","spanName":"/DSFEF/RulePort","refefd":"csai-uat","pid":"kru2w@ere","ip":"","kind":"1","hostname":"","statusCode":"0","statusMessage":"","traceState":"","attributes":"{\"serviceType\":\"unknown\",\"db.erere\":\"unknown\",\"pid\":\"e33feeeeeeeeef234423\",\"source\":\"ebpf\",\"clusterId\":\"h8fhih9h9h99eh8hief\",\"resp.header\":\"Content-Type: text/xml;charset=utf-8\\nDate: Mon, 22 Apr 2024 05:51:29 GMT\\nServer: Apache-Coyote/1.1\\n\",\"status.code\":\"200\",\"container.id\":\"93d02646a21289224e210abd12c3988660c2dfeea3a2151742014249a5d55844\",\"callType\":\"http_client\",\"source_ip\":\"172.22.7.179\",\"resp.body\":\"<?xml version=\\\"1.0\\\" ?><S:Tvelop xmlns:S=\\\"http://schemas.xml.org/soap/Tvelope/\\\"><S:Body><ns2:ddResponse xmlns:ns2=\\\"http://service.tt.sinosoft.com/\\\"><return>{"applicationNum":"232323232","businessModule":"UW","isApproved":"0","isTest":false,"subSystem":"CS","system":"MICO","verifyResultList":[{"flag":"2","returnInfo":"Please confirm whether to continue.","ruleCategories":"2","ruleName":"CSUW0300001-Dual-recording quality inspection check"}]}</return></ns2:ddResponse></S:Body></S:Tvelop>\",\"endpoint\":\"/CSRuleintface/RulePort\",\"cmonitor\":\"KSpanInfo\",\"addr_family\":\"0\",\"remote_ip\":\"10.9.1.23\",\"req.body\":\"<?xml version='1.0' encoding='UTF-8'?><S:Tvelop xmlns:S=\\\"http://schemas.xml.org/soap/Tvelop/\\\"><S:Body><ns2:fireRule xmlns:ns2=\\\"http://service.t't.sinosoft.com/\\\"><arg0>MICO</arg0><arg1>CS</arg1><arg2>UW</arg2><arg3>{\\\"csuw\\\":{\\\"applicationNum\\\":\\\"232323232\\\",\\\"policyID\\\":\\\"213u92heijodwdwq3e231\\\",\\\"policyNum\\\":\\\"0839820984004\\\",\\\"CSType\\\":\\\"108\\\",\\\"applicationDate\\\":\\\"2024-04-22\\\",\\\"systemDate\\\":\\\"2024-04-22\\\",\\\"submitChannel\\\":\\\"12\\\",\\\"CheckIdent\\\":\\\"2\\\",\\\"CSAcceptanceNum\\\":\\\"00002030087801029\\\",\\\"organCode\\\":\\\"80040204\\\",\\\"applicantList\\\":[{\\\"holderID\\\":\\\"\\\",\\\"applicantName\\\":\\\"Hu Kaiqiang\\\",\\\"applyAge\\\":26,\\\"sex\\\":\\\"M\\\",\\\"birthDate\\\":\\\"1997-10-20\\\",\\\"IDCardType\\\":\\\"1\\\",\\\"cardID\\\":\\\"110106199710200614\\\",\\\"nationality\\\":\\\"37\\\",\\\"occupationCode\\\":\\\"J001001\\\",\\\"address\\\":\\\"\\\",\\\"telephone\\\":\\\"\\\",\\\"hasInform\\\":\\\"N\\\",\\\"customerInformList\\\":[{\\\"ordinalNum\\\":\\\"13\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"15.2\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.2\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.9\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"9\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.8\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.6\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"8\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"7\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.4\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"2\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.12\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.7\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.1\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.13\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.11\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"15.1\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"4\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"5\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"3\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"6\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"14\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.5\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"12\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.3\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.10\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"1\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"15.3\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"11\\\",\\\"hasInform\\\":\\\"2\\\"}]}],\\\"insurantList\\\":[{\\\"insurantID\\\":\\\"f65239503f52407e9806efb5d3b054f4\\\",\\\"insurantName\\\":\\\"Xing\\\",\\\"applyAge\\\":25,\\\"sex\\\":\\\"F\\\",\\\"birthDate\\\":\\\"1998-11-05\\\",\\\"IDCardType\\\":\\\"3\\\",\\\"cardID\\\":\\\"246558761420\\\",\\\"nationality\\\":\\\"1\\\",\\\"occupationCode\\\":\\\"J001001\\\",\\\"address\\\":\\\"\\\",\\\"telephone\\\":\\\"\\\",\\\"hasInform\\\":\\\"N\\\",\\\"weight\\\":50.0,\\\"height\\\":165.0,\\\"healthInformList\\\":[{\\\"hasSmoking\\\":\\\"N\\\",\\\"dailySmokAmount\\\":0.0,\\\"smokeYear\\\":0.0,\\\"hasDrinking\\\":\\\"N\\\",\\\"drinkType\\\":\\\"Alcohol\\\",\\\"dailyDrinkAmount\\\":0.0,\\\"drinkYear\\\":0.0}],\\\"productList\\\":[{\\\"productCode\\\":\\\"10131011\\\",\\\"productID\\\":\\\"\\\",\\\"productName\\\":\\\"Additional Insurance\\\",\\\"isMainRisk\\\":\\\"0\\\",\\\"amount\\\":5000.0,\\\"copies\\\":0},{\\\"productCode\\\":\\\"10132005\\\",\\\"productID\\\":\\\"\\\",\\\"productName\\\":\\\"Additional Medical\\\",\\\"isMainRisk\\\":\\\"0\\\",\\\"amount\\\":1800.0,\\\"copies\\\":1}],\\\"beneficiaryList\\\":[{\\\"benefitID\\\":\\\"\\\",\\\"beneficiaryName\\\":\\\"\\\",\\\"applyAge\\\":0,\\\"sex\\\":\\\"\\\",\\\"birthDate\\\":\\\"9999-12-31\\\",\\\"IDCardType\\\":\\\"\\\",\\\"cardID\\\":\\\"\\\",\\\"nationality\\\":\\\"\\\",\\\"occupationCode\\\":\\\"\\\",\\\"address\\\":\\\"\\\",\\\"telephone\\\":\\\"\\\"}],\\\"customerInformList\\\":[{\\\"ordinalNum\\\":\\\"13\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"15.2\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.2\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.9\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"9\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.8\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.6\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"8\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"7\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.4\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"2\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.12\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.7\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.1\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.13\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.11\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"15.1\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"4\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"5\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"3\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"6\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"14\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.5\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"12\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.3\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"10.10\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"1\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"15.3\\\",\\\"hasInform\\\":\\\"2\\\"},{\\\"ordinalNum\\\":\\\"11\\\",\\\"hasInform\\\":\\\"2\\\"}]}],\\\"agentList\\\":[{\\\"agentCode\\\":\\\"100606285\\\",\\\"agentName\\\":\\\"Fei Da Yu Yi\\\",\\\"agentAge\\\":65,\\\"agentSex\\\":\\\"M\\\",\\\"agentBirth\\\":\\\"1959-01-01\\\",\\\"IDCardType\\\":\\\"1\\\",\\\"cardID\\\":\\\"370983195901019996\\\",\\\"agentChannel\\\":\\\"\\\",\\\"telephone\\\":\\\"\\\"}]}}</arg3><arg4>false</arg4></ns2:fireRule></S:Body></S:Tvelop>\",\"source_port\":\"49152\",\"cmonitor_name\":\"cmonitor-agent-9xmdd\",\"host\":\"172.22.7.179\",\"cmonitor_ip\":\"10.3.72.9\",\"net_ns\":\"32232223\",\"ali.trace.rpc\":\"/CSRuleintface/RulePort\",\"deployment\":\"cd-insurance\",\"app\":\"csai--uat\",\"k8s.pod.ip\":\"172.22.7.179\",\"workloadKind\":\"deployment\",\"component.name\":\"http\",\"rpc\":\"/CSRuleintface/RulePort\",\"remote_port\":\"8010\",\"workloadName\":\"cd-insurance\",\"process_pid\":\"364469\",\"tsid\":\"25887099391057281\",\"version\":\"HTTP/1.1\",\"pod_name\":\"cd-insurance-787b759797-lchbk\",\"http.status_code\":\"200\",\"req.header\":\"Content-Type: text/xml; charset=utf-8\\nSoapaction: \\\"\\\"\\nUser-Agent: JAX-WS RI 2.2.9-b130926.1035 svn-revision#jd2939hondohd9y9hd2313\\nConnection: keep-alive\\nContent-Length: 4212\\nAccept: text/xml, multipart/related\\n\",\"rootIp\":\"172.22.7.179\",\"peer_k8s.pod.ip\":\"10.9.1.23\",\"destId\":\"10.9.1.23\",\"slow\":\"1\",\"service\":\"csai-29jodejnkn-insurance-uat\",\"ali.trace.flag\":\"x-trace\",\"namespace\":\"csai-ts\",\"ali.trace.wmjjelwl\":\"98\",\"serverIp\":\"172.22.7.179\",\"fd\":\"45\",\"statusCode\":\"200\",\"rpcType\":\"25\"}","resources":"{\"service.name\":\"cd-insurance\"}","__pack_meta__":"1|jiewojoj==|29|7","__topic__":"","__source__":"172.17.99.3","__tag__:__pack_id__":"9034804828-53FA94","__time__":"1713765092"}Parsing requirements
Extract the value of req.body from the attributes field.
Extract the JSON string that is wrapped in the XML string from req.body.
Perform a two-level expansion on the JSON string parsed from b.
SPL statement
* | project attributes | extend a = json_extract(attributes, '$["req.body"]') | extend b=regexp_extract(try_cast(a as varchar), '<arg3>(.*)<\/arg3>',1) | project b|parse-json -path='$.csuw' b | project-away bResult
CSAcceptanceNum:00002030087801029 CSType:108 CheckIdent:2 agentList:[{"telephone":"","agentChannel":"","cardID":"370983195901019996","IDCardType":"1","agentBirth":"1959-01-01","agentSex":"M","agentAge":65,"agentName":"Fei Da Yu Yi","agentCode":"100606285"}] applicantList:[{"customerInformList":[{"hasInform":"2","ordinalNum":"13"},{"hasInform":"2","ordinalNum":"15.2"},{"hasInform":"2","ordinalNum":"10.2"},{"hasInform":"2","ordinalNum":"10.9"},{"hasInform":"2","ordinalNum":"9"},{"hasInform":"2","ordinalNum":"10.8"},{"hasInform":"2","ordinalNum":"10.6"},{"hasInform":"2","ordinalNum":"8"},{"hasInform":"2","ordinalNum":"7"},{"hasInform":"2","ordinalNum":"10.4"},{"hasInform":"2","ordinalNum":"2"},{"hasInform":"2","ordinalNum":"10.12"},{"hasInform":"2","ordinalNum":"10.7"},{"hasInform":"2","ordinalNum":"10.1"},{"hasInform":"2","ordinalNum":"10.13"},{"hasInform":"2","ordinalNum":"10.11"},{"hasInform":"2","ordinalNum":"15.1"},{"hasInform":"2","ordinalNum":"4"},{"hasInform":"2","ordinalNum":"5"},{"hasInform":"2","ordinalNum":"3"},{"hasInform":"2","ordinalNum":"6"},{"hasInform":"2","ordinalNum":"14"},{"hasInform":"2","ordinalNum":"10.5"},{"hasInform":"2","ordinalNum":"12"},{"hasInform":"2","ordinalNum":"10.3"},{"hasInform":"2","ordinalNum":"10.10"},{"hasInform":"2","ordinalNum":"1"},{"hasInform":"2","ordinalNum":"15.3"},{"hasInform":"2","ordinalNum":"11"}],"hasInform":"N","telephone":"","address":"","occupationCode":"J001001","nationality":"37","cardID":"110106199710200614","IDCardType":"1","birthDate":"1997-10-20","sex":"M","applyAge":26,"applicantName":"Hu Kaiqiang","holderID":""}] applicationDate:2024-04-22 applicationNum:232323232 insurantList:[{"beneficiaryList":[{"telephone":"","address":"","occupationCode":"","nationality":"","cardID":"","IDCardType":"","birthDate":"9999-12-31","sex":"","applyAge":0,"beneficiaryName":"","benefitID":""}],"productList":[{"copies":0,"amount":5000,"isMainRisk":"0","productName":"Additional Insurance","productID":"","productCode":"10131011"},{"copies":1,"amount":1800,"isMainRisk":"0","productName":"Additional Medical","productID":"","productCode":"10132005"}],"healthInformList":[{"drinkYear":0,"dailyDrinkAmount":0,"drinkType":"Alcohol","hasDrinking":"N","smokeYear":0,"dailySmokAmount":0,"hasSmoking":"N"}],"height":165,"hasInform":"N","telephone":"","address":"","occupationCode":"J001001","nationality":"1","IDCardType":"3","birthDate":"1998-11-05","insurantID":"f65239503f52407e9806efb5d3b054f4","customerInformList":[{"hasInform":"2","ordinalNum":"13"},{"hasInform":"2","ordinalNum":"15.2"},{"hasInform":"2","ordinalNum":"10.2"},{"hasInform":"2","ordinalNum":"10.9"},{"hasInform":"2","ordinalNum":"9"},{"hasInform":"2","ordinalNum":"10.8"},{"hasInform":"2","ordinalNum":"10.6"},{"hasInform":"2","ordinalNum":"8"},{"hasInform":"2","ordinalNum":"7"},{"hasInform":"2","ordinalNum":"10.4"},{"hasInform":"2","ordinalNum":"2"},{"hasInform":"2","ordinalNum":"10.12"},{"hasInform":"2","ordinalNum":"10.7"},{"hasInform":"2","ordinalNum":"10.1"},{"hasInform":"2","ordinalNum":"10.13"},{"hasInform":"2","ordinalNum":"10.11"},{"hasInform":"2","ordinalNum":"15.1"},{"hasInform":"2","ordinalNum":"4"},{"hasInform":"2","ordinalNum":"5"},{"hasInform":"2","ordinalNum":"3"},{"hasInform":"2","ordinalNum":"6"},{"hasInform":"2","ordinalNum":"14"},{"hasInform":"2","ordinalNum":"10.5"},{"hasInform":"2","ordinalNum":"12"},{"hasInform":"2","ordinalNum":"10.3"},{"hasInform":"2","ordinalNum":"10.10"},{"hasInform":"2","ordinalNum":"1"},{"hasInform":"2","ordinalNum":"15.3"},{"hasInform":"2","ordinalNum":"11"}],"weight":50,"cardID":"246558761420","sex":"F","applyAge":25,"insurantName":"Xing"}] organCode:80040204 policyID:213u92heijodwdwq3e231 policyNum:0839820984004 submitChannel:12 systemDate:2024-04-22
Use case 3: Filter logs from a JSON string and extract specified fields
Raw log
{"__LEVEL__":"INFO","__THREAD__":"73539","__FILE__":"nas/test/tasks/tt_task.cpp","__LINE__":"424","Vers":"100","VolumeId":"djkaywiqhiwql","Method":"HPSFRename","TaskStatus":"0","NasStatus":"0","Operation":"rename","Totallatency":"1412","TimePoints":"[ phase_receive:9 alloc_qos_token:2 qos_queue:1 taa_proc:21 storage_schedule:5 storage_proc:19090 taa_post:46 done_schedule:5 taaio_done:4]","StorageSchedule":"5","StorageProc":"1317","ExpandMsg":"(7:1(us),)","TraceId":"","RequestId":"903J3JO3J","SourceIp":"10.10.10.10","SessionId":"200_3298309203802","Priority":"0","QosTenant":"QosTenant(idx:0,gen:3)","microtime":"1731392376869111","WorkerIdx":"7","QosDelay":"2","ChannelType":"TCP","client_unique":"930290392","client_pid":"2779589","client_uid":"0","client_gid":"0","client_pidname":"MockPidName","client_hostname":"92EH92EIO0I2UO2","client_mountname":"82y9he92y92h2ejo20232","client_localmount":"/var/run/aa/bindroot/testroot","client_arrive_time":"1731392376867903","Name":"/var/run/aa/bindroot/bindroot-604c/motr.csv.temp","DstName":"mots.csv","FilePath":"","DstFilePath":"","DstDeleted":"0","ResIno":"930230283","User":"0937089203013","ConnId":"48370","SourcePort":"9058","Vip":"10.10.10.10","TunnelId":"9988","microtime_0":"1731392376869109","__pack_meta__":"83|38H2OIH2HE2DEU232YW==|363|287","__topic__":"","__source__":"16.72.37.72","__tag__:__hostname__":"c96l020901.cloud.na322","__tag__:__path__":"/apsara/FileServer/log/test_access.LOG","__tag__:__user_defined_id__":"cn-wulanchabu-c-tenant-5","__tag__:__pack_id__":"5D389283922966585F-0","__tag__:__receive_time__":"1731392378","__time__":"1731392376"}Parsing requirements
Keep only the logs in which the value of the Operation field is remove or rename.
Keep only the Name, VolumeId, and Operation fields. Rename these fields to fullPath, fsName, and event, respectively. Extract the first ID from the __tag__:__hostname__ field and assign it to the host field.
Add a new field named module and set its value to fileserver.
Convert the __time__ field to a time object and assign the result to a new field named time.
Extract the file name from the value of the fullPath field and assign it to the subPath field.
SPL statement
* | where Operation = 'remove' or Operation = 'rename' | project Name, VolumeId, "__tag__:__hostname__", __time__, Operation | project-rename fullPath=Name, fsName=VolumeId, host="__tag__:__hostname__", event=Operation | extend host = split_part(host, '.', 1),module = 'fileserver' | extend fullPath = regexp_replace(fullPath,'(^.*)([\/]+$)','\1') | parse-regexp fullPath, '([^\\/]+$)' as subPath | extend time=from_unixtime(cast(__time__ as DOUBLE) + 28800)Result
event:rename fsName:djkaywiqhiwql fullPath:/var/run/aa/bindroot/bindroot-604c/motr.csv.temp host:c96l020901 module:fileserver subPath:motr.csv.temp time:2024-11-12 14:19:36.000
Use case 4: Filter logs based on conditions and extract specified fields
Raw log
{"minReadOffset":"","minWriteOffset":"","clientUserId":"0","maxWriteOffset":"","cookie":"1","fileSetID":"2","linkCount":"1","openFlags":"0","ctime":"2024-03-13_16:44:58.423+0800","clientGroupId":"0","accessMode":"","uyt":"","mask":"0x200","nfsIp":"","bytesRead":"","maxReadOffset":"","inode":"892323","fileSize":"8187382","poolName":"system","processId":"56271","bytesWritten":"","xattrs":"","TE_JSON":"0.0.3","clusterName":"cpfe-test_test-2.cn-wulanchabu.cpfe.aliyuncs.com","fsName":"39434034nwkeowuoup","ownerUserId":"0","atime":"2024-03-13_16:44:58.073+0800","subEvent":"NONE","wd":"1","event":"IN_DELETE","eventTime":"2024-03-13_16:44:58.800+0800","permissions":"200100600","path":"/cpfe/370l64h5u6zjy8buacw/root@10.30.42.19/1710249292/.efc_2251799814382581_2984197703087879309_1710319498445559_file1nas_file_0.txt/","nodeName":"x77g11119_ext.cloud.na132","ownerGroupId":"0","mtime":"2024-03-13_16:44:58.423+0800","__pack_meta__":"0|EJIWOUE02U29HEE8220YOHDW==|1|0","__topic__":"","__source__":"16.40.20.10","__tag__:__receive_time__":"1710319498","__time__":"1710319498","__time_ns_part__":"800"}Parsing requirements
Keep only the logs in which the event field does not contain IN_MOVED_TO and the path field does not contain .mmSharedTmpDir.
Keep only the nodeName, processId, path, inode, fsName, eventTime, and event fields. Rename the eventTime and path fields to time and fullPath, respectively.
Extract the ID prefix from the nodeName field and assign it to the host field. Then, delete the nodeName field. Remove the trailing slash from the fullPath value, extract the file path, and assign it to the subPath field.
Add a new field named module and set its value to cpfs.
Convert the time field, which includes a time zone, to a time object without a time zone. Then, assign the result to the __time__ field.
SPL statement
* | where event not like '%IN_MOVED_TO%' and path not like '%.mmSharedTmpDir%' | project nodeName, processId, path, inode, fsName, eventTime, event | extend nodeName=replace(nodeName, '_ext', '') | project-rename host=nodeName, time=eventTime, fullPath=path | extend host = split_part(host, '.', 1),module ='cpfs',fullPath = regexp_replace(fullPath,'(^.*)([\/]+$)','\1'),time=replace(replace(time, '+0800', ''), '_', ' ') | parse-regexp fullPath, '([^\\/]+$)' as subPath | extend __time__ = cast(to_unixtime(cast(time as TIMESTAMP)) as bigint) - 28800Result
event:IN_DELETE fsName:39434034nwkeowuoup fullPath:/cpfe/370l64h5u6zjy8buacw/root@10.30.42.19/1710249292/.efc_2251799814382581_2984197703087879309_1710319498445559_file1nas_file_0.txt host:x77g11119 inode:892323 module:cpfs processId:56271 subPath:.efc_2251799814382581_2984197703087879309_1710319498445559_file1nas_file_0.txt time:2024-03-13 16:44:58.800 __time__:1710319499
Use case 5: Extract a JSON string from a multilayer JSON object
Raw log
{ "__source__":"log_service", "__topic__":"mse_test_access_log", "authority":"go-micro.rrzu.com", "bytes_received":"147", "bytes_sent":"328", "cluster_id":"gw-38019u2u9ndnkwdkjh", "duration":"43", "method":"POST", "path":"/diversion/test", "protocol":"HTTP/1.1", "request_id":"681292f6-4aba-42df-8e75-ab426a620382", "requested_server_name":"go-micro.ewewew.com", "response_code":"200", "response_flags":"-", "route_name":"go-diversion-route-f732e980-9738279nex892u9e2m", "start_time":"2024-11-13T04:01:23.770Z", "trace_id":"2932eh2nein2", "upstream_cluster":"outbound|2233||diversion-api-dd.go.ee.cluster.local", "upstream_host":"172.16.9.225:2233", "upstream_local_address":"100.10.0.0:12882", "upstream_service_time":"42", "upstream_transport_failure_reason":"-", "user_agent":"GuzzleHttp/7", "x_forwarded_for":"100.10.0.0:1000", "downstream_local_address":"100.10.0.0:443", "downstream_remote_address":"100.10.0.0:12767", "request_headers":"{\"envoy.lua:request_body\":\"{\"experiment_uniq_id\":\"test_name_address_remind_ali\",\"distinct_id\":\"392839072\",\"terminal\":\"alipay.ipope\",\"version\":\"3.2.35\",\"model_type\":\"abtest\"}\",\"envoy.lua:request_headers\":\":authority=go-micro.ew.com, :path=/diversion/dede, :method=POST, :scheme=https, user-agent=GuzzleHttp/7, content-type=application/json, appid=diversion, timestamp=1731470483, sign=903h2oni2biy2y9en, content-length=147, x-forwarded-for=192.168.0.35, x-forwarded-proto=https, x-envoy-internal=true, x-dddd-original-host=go-micro.rrzu.com, x-request-id=23232d2-4aba-42df-8e75-4dr32ed, x-envoy-decorator-operation=diversion-api-def.go.svc.cluster.local:2233/diversion/(.*)((\/).*)?, \"}", "response_headers":"{\"envoy.lua:response_body\":\"{\"status\":0,\"message\":\"OK\",\"data\":{\"experiment_id\":21,\"experiment_type\":1,\"experiment_uniqwqwq_id\":\"sf_real_name_address_remind_ali\",\"experiment_group_id\":100,\"experiment_group_name\":\"Control Group\",\"experiment_group_type\":0,\"is_white\":0,\"param_id\":\"sws2wu91w1w1s1s\",\"param_type\":1,\"param_val\":\"2\",\"distinct_id\":\"42465147\"}}\",\"envoy.lua:response_headers\":\":status=200, access-control-allow-headers=Content-Type, Origin, X-CSRF-Token, Authorization, AccessToken, Token, Range, X-Requested-With, mini_version, terminal, access-control-allow-methods=GET,PUT,POST,DELETE,OPTIONS, content-type=application/json; charset=utf-8, traceparent=00-3902u2jej9d286e92h-2e3r4f4f4ff-00, vary=Origin, date=Wed, 13 Nov 2024 04:01:23 GMT, content-length=328, req-cost-time=43, req-arrive-time=1731470483770, resp-start-time=1731470483813, x-envoy-upstream-service-time=42, set-cookie=canary-route=\"fr4544t5t542\"; Max-Age=10; Path=/; HttpOnly, stage=prod, set-cookie=acw_tc=681292f6-4aba-42df-8e75-029j30jej3dn39hf3h9hnhfeuy47rhnfhf4;path=/;HttpOnly;Max-Age=1800, \"}" }Parsing requirements
If the value of the authority field is not data-analysis.rrzu.com or data-behavior.rrzu.com, perform the following steps.
Rename the authority, duration, user_agent, request_id, response_code, path, and x_forwarded_for fields to host, request_time, http_user_agent, req_id, status, url, and x_forward_for, respectively.
Parse the IP address from x_forward_for. Parse request_body and request_header from request_headers. Parse response_data from response_headers.
Finally, delete the request_headers and response_headers fields.
SPL statement
* | where 'authority' != 'data-analysis.rrzu.com' and 'authority' != 'data-behavior.rrzu.com' | project-rename host=authority,request_time=duration,http_user_agent=user_agent,req_id=request_id,status=response_code,url=path,x_forward_for=x_forwarded_for | extend ip=regexp_extract(x_forward_for, '\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}') | parse-regexp request_headers,'{"envoy.lua:request_body":"(.*?)"\,"envoy.lua:request_headers":"(.*?)"}' as request_body, request_header | parse-regexp response_headers, '"envoy.lua:response_body":"({.*?})"' as response_data | project-away request_headers,response_headersResult
__source__:log_service __topic__:mse_test_access_log bytes_received:147 bytes_sent:328 cluster_id:gw-38019u2u9ndnkwdkjh downstream_local_address:100.10.0.0:443 downstream_remote_address:100.10.0.0:12767 host:go-micro.rrzu.com http_user_agent:GuzzleHttp/7 ip:100.10.0.0 method:POST protocol:HTTP/1.1 req_id:681292f6-4aba-42df-8e75-ab426a620382 request_body:{"experiment_uniq_id":"test_name_address_remind_ali","distinct_id":"392839072","terminal":"alipay.ipope","version":"3.2.35","model_type":"abtest"} request_header::authority=go-micro.ew.com, :path=/diversion/dede, :method=POST, :scheme=https, user-agent=GuzzleHttp/7, content-type=application/json, appid=diversion, timestamp=1731470483, sign=903h2oni2biy2y9en, content-length=147, x-forwarded-for=192.168.0.35, x-forwarded-proto=https, x-envoy-internal=true, x-dddd-original-host=go-micro.rrzu.com, x-request-id=23232d2-4aba-42df-8e75-4dr32ed, x-envoy-decorator-operation=diversion-api-def.go.svc.cluster.local:2233/diversion/(.*)((/).*)?, request_time:43 requested_server_name:go-micro.ewewew.com response_data:{"status":0,"message":"OK","data":{"experiment_id":21,"experiment_type":1,"experiment_uniqwqwq_id":"sf_real_name_address_remind_ali","experiment_group_id":100,"experiment_group_name":"Control Group","experiment_group_type":0,"is_white":0,"param_id":"sws2wu91w1w1s1s","param_type":1,"param_val":"2","distinct_id":"42465147"}} response_flags:- route_name:go-diversion-route-f732e980-9738279nex892u9e2m start_time:2024-11-13T04:01:23.770Z status:200 trace_id:2932eh2nein2 upstream_cluster:outbound|2233||diversion-api-dd.go.ee.cluster.local upstream_host:172.16.9.225:2233 upstream_local_address:100.10.0.0:12882 upstream_service_time:42 upstream_transport_failure_reason:- url:/diversion/test x_forward_for:100.10.0.0:1000