Simple Log Service:Syntax of scan-based query

• Raw fields: labeled [R], applicable to scan search.

• Index fields: labeled [I], applicable to index search.

[I] __topic__: nginx-access-log
[I] Status: 200
[I] Host: api.abc.com
[R] Method: PUT
[R] ClientIp: 192.168.1.1
[R] Payload: {"Item": "1122", "UserId": "112233", "Operation": "AddCart"}
[R] BeginTime: 1705029260
[R] EndTime: 1705028561
[R] RT: 87
[R] Uri: /request/path-3/file-1
[R] UserAgent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; ar) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4

Filter by different conditions​

• Equality comparison

  Status: 200 | where ClientIp = '192.168.1.1'

• Case-insensitive search

  __topic__: nginx-access-log | where lower(Method) != 'put'

• Fuzzy match

  Status: 200 | where UserAgent like '%Macintosh%'

• Numerical comparison

  Note that the default field type is varchar. Convert the type to bigint before performing numerical comparisons.

  Status: 200 |  where cast(RT as bigint) > 50

• Regular expression match

  # Find URIs that contain "path-number"
  Status: 200 | where regexp_like(Uri, 'path-\d+')

Calculate new fields

​Calculate new fields from existing ones using the extend instruction.

• Extract fields using regular expressions

  # Extract the file number from the Uri field
  * not Status: 200 | extend fileNumber=regexp_extract(Uri, 'file-(\d+)', 1)

• Extract fields from JSON

  Status:200 | extend Item = json_extract_scalar(Payload, '$.Item')

• Extract fields based on a separator

  Status:200 | extend urlParam=split_part(Uri, '/', 3)

• Calculate new fields from multiple field values

  # Calculate the time difference based on BeginTime and EndTime
  Status:200 | extend timeRange = cast(BeginTime as bigint) - cast(EndTime as bigint)

Retain, remove, and rename fields​

• Keep only specific fields and remove the rest

  Status:200 | project Status, Uri

• Remove specific fields and keep the rest

  Status:200 | project-away UserAgent

• Rename fields

  Status:200 | project-rename Latency=RT

Expand unstructured data

• Expand all fields in JSON

  # Filter non-empty Payloads and expand all JSON fields
  __topic__: nginx-access-log | where Payload is not null | parse-json Payload

• Expand JSON fields and discard the original ones

  status:200 
  | parse-json body 
  | project-away body

• Extract multiple fields using regular expressions

  Status:200 | parse-regexp Uri, 'path-(\d+)/file-(\d+)' as pathIndex, fileIndex

Multi-level pipeline cascade

You can perform all the operations mentioned above in the same search statement using multi-level pipeline cascades, executed in sequence.

Status:200 
| where Payload is not null 
| parse-json Payload 
| project-away Payload 
| where Host='api.qzzw.com' and cast(RT as bigint) > 80 
| extend timeRange=cast(BeginTime as bigint) - cast(EndTime as bigint)
| where timeRange > 500
| project UserId, Uri