Use ingest processors to process Alibaba Cloud service logs

Prerequisites

You have created a Project and a Standard Logstore, and configured log collection. For more information, see Manage a Project, Create a Standard Logstore, and Overview of log collection.

ACK

Retain only the write operation in audit logs

• Requirement description

  The audit logs of a Container Service for Kubernetes (ACK) cluster record read and write operations, distinguished by the verb field. The verb values include create, get, list, patch, update, and watch. To retain only write operations and reduce storage costs, configure an ingest processor.

• SPL statement

  * | where verb not in ('get', 'watch', 'list')

Extract log fields on the control plane

• Requirement description

  Extract the content field from a log entry.

• Raw log

  _container_name_: kube-controller-manager
  _source_: stderr
  _time_: 2024-12-19T11:10:16.251374622+08:00
  content: I1219 11:10:16.251299       1 httplog.go:134] "HTTP" verb="GET" URI="/healthz" latency="45.911µs" userAgent="kube-probe/1.30+" audit-ID="" srcIP="7.8.44.1:40092" resp=200 contentType="text/plain; charset=utf-8" resp=200

• SPL statement

  *
  | parse-regexp content, '\S+\s+\S+\s+\S+\s+(\S+)\]\s+"([^"]+)"\s+(.*)' as caller, msg, kvs
  | parse-kv -regexp kvs, '([\S+]+)="?([^"]*)"?(?:\s+|$)' 
  | project-away content, kvs

• Results

  _container_name_: kube-controller-manager
  _source_: stderr
  _time_: 2024-12-19T11:10:16.251374622+08:00
  URI: /healthz
  audit-ID:
  caller: httplog.go:134
  contentType: text/plain; charset=utf-8
  latency: 45.911µs
  msg: HTTP
  resp: 200
  srcIP: 7.8.44.1:40092
  userAgent: kube-probe/1.30+
  verb: GET

OSS

Retain data that meets specific conditions

• Requirement description

  Retain only the logs of operations that are related to Object Storage Service (OSS) objects.

• SPL statement

  * | where operation like '%Object%'

Decode URLs for the object and request_uri fields

• Requirement description

  The object and request_uri fields in OSS access logs are URL-encoded by default. You can use an ingest processor to decode these fields.

• Raw log

  object: image%2Flogo.png
  request_uri: /image%2Flogo.png HTTP/1.1

• SPL statement

  * | extend object=url_decode(object), request_uri=url_decode(request_uri)

• Results

  object: image/logo.png
  request_uri: /image/logo.png HTTP/1.1

SLB

Retain only logs for failed requests

• Requirement description

  The status field in Application Load Balancer (ALB) access logs indicates the HTTP status code of each request. You can use an ingest processor to retain only logs for failed requests.

• SPL statement

  * | where cast(status as bigint) >= 400

Mask data in the request_uri field for Layer 7 access logs

• Requirement description

  The request_uri field in ALB access logs contains both the request path and query parameters. You can use an ingest processor to strip the query parameters and retain only the path.

• Raw log

  request_uri: /api/v1/some-resource?token=xxxxxxx

• SPL statement

  * | extend request_uri=url_extract_path(request_uri)

• Results

  request_uri: /api/v1/some-resource