Documentation
Home Page Simple Log Service Data Processing Process data during ingestion (ingest processor) Ingest processor use cases

Ingest processor use cases

Updated at:
Copy as MD
Product

An ingest processor processes log data before it is written to a Logstore, supporting field modification, field parsing, data filtering, and data masking.

Prerequisites

You have created a Project and a Standard Logstore, and configured log collection. For more information, see Manage a Project, Create a Standard Logstore, and Overview of log collection.

Use cases

Assume that you want to extract three fields from a raw log: request_method, request_uri, and status. Follow these steps.

Raw log

body_bytes_sent: 22646
host: www.example.com
http_protocol: HTTP/1.1
remote_addr: 192.168.31.1
remote_user: Elisa
request_length: 42450
request_method: GET
request_time: 675
request_uri: /index.html
status: 200
time_local: 2024-12-04T13:47:54+08:00

Procedure

  1. Create an ingest processor.

    1. Log on to the Simple Log Service console.

    2. In the Projects section, click the project you want.

    3. In the left-side navigation pane, choose Resources > Data Processor.

    4. On the Ingest Processor tab, click Create. After you complete the configuration, click Save. The following table describes the parameters.

      Parameter

      Description

      Processor name

      Enter a name for the processor, such as nginx-logs-text.

      SPL

      Enter an SPL statement. Example:

      * | project request_method, request_uri, status

      For more information, see SPL commands and functions.

      Error Handling

      Action to take if SPL processing fails. The following options are available:

      • Retain Raw Data

      • Discard Raw Data

      Note

      • Processing failures refer to cases where an operator fails during execution, for example, due to invalid data. This does not include failures caused by SPL syntax errors.

      • If parsing fails due to incorrect SPL syntax, the raw log is retained by default.

  2. Associate the ingest processor with a Logstore.

    1. On the target Project page, click Log Storage in the left-side navigation pane. Hover over the target Logstore and click Modify.

    2. On the Logstore Attributes page, select Ingest Processor, click Modify, select the desired ingest processor, and then click Save.

      Note

      The associated ingest processor takes effect only for newly ingested logs. The change takes effect in about one minute.

More use cases

  • Modify fields

    SPL commands such as project, project-away, project-rename, and extend add, delete, or modify log fields. For example, consider the following raw log:

    body_bytes_sent: 22646
host: www.example.com
http_protocol: HTTP/1.1
referer: www.example.com
remote_addr: 192.168.31.1
remote_user: Elisa
request_length: 42450
request_method: PUT
request_time: 675
request_uri: /request/path-1/file-6?query=123456
status: 200
time_local: 2024-12-04T13:47:54+08:00

    Use case

    Description

    SPL statement

    Result

    Retain specific fields

    Retain only the following three fields:

    • request_method

    • request_uri

    • status

    		 
    * | project request_method, request_uri, status
    		 
    request_method: PUT
request_uri: /request/path-1/file-6?query=123456
status: 200

    Retain only the following three fields, and rename some of them:

    • Rename request_method to method.

    • Rename request_uri to uri.

    • Keep the status field.

    		 
    * | project method=request_method, uri=request_uri, status
    		 
    method: PUT
uri: /request/path-1/file-6?query=123456
status: 200

    Retain all fields that start with request_.

    		 
    * | project -wildcard "request_*"
    		 
    request_length: 42450
request_method: PUT
request_time: 675
request_uri: /request/path-1/file-6?query=123456

    Delete specific fields

    Delete the following fields:

    • http_protocol

    • referer

    • remote_addr

    • remote_user

    		 
    * | project-away http_protocol, referer, remote_addr, remote_user
    		 
    body_bytes_sent: 22646
host: www.example.com
request_length: 42450
request_method: PUT
request_time: 675
request_uri: /request/path-1/file-6?query=123456
status: 200
time_local: 2024-12-04T13:47:54+08:00

    Delete all fields that start with request_.

    		 
    * | project-away -wildcard "request_*"
    		 
    body_bytes_sent: 22646
host: www.example.com
http_protocol: HTTP/1.1
referer: www.example.com
remote_addr: 192.168.31.1
remote_user: Elisa
status: 200
time_local: 2024-12-04T13:47:54+08:00

    Add a new field

    Add an app field with the value test-app.

    		 
    * | extend app='test-app'
    		 
    app: test-app
body_bytes_sent: 22646
host: www.example.com
http_protocol: HTTP/1.1
referer: www.example.com
remote_addr: 192.168.31.1
remote_user: Elisa
request_length: 42450
request_method: PUT
request_time: 675
request_uri: /request/path-1/file-6?query=123456
status: 200
time_local: 2024-12-04T13:47:54+08:00

    Add a request_query field and extract its value from request_uri.

    		 
    * | extend request_query=url_extract_query(request_uri)
    		 
    body_bytes_sent: 22646
host: www.example.com
http_protocol: HTTP/1.1
referer: www.example.com
remote_addr: 192.168.31.1
remote_user: Elisa
request_length: 42450
request_method: PUT
request_query: query=123456
request_time: 675
request_uri: /request/path-1/file-6?query=123456
status: 200
time_local: 2024-12-04T13:47:54+08:00

    Rename a field

    Rename the time_local field to time.

    		 
    * | project-rename time=time_local
    		 
    body_bytes_sent: 22646
host: www.example.com
http_protocol: HTTP/1.1
referer: www.example.com
remote_addr: 192.168.31.1
remote_user: Elisa
request_length: 42450
request_method: PUT
request_time: 675
request_uri: /request/path-1/file-6?query=123456
status: 200
time: 2024-12-04T13:47:54+08:00

    Modify a field value

    Process request_uri to keep only the path and remove the query parameters.

    		 
    * | extend request_uri=url_extract_path(request_uri)

    or

    * | extend request_uri=regexp_replace(request_uri, '\?.*', '')
    		 
    body_bytes_sent: 22646
host: www.example.com
http_protocol: HTTP/1.1
referer: www.example.com
remote_addr: 192.168.31.1
remote_user: Elisa
request_length: 42450
request_method: PUT
request_time: 675
request_uri: /request/path-1/file-6
status: 200
time_local: 2024-12-04T13:47:54+08:00

  • Parse fields

    SPL commands such as parse-regexp, parse-json, and parse-csv, along with SQL functions for regular expressions and JSON, parse and extract data from fields.

    Use case

    Raw data

    Description

    SPL statement

    Result

    Regex parsing

    		 
    content: 192.168.1.75 - David [2024-07-31T14:27:24+08:00] "PUT /request/path-0/file-8 HTTP/1.1" 819 21577 403 73895 www.example.com www.example.com "Mozilla/5.0 (Windows NT 5.2; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.41 Safari/535.1"

    Use a regular expression to extract fields from an NGINX access log and then discard the original content field.

    		 
    * 
| parse-regexp content, '(\S+)\s-\s(\S+)\s\[(\S+)\]\s"(\S+)\s(\S+)\s(\S+)"\s(\d+)\s(\d+)\s(\d+)\s(\d+)\s(\S+)\s(\S+)\s"(.*)"' as remote_addr, remote_user, time_local, request_method, request_uri, http_protocol, request_time, request_length, status, body_bytes_sent, host, referer, user_agent
| project-away content
    		 
    body_bytes_sent: 73895
host: www.example.com
http_protocol: HTTP/1.1
referer: www.example.com
remote_addr: 192.168.1.75
remote_user: David
request_length: 21577
request_method: PUT
request_time: 819
request_uri: /request/path-0/file-8
status: 403
time_local: 2024-07-31T14:27:24+08:00
user_agent: Mozilla/5.0 (Windows NT 5.2; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.41 Safari/535.1
    		 
    request_method: PUT
request_uri: /request/path-0/file-8
status: 200

    Extract the file-8 part from the request_uri field and name it the file field.

    		 
    * | extend file=regexp_extract(request_uri, 'file-.*')

    		 
    file: file-8
request_method: PUT
request_uri: /request/path-0/file-8
status: 200

    JSON parsing

    		 
    headers: {"Authorization": "bearer xxxxx", "X-Request-ID": "29bbe977-9a62-4e4a-b2f4-5cf7b65d508f"}

    Parse the headers field as JSON and then discard the original headers field.

    		 
    *
| parse-json headers
| project-away headers

    		 
    Authorization: bearer xxxxx
X-Request-ID: 29bbe977-9a62-4e4a-b2f4-5cf7b65d508f

    Extract specific fields from the headers field. For example, extract the Authorization field and name it token.

    		 
    * | extend token=json_extract_scalar(headers, '$.Authorization')

    		 
    headers: {"Authorization": "bearer xxxxx", "X-Request-ID": "29bbe977-9a62-4e4a-b2f4-5cf7b65d508f"}
token: bearer xxxxx

    		 

    request: {"body": {"user_id": 12345, "user_name": "Alice"}}

    Parse the body subfield in the request field as JSON.

    		 
    * | parse-json -path='$.body' request
    		 
    request: {"body": {"user_id": 12345, "user_name": "Alice"}}
user_id: 12345
user_name: Alice

    Delimiter parsing

    		 
    content: 192.168.0.100,"10/Jun/2019:11:32:16,127 +0800",www.example.com

    Use commas to split the field, and then discard the original content field.

    		 
    *
| parse-csv -quote='"' content as ip, time, host
| project-away content
    		 
    host: www.example.com
ip: 192.168.0.100
time: 10/Jun/2019:11:32:16,127 +0800
    		 
    content: 192.168.0.100||10/Jun/2019:11:32:16,127 +0800||www.example.com

    Use || as the delimiter to split fields, and discard the original content field.

    		 
    *
| parse-csv -delim='||' content as ip, time, host
| project-away content
    		 
    host: www.example.com
ip: 192.168.0.100
time: 10/Jun/2019:11:32:16,127 +0800

  • Filter data

    The where command filters data.

    During SPL processing, all original fields are treated as text by default. To perform numerical operations, convert the data types by using a data type conversion function.

    Raw data

    Description

    SPL statement

    Result

    		 
    request_id: ddbde824-7c3e-4ff1-a6d1-c3a53fd4a919
status: 200

---

request_id: 7f9dad20-bc57-4aa7-af0e-436621f1f51d
status: 500

    Keep logs with a status of 200.

    		 
    * | where status='200'

    or

    * | where cast(status as bigint)=200
    		 
    request_id: ddbde824-7c3e-4ff1-a6d1-c3a53fd4a919
status: 200
    		 
    request_id: ddbde824-7c3e-4ff1-a6d1-c3a53fd4a919
status: 200

---

request_id: 7f9dad20-bc57-4aa7-af0e-436621f1f51d
status: 500
error: something wrong

    Keep logs without an error field.

    		 
    * | where error is null
    		 
    request_id: ddbde824-7c3e-4ff1-a6d1-c3a53fd4a919
status: 200

    Keep logs with an error field.

    		 
    * | where error is not null
    		 
    request_id: 7f9dad20-bc57-4aa7-af0e-436621f1f51d
status: 500
error: something wrong
    		 
    method: POST
request_uri: /app/login

---

method: GET
request_uri: /user/1/profile
status: 404

---

method: GET
request_uri: /user/2/profile
status: 200

    Retain only data where request_uri starts with /user/.

    		 
    * | where regexp_like(request_uri, '^\/user\/')
    		 
    method: GET
request_uri: /user/1/profile
status: 404

---

method: GET
request_uri: /user/2/profile
status: 200

    Retain data where the request_uri starts with /user/ and the status is 200.

    		 
    * | where regexp_like(request_uri, '^\/user\/') and status='200'
    		 
    method: GET
request_uri: /user/2/profile
status: 200

  • Mask data

    • Use the mask function

      The mask function supports both built-in rules and keyword matching to precisely mask structured and unstructured data.

      Raw data

      Description

      SPL statement

      Result

      		 
      client_ip: 192.168.1.123
latency: 100

      Mask the IP address.

      		 
      * | extend client_ip =  mask(client_ip,'[
             {"mode":"buildin","types":["IP_ADDRESS"],"maskChar":"*","keepPrefix":3,"keepSuffix":3}
            ]')
      		 
      client_ip: 192*****123
latency: 100
      		 
      2025-08-20 18:04:40,998 INFO  blockchain-event-poller-3 [10.0.1.20] [com.service.listener.TransactionStatusListener:65] [TransactionStatusListener#handleSuccessfulTransaction]{"message":"On-chain transaction successfully confirmed","confirmationDetails":{"transactionHash":"0x2baf892e9a164b1979","status":"success","blockNumber":45101239,"gasUsed":189543,"effectiveGasPrice":"58.2 Gwei","userProfileSnapshot":{"wallet":"0x71C7656EC7a5f6d8A7C4","sourceIp":"203.0.113.55","phone":"19901012345","address":"123 Main St, Anytown, USA","birthday":null}}}

      Mask sensitive fields in the log, such as the wallet address, location address, source IP address, phone number, and transaction hash.

      		 
      *| extend content =  mask(content,'[
           {"mode":"keyword","keys":["wallet","address","sourceIp","phone","transactionHash"], "maskChar":"*","keepPrefix":3,"keepSuffix":3}
         ]')
      		 
      2025-08-20 18:04: 40, 998 INFO blockchain-event-poller-3 [10.0.1.20][com.service.listener.TransactionStatusListener: 65]][TransactionStatusListener#handleSuccessfulTransaction]{"message": "On-chain transaction successfully confirmed", "confirmationDetails": {"transactionHash": "0x2**************979", "status": "success", "blockNumber": 45101239, "gasUsed": 189543, "effectiveGasPrice": "58.2 Gwei", "userProfileSnapshot": {"wallet": "0x7****************7C4", "sourceIp": "203******.55", "phone": "199*****345", "address": "123****************USA", "birthday": null}}}

    • Use regular expressions

      The SPL extend command, combined with SQL functions for regular expressions, strings, and URLs, can also mask data.

      The regexp_replace function supports capturing groups. In the replacement string, you can use \1, \2, and so on to refer to the captured values.

      For example, the statement regexp_replace('192.168.1.1', '(\d+)\.(\d+)\.\d+\.\d+', '\1.\2.*.*') returns 192.168.*.*.

      Raw data

      Description

      SPL statement

      Result

      		 
      request_uri: /api/v1/resources?user=123&ticket=abc
status: 200

      Remove sensitive information from the request_uri field.

      		 
      * | extend request_uri=url_extract_path(request_uri)

      or

      * | extend request_uri=regexp_replace(request_uri, '\?.*', '')
      		 
      request_uri: /api/v1/resources
status: 200
      		 
      client_ip: 192.168.1.123
latency: 100

      Mask the IP address by replacing the middle two octets with asterisks (*).

      		 
      * | extend client_ip=regexp_replace(client_ip, '(\d+)\.\d+\.\d+\.(\d+)', '\1.*.*.\2')
      		 
      client_ip: 192.*.*.123
latency: 100
      		 
      sql: SELECT id, name, config FROM app_info WHERE name="test-app"
result_size: 1024

      The sql field may contain sensitive information. Keep only the operation and the table name.

      		 
      *
| extend table=regexp_extract(sql, '\bFROM\s+([^\s;]+)|\bINTO\s+([^\s;]+)|\bUPDATE\s+([^\s;]+)', 1)
| extend action=regexp_extract(sql,'\b(SELECT|INSERT|UPDATE|DELETE|CREATE|DROP|ALTER)\b', 1)
| project-away sql
      		 
      action: SELECT
table: app_info
result_size: 1024
Previous：Manage ingest processorsNext：Use ingest processors to process Alibaba Cloud service logs