If you need to process log data before it is written to a Logstore, such as for data filtering, field extraction, field extension, or data masking, you can use an ingest processor.
How it works
-
When you collect log data using Logtail, the Simple Log Service SDK, or web tracking, the data first passes through an ingest processor and is then written to a Logstore. The data processing occurs within Simple Log Service and does not consume client-side resources.
-
The ingest processor, log query and analysis, and data transformation all support SPL syntax.
Use cases
-
Field extraction: Extract new fields from raw log fields using parsing methods such as regular expressions, key-value, or JSON.
-
Field extension: Add new fields to raw logs.
-
Field deletion: Remove specific fields from raw logs.
-
Data masking: Mask sensitive information in raw logs.
-
Data filtering: Discard unwanted log entries.
Limitations
|
Limit |
Description |
|
Number of ingest processors |
You can create up to 100 ingest processors in each Project. |
|
Length of SPL statements in an ingest processor configuration |
Each SPL statement cannot exceed 4,000 characters. |
|
SPL command limitations in an ingest processor |
Only row-level processing commands are supported. Aggregation and conditional logic commands are not supported. |
|
Number of ingest processors associated with a Logstore |
|
|
Time for changes to take effect |
Changes to an ingest processor configuration, deletions of ingest processors, or modifications to Logstore associations take effect within one minute. |
Billing
Ingest processors are billed in OCUs based on the computing resources consumed during data processing.
The Observability Capacity Unit (OCU) is a new billing unit for Alibaba Cloud-native Observability. OCU usage is automatically calculated based on hourly resource consumption.
The billing for compute-optimized features of SLS is gradually shifting to an OCU-based model, which bills based on the actual compute resources consumed. In CPU scenarios, one OCU is equivalent to approximately 0.5 CPU cores, 2 GB of memory, and 3,000 input/output operations per second (IOPS). To calculate the total number of OCUs, SLS determines three separate OCU counts based on the consumed CPU cores, memory, and IOPS. The highest of these three values is used as the final OCU value for billing.
For example, if a computation job consumes 1 CPU core, 2 GB of memory, and 3,000 IOPS, the job consumes 2 OCUs. In a 1-hour billing cycle, an Ingest Processor consumes approximately 1/3 of an OCU to process 1 GB of data. The new version of data transformation consumes approximately 1/3 of an OCU to process 1 GB of data. Rule-based consumption consumes approximately 0.3 OCUs to process 1 GB of data.
For more information about OCU billing, see Billable items of the pay-by-ingested-data mode and Billable items of the pay-by-feature billing model.