Ingest processors preprocess logs before they are written to a Logstore. You can use ingest processors to filter data, extract fields, extend fields, and mask data.
How it works
When you collect log data using Logtail, the Simple Log Service SDK, or web tracking, the data is first processed by an ingest processor and then written to a Logstore. The data processing is performed within Simple Log Service and does not consume client resources.
Ingest processors, log query and analysis, and data transformation all support SPL syntax.
Scenarios
Field extraction: Extract new fields from raw logs using parsing methods such as regular expressions, Key-Value format, and JSON.
Field extension: Add new fields to raw logs.
Field deletion: Delete specific fields from raw logs.
Data masking: Mask sensitive information in raw logs.
Data filtering: Discard specific data from raw logs.
Limits
Limit | Description |
Number of ingest processors | Each project supports a maximum of 100 ingest processors. |
Length of SPL statements in an ingest processor configuration | Each SPL statement can be up to 4,000 characters long. |
SPL instruction limits in an ingest processor | Only row processing instructions are supported. Instructions such as aggregation and logical judgment are not supported. |
Number of ingest processors associated with each Logstore |
|
Effective time after an ingest processor is updated or deleted | Changes to an ingest processor configuration or the association between a Logstore and an ingest processor take effect within one minute. |
Billing
Ingest processors are billed based on the amount of compute resources consumed during data processing. The billing unit is the Observability Capacity Unit (OCU).
The Observability Capacity Unit (OCU) is a new billing unit from Alibaba Cloud-native Observability. OCU usage is calculated automatically based on hourly resource consumption.
The billing for compute-optimized features of SLS is gradually shifting to an OCU-based model, which bills based on the actual compute resources consumed. In CPU scenarios, one OCU is equivalent to approximately 0.5 CPU cores, 2 GB of memory, and 3,000 IOPS. To calculate the total number of OCUs, SLS determines three separate OCU counts based on the consumed CPU cores, memory, and IOPS. The highest of these three values is used as the final OCU value for billing.
Suppose a computation job consumes 1 CPU core, 2 GB of memory, and 3,000 IOPS. In this case, the job consumes 2 OCUs. In a 1-hour billing cycle, an ingest processor consumes approximately 1/3 of an OCU to process 1 GB of data. The new version of data transformation consumes approximately 1/3 of an OCU to process 1 GB of data. Rule-based consumption consumes approximately 0.3 OCUs to process 1 GB of data.
For billing information about OCUs, see Billable items of pay-by-ingested-data and Billable items for the pay-by-feature billing model.