An ingest processor processes log data before it is written to a Logstore. Use an ingest processor to filter data, extract or add fields, and mask sensitive information.
How it works
Log data collected through Logtail, the Simple Log Service SDK, or web tracking passes through the ingest processor before being written to a Logstore. Processing runs within Simple Log Service and consumes no client-side resources.
The ingest processor, log query and analysis, and data transformation all support SPL syntax.
Use cases
Field extraction: Extract fields from raw logs using regular expressions, key-value pairs, or JSON parsing.
Field extension: Add new fields to raw logs.
Field deletion: Remove specific fields from raw logs.
Data masking: Mask sensitive information in raw logs.
Data filtering: Discard unwanted log entries.
Limitations
Item | Limit |
Ingest processors per project | Up to 100 per project. |
SPL statement length | Up to 4,000 characters per SPL statement. |
Supported SPL commands | Row-level commands only. Aggregation and conditional logic are not supported. |
Ingest processors per Logstore |
|
Time for changes to take effect | Configuration changes, deletions, or Logstore association updates take effect within one minute. |
Billing
Ingest processors are billed in OCUs based on computing resources consumed.
The Observability Capacity Unit (OCU) is a new billing unit from Alibaba Cloud-native Observability. OCU usage is automatically calculated based on hourly resource consumption.
Billing for the compute-optimized features of SLS is transitioning to an OCU-based model that charges based on actual compute resource consumption. In CPU scenarios, one OCU is equivalent to approximately 0.5 CPU cores, 2 GB of memory, and 3,000 IOPS. SLS calculates separate OCU values based on the consumption of CPU cores, memory, and IOPS. The highest of these three values determines the final OCU count for billing.
Suppose a computation job consumes 1 CPU core, 2 GB of memory, and 3,000 IOPS. In this case, the job consumes 2 OCUs. In a one-hour metering period, processing 1 GB of data consumes approximately 1/3 of an OCU for the ingest processor, 1/3 of an OCU for data transformation (new version), and 0.3 OCUs for rule-based consumption.
Pricing: Billable items of the pay-by-ingested-data mode and Billable items of the pay-by-feature billing model.