An ingest processor processes log data before it is written to a Logstore. Use an ingest processor to filter data, extract or add fields, and mask sensitive information.
How it works
-
Log data collected through Logtail, the Simple Log Service SDK, or web tracking passes through the ingest processor before being written to a Logstore. Processing runs within Simple Log Service and consumes no client-side resources.
-
The ingest processor, log query and analysis, and data transformation all support SPL syntax.
Use cases
-
Field extraction: Extract fields from raw logs using regular expressions, key-value pairs, or JSON parsing.
-
Field extension: Add new fields to raw logs.
-
Field deletion: Remove specific fields from raw logs.
-
Data masking: Mask sensitive information in raw logs.
-
Data filtering: Discard unwanted log entries.
Limitations
|
Item |
Limit |
|
Ingest processors per project |
Up to 100 per project. |
|
SPL statement length |
Up to 4,000 characters per SPL statement. |
|
Supported SPL commands |
Row-level commands only. Aggregation and conditional logic are not supported. |
|
Ingest processors per Logstore |
|
|
Time for changes to take effect |
Configuration changes, deletions, or Logstore association updates take effect within one minute. |
Billing
Ingest processors are billed in OCUs based on computing resources consumed.
The Observability Capacity Unit (OCU) is a new billing unit from Alibaba Cloud-native Observability. OCU usage is automatically calculated based on hourly resource consumption.
Billing for compute-intensive features of Log Service (SLS) is shifting to an OCU-based model that bills based on the computing resources consumed. In CPU-based scenarios, one OCU is equivalent to approximately 0.5 CPU cores, 2 GB of memory, and 3,000 IOPS. To calculate the total OCU count, SLS calculates separate OCU values for CPU core, memory, and IOPS consumption. The highest of these values determines the final OCU value for billing.
Suppose a computation job consumes 1 CPU core, 2 GB of memory, and 3,000 IOPS. In this case, the job consumes 2 OCUs. In a one-hour metering period, processing 1 GB of data consumes approximately 1/3 of an OCU for the ingest processor, 1/3 of an OCU for data transformation (new version), and 0.3 OCUs for rule-based consumption.
Pricing: Billable items of the pay-by-ingested-data mode and Billable items of the pay-by-feature billing model.