Log Service provides you with a fully managed, highly available, and scalable data processing service. This data processing feature is widely used in scenarios such as data standardization, enrichment, distribution, aggregation, and reindexing.

The data processing procedure can be divided into three steps: First, use a consumer group to read and consume log data in the source Logstore. Then, process each log that is read. Finally, write the processed log data to the specified destination Logstore. You can view the processed log data in the destination Logstore.

The data processing feature is available in regions excluding China (Qingdao).

Scenarios

  • Data standardization (one-to-one): Log data is read from a Logstore and written to another Logstore after being processed.
    In this scenario, data is standardized and enriched, and relevant indexes are re-created.Data standardization
  • Data distribution (one-to-many): Log data is read from a Logstore and written to different Logstores after being processed.Data distribution
  • Multi-source data aggregation (many-to-one): Log data is read from different Logstores and written to a specified Logstore after being processed.Multi-source data aggregation
  • Common data processing.
    Common data processing covers all typical data processing scenarios to filter, split, transform, and enrich data.Data processing
    LOG domain specific language (DSL) provides more than 200 built-in functions and more than 400 regular expression patterns and allows you to create user-defined functions (UDFs) to meet the needs in various scenarios:
    • Filters out specified logs.
    • Splits a log into multiple logs.
    • Extracts, deletes, and modifies certain fields and transforms the field content.
    • Associates fields with external resources and enriches field information.

Benefits

  • Provides more than 200 built-in functions, including text processing functions, text search functions, and enrichment functions, and more than 400 grok patterns.
  • Allows you to use DSL to orchestrate operations as needed. For example, you can filter, extract, split, transform, enrich, and distribute data.
  • Processes data in real time and allows you to view data in seconds. Automatically extends or shrinks the computation capability based on the data size. Provides a high throughput.
  • Applies to log analysis scenarios and provides out-of-the-box functions.
  • Provides the real-time gauge, overview and statistical curves, and integrated exception log and alerting feature.
  • Offers a fully managed and maintenance-free service that can be integrated with Alibaba Cloud big data products and open-source ecosystems.

Billing

Expenses for reading data from the source Logstore and writing data to the destination Logstore are charged based on the billing standards of Log Service. For more information, see Billing method. Currently, you are not charged for servers and network resources used by the data processing feature.