Log Service provides a fully managed, highly available, and scalable data transformation feature. This feature is widely used in scenarios such as data standardization, enrichment, distribution, aggregation, and re-indexing.

The data transformation feature transforms data by performing the following steps:
  1. Uses a consumer group to read log data from the source Logstore.
  2. Transforms every log entry based on the transformation rule.
  3. Writes transformed log data to one or more destination Logstores.
After the data is transformed, you can view the data in the destination Logstores.
Note The data transformation feature is available in all regions except the China (Qingdao) region.

Scenarios

  • Data standardization: Log data is read from a Logstore, transformed, and written to another Logstore.
    In this scenario, data is standardized, enriched, and re-indexed.Data standardization
  • Data distribution: Log data is read from a Logstore, transformed, and written to multiple Logstores.Data distribution
  • Multi-source data aggregation (many-to-one): Log data is read from multiple Logstores, transformed, and written to a specified Logstore.Multi-source data aggregation
  • Common data transformation scenarios.
    Common data transformation scenarios include data filtering, splitting, transformation, and enrichment.Data transformation
    Log Service provides more than 200 built-in functions and more than 400 regular expressions. You can also use the domain specific language (DSL) for Log Service to create user-defined functions (UDFs) based on your business requirements. The following scenarios are common scenarios that you may encounter:
    • Filters out specified logs.
    • Splits a log entry into multiple log entries.
    • Extracts, deletes, and modifies fields and transforms the field values.
    • Maps fields from external resources and enriches raw logs with the fields.

Benefits

  • Provides more than 200 built-in functions, including text processing functions, text search functions, and enrichment functions, and more than 400 grok patterns.
  • Allows you to use the DSL for Log Service to orchestrate functions as needed. Then you can use the functions to filter, extract, split, transform, enrich, and distribute data.
  • Processes data in real time and allows you to view data in seconds. Automatically extends or shrinks the computation capability based on the data size. Provides a high throughput.
  • Applies to log analysis scenarios and provides out-of-the-box functions.
  • Provides charts and dashboards for real-time data analysis, and alerts for exception logs.
  • Offers a fully managed and maintenance-free service that can be integrated with Alibaba Cloud big data services and open-source ecosystems.

Billing

  • You are charged for data reads from the source Logstore and data writes to one or more destination Logstores. For more information, see Pay-as-you-go. The server and network resources that you consume during data transformation are free of charge.
  • You can delete the indexes that you create in the source Logstore and set relatively short data retention period for the data. This reduces your costs. For more information about how to optimize costs, see Performance guide or Cost optimization guide.