All Products
Search
Document Center

Log Service:Overview of data transformation

Last Updated:Oct 12, 2023

Data transformation is a fully managed feature that provides high availability and scalability in Simple Log Service. You can use the data transformation feature to standardize, enrich, transfer, mask, and filter data.

Transformation process

Simple Log Service transforms data in three steps.

  1. A consumer group reads data from a source Logstore.

  2. Simple Log Service transforms each data entry based on a transformation rule.

  3. Simple Log Service writes the transformed data to a destination Logstore.

    After data is transformed, you can view the results in the destination Logstore.

Features

Simple Log Service

  • Data standardization: Simple Log Service can extract fields from logs in different formats and convert the log formats to obtain structured data for stream processing and computing in data warehouses.

  • Data enrichment: Simple Log Service can join the fields of logs and dimension tables to link logs with dimension information. For example, Simple Log Service can join the fields of order logs and a user information table. This facilitates data analysis.

  • Data transfer: Simple Log Service can transfer logs from regions outside the Chinese mainland to a central region by using the global acceleration feature. This helps you manage global logs in a centralized manner.

  • Data masking: Simple Log Service can mask sensitive information in data, such as passwords, mobile phone numbers, and addresses.

  • Data filtering: Simple Log Service can filter logs to obtain key service logs. This helps further analysis.

Scenarios

  • Data standardization: Log data is read from a source Logstore, transformed, and then written to a destination Logstore. 数据规整

  • Data transfer: Log data is read from a source Logstore, transformed, and then written to multiple destination Logstores.数据分派

  • Multi-source data aggregation: Log data is read from multiple source Logstores, transformed, and then written to a destination Logstore.多源汇集

Transformation syntax

The domain-specific language (DSL) for Simple Log Service provides more than 200 built-in functions and more than 400 regular expressions. You can use the DSL for Simple Log Service to create user-defined functions (UDFs). For more information, see Syntax overview.

Benefits

  • Allows you to use the DSL for Simple Log Service to orchestrate functions based on your business requirements. You can use the orchestrated functions to filter, standardize, enrich, transfer, and mask data.

  • Processes data in real time and allows you to view data within seconds. The feature scales the computing capability based on the size of data and provides a high throughput.

  • Is suitable for log analysis scenarios and provides out-of-the-box functions.

  • Provides real-time dashboards, exception logs, and alerts.

  • Offers a fully-managed and maintenance-free service that can be integrated with big data services of Alibaba Cloud and open source ecosystems.

Billing

  • If your Logstores use the pay-by-ingested-data billing mode, you are not charged for data transformation. However, if data is pulled over a public Simple Log Service endpoint, you are charged for read traffic over the Internet. The traffic is calculated based on the size of data after compression. For more information, see Billable items of pay-by-ingested-data.

  • If your Logstores use the pay-by-feature billing mode, you are charged for data transformation based on the machine and network resources that are consumed. For more information, see Billable items of pay-by-feature.

  • You can disable the indexing feature for source Logstores and shorten the data retention period of the Logstores to reduce costs. For more information, see Performance guide and Cost optimization guide.