Data transformation is a fully managed feature that provides high availability and scalability in Simple Log Service. You can use the data transformation feature to standardize, enrich, transfer, mask, and filter data.
Transformation process
The data transformation service processes data in the following three steps.
A consumer group reads data from a source Logstore.
Simple Log Service transforms each data entry based on a transformation rule.
Simple Log Service writes the transformed data to a destination Logstore.
After data is transformed, you can view the results in the destination Logstore.
Features
Simple Log Service provides a data transformation feature for structuring, enriching, distributing, desensitizing, and filtering data. The details are as follows:
Data standardization: Simple Log Service can extract fields from logs in different formats and convert the log formats to obtain structured data for stream processing and computing in data warehouses.
Data enrichment: Simple Log Service can join the fields of logs and dimension tables to link logs with dimension information. For example, Simple Log Service can join the fields of order logs and a user information table. This facilitates data analysis.
Data transfer: Simple Log Service can transfer logs from regions outside the Chinese mainland to a central region using the global acceleration feature. This helps you manage global logs in a centralized manner.
Data masking: Simple Log Service can mask sensitive information in data, such as passwords, mobile phone numbers, and addresses.
Data filtering: Simple Log Service can filter logs to obtain key service logs. This helps further analysis.
Scenarios
Data standardization: Log data is read from a source Logstore, transformed, and then written to a destination Logstore.
Data transfer: Log data is read from a source Logstore, transformed, and then written to multiple destination Logstores.
Multi-source data aggregation: Log data is read from multiple source Logstores, transformed, and then written to a destination Logstore.
Transformation syntax
The Simple Log Service (SLS) domain-specific language (DSL) provides more than 200 built-in functions and 400 regular expression patterns. For more information, see Syntax overview.
Benefits
Allows you to use the DSL for Simple Log Service to orchestrate functions as needed. You can use the orchestrated functions to filter, standardize, enrich, transfer, and mask data.
Processes data in real time and lets you view data within seconds. The feature scales the computing capability based on the size of data and provides a high throughput.
Is suitable for log analysis scenarios and provides out-of-the-box functions.
Provides real-time dashboards, exception logs, and alerts.
Offers a fully-managed and maintenance-free service that can be integrated with big data services of Alibaba Cloud and open source ecosystems.
Billing
If your Logstores use the pay-by-ingested-data billing mode, you are not charged for data transformation. However, if data is pulled over a public Simple Log Service endpoint, you are charged for read traffic over the Internet. The traffic is calculated based on the size of data after compression. For more information, see Billable items of pay-by-ingested-data.
If your Logstores use the pay-by-feature billing mode, you are charged for data transformation based on the machine and network resources that are consumed. For more information, see Billable items of pay-by-feature.
You can disable the indexing feature for source Logstores and shorten the data retention period of the Logstores to reduce costs. For more information, see Performance guide and Cost optimization guide.