This topic describes a complete data transformation process to walk you through the data transformation feature and related operations. Website access logs are used as an example to describe the process.
Prerequisites
- A project named web-project is created. For more information, see Create a project.
- A Logstore named website_log is created in the web-project project, and the Logstore is used as the source Logstore. For more information, see Create a Logstore.
- Website access logs are collected and stored in the website_log Logstore. For more information, see Data collection overview.
- Destination Logstores are created in the web-project project. The following table
lists the details about the destination Logstores.
Destination Logstore Description website-success Logs for successful access are stored in the website-success Logstore, which is configured in the target-success storage destination. website-fail Logs for failed access are stored in the website-fail Logstore, which is configured in the target-fail storage destination. website-etl Other access logs are stored in the website-etl Logstore, which is configured in the target0 storage destination. - If you use a Resource Access Management (RAM) user, you must grant the user the permissions to transform data. For more information, see Authorize a RAM user to manage a data transformation task.
- Indexes are configured for the source and destination Logstores. For more information,
see Configure indexes.
Note Data transformation does not require indexes. However, if you do not configure indexes, you cannot perform query or analysis operations.
Background information
All access logs of a website are stored in a Logstore. You need to specify different
topics for the logs to distinguish between logs for successful access and logs for
failed access. In addition, you need to distribute the two types of logs to different
Logstores for analysis. Log sample:
body_bytes_sent:1061
http_user_agent:Mozilla/5.0 (Windows; U; Windows NT 5.1; ru-RU) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5
remote_addr:192.0.2.2
remote_user:vd_yw
request_method:DELETE
request_uri:/request/path-1/file-5
status:207
time_local:10/Jun/2021:19:10:59
Step 1: Create a data transformation job
After logs are distributed to the destination Logstores, you can perform query and analysis operations on the destination Logstores. For more information, see Query and analyze logs.