数据加工流程及操作 - Simple Log Service - Alibaba Cloud Documentation Center

This topic uses website access logs to demonstrate the data transformation process and help you quickly learn its features and operations.

Prerequisites

A project named web-project is created. For more information, see Manage projects.
A source Logstore named website_log is created in the web-project project. For more information, see Create a basic Logstore.
Website access logs are collected and stored in the source Logstore (website_log). For more information, see Data collection.

Destination Logstores are created in the web-project project. The following table describes the destination Logstores.

Destination Logstore	Description
website-success	Stores logs for successful access requests in the `website-success` Logstore, which corresponds to the `target-success` storage target.
website-fail	Stores logs for failed access requests in the `website-fail` Logstore, which corresponds to the `target-fail` storage target.
website-etl	Stores all other access logs in the `website-etl` Logstore, which corresponds to the `target0` storage target.

If you use a RAM user, grant the RAM user the required permissions to perform data transformation operations. For more information, see Grant permissions to a RAM user for data transformation.
Indexes are configured for the source and destination Logstores. For more information, see Create an index.

Important
A data transformation job does not depend on an index. However, you cannot query or analyze data if you do not configure an index.

Background

A website stores all its access logs in a single Logstore. You want to set different topics for successful and failed access logs and then route them to different Logstores for easier analysis. The following is a sample log:

body_bytes_sent:1061
http_user_agent:Mozilla/5.0 (Windows; U; Windows NT 5.1; ru-RU) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5
remote_addr:192.0.2.2
remote_user:vd_yw
request_method:DELETE
request_uri:/request/path-1/file-5
status:207
time_local:10/Jun/2021:19:10:59

Step 1: Create a data transformation job

Log on to the Simple Log Service console.
Go to the data transformation page.
1. In the Projects section, click the project you want.
2. On the Log Storage > Logstores tab, click the logstore you want.
3. On the query and analysis page, click Data Transformation.
In the upper-right corner, select a time range for the data.

Make sure that logs appear on the Raw Logs tab after you select the time range.
In the editor, enter the transformation statements.
```
e_if(e_search("status:[200,299]"),e_compose(e_set("__topic__","access_success_log"),e_output(name="target-success")))
e_if(e_search("status:[400,499]"),e_compose(e_set("__topic__","access_fail_log"),e_output(name="target-fail")))
```
The e_if function performs a specified operation when a condition is met. For more information, see e_if.
- Condition: e_search("status:[200,299]")
  
  When the value of the status field meets the condition, perform Operation 1 and Operation 2. For more information, see e_search.
- Operation 1: e_set("__topic__","access_success_log")
  
  Add the topic field and set its value to access_success_log. For more information, see e_set.
- Operation 2: e_output(name="target-success", project="web-project", logstore="website-success")
  
  This operation writes the transformed data to a storage target. In this example, the data is written to the website-success Logstore. For more information, see e_output.
Preview the data.
1. Click Quick.
  
  Simple Log Service supports Quick and Advanced preview modes. For more information, see Preview and debug data.
2. Click Preview Data.
  
  View the preview results.
  
  Important
  During the preview, logs are not sent to the destination Logstore. Instead, they are sent to a Logstore named internal-etl-log. Simple Log Service automatically creates this dedicated Logstore in the current project the first time you preview a data transformation. You cannot modify its configuration or write other data to it. This Logstore is free of charge.
  
  Click the Transformation Results tab. Verify that the log is successfully delivered to target-success. The __topic__ field has a value of access_success_log, and the status field has a value of 207. This indicates that the transformation rule correctly matched and routed the log.

Create the data transformation job.

Click Save as Transformation Job.

In the Create Data Transformation Job panel, configure the following parameters.

Parameter	Description
Job name	The name of the data transformation job.
Authorization method	The data transformation job needs permission to read data from the source Logstore. Grant this permission using one of the following methods: Default Role: The job assumes the `AliyunLogETLRole` system role to read data from the source logstore. Custom Role: The job assumes a custom role to read data from the source logstore. You must first grant the custom role permissions to read data from the source logstore. Then, enter the ARN of the custom role in the Role ARN field. For more information, see Use a custom role to access data.
Storage target
Target name	The name of the storage target. A storage target includes settings such as the project and Logstore. Must match the setting in step 4: the name. Note Simple Log Service uses the first storage target that you configure (in this example, `target0`) as the default target to store logs that do not match any other conditions.
Target region	Select the region where the destination project is located. Cross-region data transformation is secured using HTTPS. Cross-region data transformation transmits data over the public network, which may cause job delays due to network instability. You can select the DCDN Acceleration checkbox to speed up cross-region transmission. When you use DCDN acceleration, make sure that the acceleration feature is enabled for the corresponding project. For more information, see Accelerate log collection. Important If data is pulled from a Simple Log Service endpoint over the public network, you are charged for outbound public network traffic, which is calculated based on the compressed data size. For more information, see Pay-by-feature billable items.
Target project	The destination project for the transformation results.
Target Logstore	The destination Logstore for the transformation results.
Authorization method	The data transformation job needs permission to write to the destination Logstore. Grant this permission using one of the following methods: Default Role: The job assumes the `AliyunLogETLRole` system role to write transformation results to the destination Logstore. Custom Role: The job assumes a custom role to write transformation results to the destination Logstore. You must first grant the custom role permissions to write data to the destination Logstore. Then, enter the ARN of the custom role in the Role ARN field. For more information, see Use a custom role to access data.
Processing range
Time range	Specify the time range of data to process. Note The time range is based on when Simple Log Service receives the logs. All: Processes data from the time the first log is received in the Logstore until the job is manually stopped. From Specific Time: Specifies a start time. The job processes data from the specified time until it is manually stopped. Within Specific Period: Specifies a start and end time. The job automatically stops after it processes data up to the specified end time.

Click OK.

After the logs are routed to the destination Logstores, you can query and analyze them. For more information, see Quick start for log query and analysis.

Step 2: View the data transformation job

In the left-side navigation pane, choose Job Management > Data Transformation.
In the list of data transformation jobs, click the target job.
On the Data Transformation Overview page, view the job details.

You can view job details and status, and modify, start, stop, or delete the job. For more information, see Manage a data transformation job.

On the job details page, the Basic information section shows that the source Logstore is website-log, the status is Running, and the time range is Continuous. The Consumption progress table shows normal consumption for both shard 0 and shard 1. The Storage target section lists three destination Logstores: target0 (website-etl), target-success (website-success), and target-fail (website-fail), all located in the web-project project in the China (Chengdu) region. The Data transformation diagnostics section below shows that Read Logs and Delivered Logs are both 12.517Mil Lines, Failed Logs is 0 Lines, and the delivery rate is 100.0%. This indicates that the data transformation job is running correctly without data loss.