Data transformation (new version) quick start - Simple Log Service

Filter website access logs, remove personal user information, and write the results to a destination LogStore by using data transformation (new version) in Simple Log Service. This quickstart walks you through the transformation workflow, from writing an SPL rule to observing the resulting job.

Background information

A website stores all of its access logs in a LogStore named website_log. To improve user experience, the website needs to analyze access errors. The goal is to filter access logs with a 4XX status code, remove personal user information, and write the results to a new LogStore named website_fail for business analysts. The following sample log entry is provided:

body_bytes_sent: 1061
http_user_agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ru-RU) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5
remote_addr: 192.0.2.2
remote_user: vd_yw
request_method: GET
request_uri: /request/path-1/file-5
status: 400
time_local: 10/Jun/2021:19:10:59
error: Invalid time range

Prerequisites

(Required) A project named web-project is created. For more information, see Manage projects.
(Required) A source LogStore named website_log is created in the web-project project. For more information, see Manage LogStores.
(Required) Website access logs are collected in the source LogStore (website_log). For more information, see Data collection overview.
(Required) A destination LogStore named website_fail is created in the web-project project.
(Conditional) If you use a Resource Access Management (RAM) user, you must first grant the RAM user the permissions for data transformation. For more information, see Authorize a RAM user to perform data transformation operations.
(Optional) Indexes are configured for both the source LogStore and the destination LogStore. For more information, see Create an index.
Note
Indexes are not required for data transformation jobs. However, if indexes are not configured, you cannot perform query and analysis operations.

Step 1: Create a data transformation job

Log on to the Simple Log Service console.
Go to the data transformation page.
1. In the Projects section, click the project you want.
2. On the Log Storage > Logstores tab, click the logstore you want.
3. On the query and analysis page, click Data Transformation.
In the upper-right corner of the page, select a time range.
After selecting a time range, verify that logs appear on the Raw Logs tab.
In the editor, enter the following transformation SPL rule.
```
*
| extend status=cast(status as BIGINT)
| where status>=400 AND status<500
| project-away remote_addr, remote_user
```
The rule applies three operators in sequence:
- extend status=cast(status as BIGINT) converts the status field to the BIGINT type.
- where status>=400 AND status<500 keeps only the access logs whose status code is 4XX.
- project-away remote_addr, remote_user removes the remote_addr and remote_user fields, which carry the personal user information.
Debug the SPL rule.
1. On the Raw Data tab, select test data or manually enter test data.
```
[
{
    "body_bytes_sent": "1061",
    "http_user_agent": "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru-RU) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5",
    "remote_addr": "192.0.2.2",
    "remote_user": "vd_yw",
    "request_method": "GET",
    "request_uri": "/request/path-1/file-5",
    "status": "400",
    "time_local": "10/Jun/2021:19:10:59",
    "error": "Invalid time range"
}
]
```
2. Click the run icon (▷) to run the debug.
3. View the preview results. After the execution is complete, switch to the Transformation Results tab to view the results. The transformed log is structured into multiple fields, such as body_bytes_sent, error, http_user_agent, request_method, request_uri, status, and time_local. Verify that remote_addr and remote_user are no longer present and that only access logs with a 4XX status code remain.

Create a data transformation job.

Click Save as Transformation Job (New Version).
In the Create Data Transformation Job (New Version) panel, configure the following parameters and click OK.

Parameter	Description
Task Name	The name of the data transformation job.
Display Name	The display name of the job.
Job Description	The description of the job.
Authorization Method	Grant the job read access to the source logstore using one of these methods: Default Role: The job assumes the AliyunLogETLRole system role to read data from the source logstore. Click Authorize the system role AliyunLogETLRole and complete the authorization as prompted. For more information, see Access data using a default role. Important If you use a RAM user, an Alibaba Cloud account must complete the authorization first. If your Alibaba Cloud account is already authorized, you can skip this step. Custom Role: The job assumes a custom role to read data from the source logstore. You must first grant the custom role permissions to read data from the source logstore, and then enter the ARN of the role in the Role ARN field. For more information, see Access data using a custom role. AccessKey: For security reasons, you can no longer use an AccessKey pair (AK/SK) to create jobs.
Storage Destination
Destination Name	The storage destination name. Includes configurations such as the Project and logstore.
Destination Region	The region where the destination Project is located.
Destination Project	The destination Project for transformation results. A Project specified in your SPL statement overrides this setting. Dynamic destination Project/logstore output. Important The Project that you dynamically specify in the SPL statement must match the region and authorization that you configure here.
Target Store	The destination logstore for transformation results. A logstore specified in your SPL statement overrides this setting. Dynamic destination Project/logstore output. Important The logstore that you dynamically specify in the SPL statement must match the region, authorization, and Project that you configure here. The destination logstore cannot be the same as the source logstore. Warning Do not configure the destination store as the current source store (same-source configuration). Otherwise, logs may be written in a loop, which incurs additional storage and traffic costs. You are responsible for the resource consumption and costs incurred.
Authorization Method	Grant the job write access to the destination logstore using one of these methods: Default Role: The job assumes the AliyunLogETLRole system role to write results to the destination logstore. Click Authorize the system role AliyunLogETLRole and complete the authorization as prompted. For more information, see Access data using a default role. Important If you use a RAM user, an Alibaba Cloud account must complete the authorization first. If your Alibaba Cloud account is already authorized, you can skip this step. Custom Role: The job assumes a custom role to write results to the destination logstore. You must first grant the custom role permissions to write data to the destination logstore, and then enter the ARN of the role in the Role ARN field. For more information, see Access data using a custom role. AccessKey: For security reasons, you can no longer use an AccessKey pair (AK/SK) to create jobs.
Write to Result Set	The dataset to write to the destination logstore. Multiple datasets can target one destination, and multiple destinations can share a dataset. Dataset description.
Processing scope
Time Range (Data Receiving Time)	The time range for the job: All: Processes all data from the first log entry until manually stopped. From Specific Time: Processes data from the specified start time until manually stopped. Specific Time Range: Processes data within the specified time window and stops automatically at the end time.
Advanced Options
Advanced Parameter Settings	Store sensitive information (such as database passwords) as key-value pairs, and reference them in your statement using `res_local("key")`. Click + to add multiple key-value pairs. For example, `config.vpc.vpc_id.test1:vpc-uf6mskb0b****n9yj` specifies the ID of the VPC where the RDS instance resides.

Step 2: Observe the data transformation job

In the left-side navigation pane, choose Job Management > Data Transformation.
In the list of data transformation jobs, find and click the data transformation job that you want to manage.
On the Data Transformation Overview (New Version) page, view the job details and status. You can also observe the running status and metrics of the job. For more information, see Observe and monitor data transformation jobs (new version). To modify, start, stop, or delete the job, see Manage data transformation jobs (new version).
Confirm that the transformation results are written to the destination LogStore. Go to the destination LogStore (website_fail) to perform query and analysis operations. For more information, see Quick guide to query and analysis.