Consolidate log data from multiple LogStores

Simple Log Service allows you to use data transformation jobs to consolidate data from multiple source LogStores into a single LogStore. This topic describes a typical use case and provides the required steps.

Background information

A news website with globally distributed services stores user access logs from different news channels in LogStores across multiple Alibaba Cloud accounts. To consolidate this log data into a single LogStore for centralized query and analysis, you can use the e_output function in a data transformation job.

This topic provides an example of consolidating logs from different LogStores that are located in the same region, UK (London). The following section describes the logs:

Account 1 stores its raw logs in LogStore_1, which is in Project_1 in the UK (London) region.

"Log 1"
request_id: 1
http_host:  example.com
http_status:  200
request_method:  GET
request_uri:  /pic/icon.jpg
"Log 2"
request_id: 2
http_host:  aliyundoc.com
http_status:  301
request_method:  POST
request_uri:  /data/data.php

Account 2 stores its logs in LogStore_2, which is in Project_2 in the UK (London) region.

"Log 1"
request_id: 3
host:  example.edu
status:  404
request_method:  GET
request_uri:  /category/abc/product_id
"Log 2"
request_id: 4
host:  example.net
status:  200
request_method:  GET
request_uri:  /data/index.html

Transformation requirements
- Export log events that have an http_status of 200 from LogStore_1 in account 1 to LogStore_3 in account 3.
- For log events from LogStore_2 in account 2 where the status is 200, rename host to http_host and status to http_status to match the field names in LogStore_1. Then, export the transformed logs to LogStore_3 in account 3.

Step 1: Configure transformation for LogStore_1

Go to the data transformation page for LogStore_1 in account 1. For detailed steps, see Create a data transformation job.
On the data transformation page, configure the following rule to export log events with an http_status of 200 from LogStore_1 in account 1 to LogStore_3 in account 3.
```
e_if(e_match("http_status", "200"), e_output("target_logstore"))
```
Create a data transformation job and configure parameters such as Storage Destination, Destination Name, Destination Region, Destination Project, and Destination Logstore in the Storage Destination section. For a description of the authorization method parameter, see Create a data transformation job.

Set Destination Name to target_logstore. For Destination Region, select UK (London). For Destination Project, enter Project_3. For Target Store, enter LogStore_3. For Authorization Method, select AccessKey, enter the AccessKey ID and AccessKey Secret, and then click Add.

Step 2: Configure transformation for LogStore_2

Go to the data transformation page for LogStore_2 in account 2. For detailed steps, see Create a data transformation job.
On the data transformation page, configure the following rule. This rule processes logs from LogStore_2 where the status is 200. It renames host to http_host and status to http_status to match the field names in LogStore_1, and then exports the transformed logs to LogStore_3 in account 3.
```
e_if(e_match("status", "200"), e_compose(e_rename("status", "http_status", "host", "http_host"), e_output("target_logstore")))
```
In the preview results, the fields have been renamed as expected: host is renamed to http_host, and status is renamed to http_status.
Create a data transformation job. In the Storage Destination area, configure parameters such as Destination Name, Destination Region, Destination Project, and Destination Logstore. For a description of the Authorization Method parameter, see Create a data transformation job.

For Destination Name, enter target_logstore. For Destination Region, select UK (London). For Destination Project, enter Project_3. For Target Store, enter LogStore_3. For Authorization Method, select AccessKey, enter the AccessKey ID and AccessKey Secret, and then click Add.

View aggregation results

In LogStore_3 in the UK (London) region, query and analyze the logs. The following are sample logs.

"Log 1"
request_id: 1
http_host:  example.com
http_status:  200
request_method:  GET
request_uri:  /pic/icon.jpg
"Log 2"
request_id: 4
http_host:  example.net
http_status:  200
request_method:  GET
request_uri:  /data/index.html