Log Service allows you to aggregate data from multiple source Logstores to a single destination Logstore. To do this, you can configure a data transformation task for each source Logstore. This topic describes how to aggregate data from multiple source Logstores to a destination Logstore in a typical scenario.
Background information
Aggregate data from multiple source Logstores
- Raw log entries
- The raw log entries are stored in a Logstore named Logstore_1 of a project named Project_1.
The project belongs to Account 1 and resides in the UK (London) region.
Log entry 1 request_id: 1 http_host: example.com http_status: 200 request_method: GET request_uri: /pic/icon.jpg Log entry 2 request_id: 2 http_host: aliyundoc.com http_status: 301 request_method: POST request_uri: /data/data.php
- The raw log entries are stored in a Logstore named Logstore_2 of a project named Project_2.
The project belongs to Account 2 and resides in the UK (London) region.
Log entry 1 request_id: 3 host: example.edu status: 404 request_method: GET request_uri: /category/abc/product_id Log entry 2 request_id: 4 host: example.net status: 200 request_method: GET request_uri: /data/index.html
- The raw log entries are stored in a Logstore named Logstore_1 of a project named Project_1.
The project belongs to Account 1 and resides in the UK (London) region.
- Transformation requirements
- Aggregate the log entries whose
http_status
is 200 in Logstore_1 of Account 1 and Logstore_2 of Account 2 to Logstore_3 of Account 3. - Set the log field names in Logstore_1 of Account 1 and Logstore_2 of Account 2. Set
the field name
host
tohttp_host
and the field namestatus
tohttp_status
.
- Aggregate the log entries whose
- DSL orchestration
- Configure the following transformation rules for Logstore_1 of Account 1. On the Create Data Transformation Rule page, set Target Name to target_logstore, set Target Project to Project_3, set Target
Logstore to Logstore_3, and specify the authorization method. For more information,
see Create a data transformation task.
e_if(e_match("http_status", "200"), e_output("target_logstore"))
- Configure the following transformation rules for Logstore_2 of Account 2. On the Create Data Transformation Rule page, set Target Name to target_logstore, set Target Project to Project_3, set Target
Logstore to Logstore_3, and specify the authorization method.
e_if(e_match("status", "200"), e_compose(e_rename("status", "http_status", "host", "http_host"), e_output("target_logstore")))
- Configure the following transformation rules for Logstore_1 of Account 1. On the Create Data Transformation Rule page, set Target Name to target_logstore, set Target Project to Project_3, set Target
Logstore to Logstore_3, and specify the authorization method. For more information,
see Create a data transformation task.
- Result
The following log entries are aggregated in the Logstore_3 of Project_3 that belongs to Account 3. The project resides in the UK (London) region.
Log entry 1 request_id: 1 http_host: example.com http_status: 200 request_method: GET request_uri: /pic/icon.jpg Log entry 2 request_id: 4 http_host: example.net http_status: 200 request_method: GET request_uri: /data/index.html