The data transformation feature helps reduce your time and labor costs to tidy data and boost your business. This topic describes how to configure rules to transform data at an optimal cost.

Typical configurations

Based on Processing principles and Performance guide, we recommend that you import log data into one or more Logstores and use the data transformation feature to dispatch transformed data to target Logstores. In addition, configure retention periods and indexes for data in different target Logstores.Typical configurations

Cost factors

According to the billing method, the following factors decide your costs for using Log Service:
  • The amount of data imported per day
  • Data retention period
  • Index creation

Based on the cost factors, the following describes how to optimize costs.

Optimize imported logs

If you import a daily amount of 100 GB of raw logs collected from an application into a source Logstore, create indexes for the data, and retain the data for 30 days, you are billed about USD 320 per month.

However, of all these raw logs, you may want the logs of certain pods, such as operations logs and error logs, which account for 20% of the total amount. And you want to retain this proportion of logs for 30 days and other logs for just 7 days. In this case, we recommend that you use the following method.
  • Create a source Logstore that retains logs for 3 days. Do not create indexes for the log data.
  • Create a target Logstore to store operations logs and error logs for 30 days and creates indexes for the log data.
  • Create another target Logstore to store other logs for 7 days and creates indexes for the log data.

This method helps reduce your cost by 25% to USD 240 per month.

You can use the data transformation feature to find the logs that are important to your business.Then, you can retain the important logs for 60 days and the remaining for 7 days, instead of retaining all logs for 30 days. If 20% of your logs are important, your costs are reduced by 12% while the retention period of important logs is doubled.

Optimize log entries in a log

If you import a daily amount of 100 GB of raw logs collected from an application into a source Logstore, create indexes for the data, and retain the data for 30 days, you are billed about USD 320 per month.

The following is a raw log entry with a size of 1021 bytes.
__source__:  1.2.3.4
__topic__:  ddos_access_log
body_bytes_sent:  3866
cc_action:  none
cc_blocks:  
cc_phase:  
content_type:  text/x-flv
host:  www.dbb.mock-domain.com
http_cookie:  i1=w1;x2=q2
http_referer:  http://www.cbc.mock-domain.com
http_user_agent:  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36
http_x_forwarded_for:  105.120.151.10
https:  true
isp_line:  BGP
matched_host:  www.cbd.mock-host.com
method:  GET
real_client_ip:  105.120.160.17
remote_addr:  105.120.160.0
remote_port:  48196
request_length:  2946
request_method:  GET
request_time_msec:  78920
request_uri:  /request/nvwlvvkhw
server_name:  www.bd.mock-host.com
status:  502
time:  2019-07-22T17:40:26+08:00
ua_browser:  mozilla
ua_browser_family:  
ua_browser_type:  
ua_browser_version:  9.0
ua_device_type:  
ua_os:  windows_7
ua_os_family:  
upstream_addr:  106.120.157.15:80
upstream_ip:  109.120.152.11
upstream_response_time:  0.858
upstream_status:  200
user_id:  st0s2b5
If you want only some fields in the log entry, you can use the data transformation feature to retain the target fields for 30 days and create indexes for these fields while retaining other fields for 3 days. To do this, apply the following method.
  • Create a source Logstore that retains logs for 3 days. Do not create indexes for the log data.
  • Create a target Logstore to store operations logs and error logs for 30 days and creates indexes for the log data.

If the size of a transformed log entry is 60% the size of raw log entry, this method helps reduce your cost by 30% to USD 224 per month.

The following is the 618-byte log entry transformed from the 1021-byte raw log entry.
__source__:  1.2.3.4
__topic__:  ddos_access_log
body_bytes_sent:  3866
content_type:  text/x-flv
host:  www.dbb.mock-domain.com
http_referer:  http://www.cbc.mock-domain.com
ua_browser:  mozilla
ua_browser_family:  
ua_browser_type:  
ua_browser_version:  9.0
ua_device_type:  
ua_os:  windows_7
http_x_forwarded_for:  105.120.151.10
matched_host:  www.cbd.mock-host.com
method:  GET
real_client_ip:  105.120.160.17
request_length:  2946
request_uri:  /request/nvwlvvkhw
status:  502
upstream_addr:  106.120.157.15:80
upstream_ip:  109.120.152.11
upstream_response_time:  0.858
upstream_status:  200
user_id:  st0s2b5