The data transformation feature helps reduce your time and labor costs to tidy data and boost your business. This topic describes how to configure rules to transform data at an optimal cost.
Typical configurations
Cost factors
- The amount of data imported per day
- The data retention period
- The number of indexes that you create
The following examples describe how to optimize costs.
Optimize imported logs
Assume that you collect logs from an application and import 100 GB of log data into a source Logstore per day. You also create full-text indexes for the log data and set a data retention period of 30 days. In this case, you are billed about USD 562 per month.
- Create a source Logstore that retains logs for three days. Do not create indexes for the log data.
- Create a destination Logstore to store operations logs and error logs for 30 days and create indexes for the log data.
- Create another destination Logstore to store other logs for seven days and create indexes for the log data.
In this case, you are billed about USD 421 per month. This method can reduce your costs by 25%.
You can use the data transformation feature to retain important logs for 60 days and the other logs for seven days. If you want to retain 20% of your logs, your costs are reduced by 12% and the retention period of those logs is doubled.
Optimize log entries in a log
Assume that you collect logs from an application and import 100 GB of log data into a source Logstore per day. You also create full-text indexes for the log data and set a data retention period of 30 days. In this case, you are billed about USD 562 per month.
__source__: 192.0.2.0
__topic__: ddos_access_log
body_bytes_sent: 3866
cc_action: none
cc_blocks:
cc_phase:
content_type: text/x-flv
host: www.example.com
http_cookie: i1=w1;x2=q2
http_referer: http://www.example.com
http_user_agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.0.2.1 Safari/537.36
http_x_forwarded_for: 192.0.2.2
https: true
isp_line: BGP
matched_host: www.example.com
method: GET
real_client_ip: 192.0.2.3
remote_addr: 192.0.2.4
remote_port: 48196
request_length: 2946
request_method: GET
request_time_msec: 78920
request_uri: /request/nvwlvvkhw
server_name: www.example.com
status: 502
time: 2019-07-22T17:40:26+08:00
ua_browser: mozilla
ua_browser_family:
ua_browser_type:
ua_browser_version: 9.0
ua_device_type:
ua_os: windows_7
ua_os_family:
upstream_addr: 192.0.2.4:80
upstream_ip: 192.0.2.5
upstream_response_time: 0.858
upstream_status: 200
user_id: st0s2b5
- Create a source Logstore that retains logs for three days. Do not create indexes for the log data.
- Create a destination Logstore to store operations logs and error logs for 30 days and create indexes for the log data.
If the size of a transformed log entry is 60% the size of the raw log entry, you are billed about USD 393 per month. This method can reduce your costs by 30%.
__source__: 192.0.2.0
__topic__: ddos_access_log
body_bytes_sent: 3866
content_type: text/x-flv
host: www.example.com
http_referer: http://www.example.com
ua_browser: mozilla
ua_browser_family:
ua_browser_type:
ua_browser_version: 9.0
ua_device_type:
ua_os: windows_7
http_x_forwarded_for: 192.0.2.2
matched_host: www.example.com
method: GET
real_client_ip: 192.0.2.3
request_length: 2946
request_uri: /request/nvwlvvkhw
status: 502
upstream_addr: 192.0.2.4:80
upstream_ip: 192.0.2.5
upstream_response_time: 0.858
upstream_status: 200
user_id: st0s2b5