Cost optimization guide

Typical configuration

Based on the transformation principles and performance guide, simplify your collection plan: ingest data into one or more Logstores, then use data transformation to distribute it. Configure the storage period and index for each destination Logstore as needed. Typical configuration

Cost factors

The and billing method show that cost depends on three factors:

Daily data ingestion volume.
Data retention period.
Index configuration.

The following two examples show how to optimize costs by adjusting storage structure and content.

Optimize storage structure

Baseline: an application writes 100 GB/day, stored for 30 days with a full-text index. Monthly cost: approximately USD 562.

If only 20% of logs (such as operation and error logs) need 30-day retention while the rest only need 7 days, use the following transformation plan:

Create a source Logstore to store data for 3 days without an index.
Create destination Logstore 1 to store operation and error logs for 30 days with an index.
Create destination Logstore 2 to store general logs for 7 days with an index.

Monthly cost drops to approximately USD 421, a saving of approximately 25%.

For a 60-day baseline, storing 20% of important logs for 60 days and the rest for 7 days saves 12% while doubling the retention of critical logs.

Optimize storage content

Using the same baseline (100 GB/day, 30-day retention, full-text index), the monthly cost is approximately USD 562.

Example raw log (1021 bytes):

__source__:  192.0.2.0
__topic__:  ddos_access_log
body_bytes_sent:  3866
cc_action:  none
cc_blocks:  
cc_phase:  
content_type:  text/x-flv
host:  www.example.com
http_cookie:  i1=w1;x2=q2
http_referer:  http://www.example.com
http_user_agent:  Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.0.2.1 Safari/537.36
http_x_forwarded_for:  192.0.2.2
https:  true
isp_line:  BGP
matched_host:  www.example.com
method:  GET
real_client_ip:  192.0.2.3
remote_addr:  192.0.2.4
remote_port:  48196
request_length:  2946
request_method:  GET
request_time_msec:  78920
request_uri:  /request/nvwlvvkhw
server_name:  www.example.com
status:  502
time:  2019-07-22T17:40:26+08:00
ua_browser:  mozilla
ua_browser_family:  
ua_browser_type:  
ua_browser_version:  9.0
ua_device_type:  
ua_os:  windows_7
ua_os_family:  
upstream_addr:  192.0.2.4:80
upstream_ip:  192.0.2.5
upstream_response_time:  0.858
upstream_status:  200
user_id:  st0s2b5

To keep only important fields indexed for 30 days and discard the rest after 3 days, use the following transformation plan:

Create a source Logstore with a 3-day data retention period and no index.
Create a destination Logstore with a 30-day data retention period and an index for the required fields.

If the processed log is about 60% of its original size, monthly cost drops to approximately USD 393, a saving of approximately 30%.

After transformation, the log shrinks from 1021 bytes to 618 bytes:

__source__:  192.0.2.0
__topic__:  ddos_access_log
body_bytes_sent:  3866
content_type:  text/x-flv
host:  www.example.com
http_referer:  http://www.example.com
ua_browser:  mozilla
ua_browser_family:  
ua_browser_type:  
ua_browser_version:  9.0
ua_device_type:  
ua_os:  windows_7
http_x_forwarded_for:  192.0.2.2
matched_host:  www.example.com
method:  GET
real_client_ip:  192.0.2.3
request_length:  2946
request_uri:  /request/nvwlvvkhw
status:  502
upstream_addr:  192.0.2.4:80
upstream_ip:  192.0.2.5
upstream_response_time:  0.858
upstream_status:  200
user_id:  st0s2b5