Lindorm:Cold data archiving

Overview

For scenarios in which data is continuously written into a time series database, the frequency at which historical time series data is queried gradually decreases over time. These historical data is referred to as cold data. Data that is frequently accessed is referred to as hot data. The growing size of cold data increases the storage costs. Enterprises that use LindormTSDB needs to reduce the storage costs for cold data and improve query performance for hot data.

LindormTSDB provides tiered storage that stores hot data, warm data, and cold data to different media.

• Hot data

  The time series data that is most recently written. In most cases, the timestamps of hot data are very close to the current timestamp. Hot data is frequently queried in time series applications.

• Warm data

  The time series data that has been stored for a period of time. Warm data is not queried as frequently as hot data.

• Cold data

  The time series data that has been stored for a long period of time. In most cases, cold data can be archived. In time series applications, large amounts of cold data is accumulated is rarely queried.

LindormTSDB stores hot data, warm data, and cold data in different storage media based on the size and query frequencies of different types of data. The following figure shows how LindormTSDB implements tiered data storage.

Cold data accounts for the largest share of data storage. LindormTSDB automatically archives cold data to a cold storage medium based on the hot and cold boundary that you specified based on timestamps. When you need to query cold data, you do not need to modify the configurations of your application. You can use normal queries to query the cold data. Lindorm uses Capacity storage to store cold data. For more information about the read IOPS of Capacity storage, see Limits on the read IOPS of Capacity storage.

Enable cold storage

After LindormTSDB is activated for a Lindorm instance, the instance provides the tiered storage capabilities for hot data and warm data by default. If you want to enable the Lindorm instance to automatically archive cold data to Capacity storage, enable cold storage when you create the instance in advance.

To enable cold storage, perform the following operations:

• When you purchase a Lindorm instance, configure the Purchase Cold Storage and Cold Storage Capacity parameters on the buy page based on your business requirements.

  

• If you do not enable cold storage for a Lindorm instance when you create the instance, you can enable the feature in the Lindorm console after the instance is created. The following figure shows the console page on which you can enable cold storage after the instance is created.

  

After cold storage is enabled, you can configure a hot and cold data boundary to specify the time when LindormTSDB archives data to Capacity storage.

Configure a hot and cold data boundary

Current timestamp - Timestamp of a data record >= Hot and cold data boundary (days)

Query cold data

You can use standard SQL statements to query cold data without specifying additional conditions. However, LindormTSDB takes more time to return query results when you query cold data archived in Capacity storage.