Lindorm:Cold data archiving

Overview

For scenarios in which data is continuously written into a time series database, the query frequency of the time series data that was previously written is decreased over time. Data that is infrequently accessed is referred to as cold data, and data that is frequently accessed is referred to as hot data. The growing volume of cold data increases the storage costs. Reduced storage costs for cold data and improved query performance for hot data are important requirements of enterprises for time series engines.

LindormTSDB provides tiered storage and stores hot data, warm data, and cold data to different media. The following concepts are introduced:

• Hot data

  The time series data that is most recently written. In most cases, the timestamps of hot data are very close to the current timestamp. Hot data is frequently queried in time series applications.

• Warm data

  The time series data that is stored for a period of time. Warm data is not queried as frequently as hot data.

• Cold data

  The time series data that is stored for a long period of time. In most cases, cold data is regarded as archived data. In time series applications, the volume of cold data is very large, but only a small volume of cold data is used for queries.

LindormTSDB stores hot data, warm data, and cold data in different storage media based on the data volumes and query frequencies of different types of data. The following figure shows how LindormTSDB implements tiered data storage.

Cold data accounts for the largest share of data storage. LindormTSDB automatically archives data to a cold storage medium based on the hot/cold data boundary that you specified based on timestamps. When you need to query cold data, you do not need to modify the configurations of your application. You can run normal queries to query the cold data.

Prerequisites

After a Lindorm instance that runs a time series engine is created, the instance provides the tiered storage capabilities for hot data and warm data by default. If you want to enable the Lindorm instance to automatically archive cold data to the cold storage medium, enable cold storage when you create the instance in advance.

To enable cold storage, perform the following operations:

• When you purchase a Lindorm instance, configure the Purchase Cold Storage and Cold Storage Capacity parameters on the buy page based on your business requirements.

• If you do not enable cold storage for a Lindorm instance when you create the instance, you can enable the feature in the Lindorm console after the instance is created. The following figure shows the console page on which you can enable cold storage after the instance is created.

  

After cold storage is enabled, you can configure a hot/cold data boundary to specify the time when LindormTSDB archives data to the cold storage medium.

Configure a hot/cold data boundary

Query cold data

When you query data, you can run normal SQL queries without the need to identify cold data and hot data.

Take note that the query latency increases if the data that you want to query is stored in cold storage media.