All Products
Search
Document Center

Lindorm:Cold data archiving

Last Updated:Jul 03, 2024

Lindorm uses Capacity storage to store cold data. This topic describes how to use the cold data archiving feature provided by LindormTSDB.

Overview

For scenarios in which data is continuously written into a time series database, the frequency at which historical time series data is queried gradually decreases over time. These historical data is referred to as cold data. Data that is frequently accessed is referred to as hot data. The growing size of cold data increases the storage costs. Enterprises that use LindormTSDB needs to reduce the storage costs for cold data and improve query performance for hot data.

LindormTSDB provides tiered storage that stores hot data, warm data, and cold data to different media.

  • Hot data

    The time series data that is most recently written. In most cases, the timestamps of hot data are very close to the current timestamp. Hot data is frequently queried in time series applications.

  • Warm data

    The time series data that has been stored for a period of time. Warm data is not queried as frequently as hot data.

  • Cold data

    The time series data that has been stored for a long period of time. In most cases, cold data can be archived. In time series applications, large amounts of cold data is accumulated is rarely queried.

LindormTSDB stores hot data, warm data, and cold data in different storage media based on the size and query frequencies of different types of data. The following figure shows how LindormTSDB implements tiered data storage.

image

Cold data accounts for the largest share of data storage. LindormTSDB automatically archives cold data to a cold storage medium based on the hot and cold boundary that you specified based on timestamps. When you need to query cold data, you do not need to modify the configurations of your application. You can use normal queries to query the cold data. Lindorm uses Capacity storage to store cold data. For more information about the read IOPS of Capacity storage, see Limits on the read IOPS of Capacity storage.

Enable cold storage

After LindormTSDB is activated for a Lindorm instance, the instance provides the tiered storage capabilities for hot data and warm data by default. If you want to enable the Lindorm instance to automatically archive cold data to Capacity storage, enable cold storage when you create the instance in advance.

To enable cold storage, perform the following operations:

  • When you purchase a Lindorm instance, configure the Purchase Cold Storage and Cold Storage Capacity parameters on the buy page based on your business requirements.

    image.png

  • If you do not enable cold storage for a Lindorm instance when you create the instance, you can enable the feature in the Lindorm console after the instance is created. The following figure shows the console page on which you can enable cold storage after the instance is created.

    管理控制台的冷存开通

After cold storage is enabled, you can configure a hot and cold data boundary to specify the time when LindormTSDB archives data to Capacity storage.

Configure a hot and cold data boundary

Definition of a hot/cold data boundary

A hot and cold data boundary is used to distinguish between hot data and cold data. You can configure a hot and cold data boundary based on the data size and the time range of queries. The hot and cold data boundary specifies the retention period of data after which LindormTSDB archives the data to Capacity storage as cold data.

In LindormTSDB, the hot and cold data boundary is measured in days. If the timestamp of a time series data record meets a predetermined condition, LindormTSDB determines that the data record is a cold data record.

LindormTSDB determines whether a data record is cold based on the following formula:

Current timestamp - Timestamp of a data record >= Hot and cold data boundary (days)

Procedure

You can use one of the following methods to configure a hot and cold data boundary:

  • Specify the hot and cold data boundary in the CREATE DATABASE or ALTER DATABASE statement.

    Note

    For more information, see CREATE DATABASE or ALTER DATABASE.

  • Specify the hot and cold data boundary in the Lindorm console.

    Note

    For more information, see Manage databases.

Query cold data

You can use standard SQL statements to query cold data without specifying additional conditions. However, LindormTSDB takes more time to return query results when you query cold data archived in Capacity storage.