Lindorm:Hot and cold data separation FAQ

Lindorm archives cold data asynchronously via a compaction process. The archival interval is calculated based on your hot-cold data boundary:

For example, if your hot-cold data boundary is 3 days, a compaction runs every 1.5 days. If the boundary is 1 day, it runs once per day.

Yes. Run major_compact 'tableName' in an HBase Shell to force a compaction and immediately archive cold data to cold storage.

This can happen if the data you wrote has not yet been flushed to disk. Run a flush first to persist the in-memory data to disk, then run a compaction to move it to cold storage. To trigger compaction:

• In HBase Shell, run the major_compact command.

• In Lindorm-cli, use ALTER TABLE syntax.

Compaction backlog. If compactions are queuing up faster than they complete, hot-to-cold archiving slows down. Check the Compaction Queue Size(count) metric on the Instance Monitoring page under Table Metrics-Cluster Load. If the value is consistently above 0 and rising, you have a backlog. To resolve it, scale out or upgrade your instance to increase CPU resources. For details, see View monitoring information.

Outdated LindormTable version. Newer versions archive cold data faster. Check your current version in Release notes of LindormTable, then follow the steps in Minor version update to upgrade.

If neither resolves the issue, contact technical support.

It depends on whether the update changes the custom time column.

Example: A table has primary key columns p1 and p2, and non-primary key columns c1 and c2. A row has values p1=row1, p2=2023-01-28, c1="c1", and c2="c2". The hot-cold data boundary is 1 day and the current date is 2023-01-30, so this row is cold.

• Updating c1 or c2: row remains cold.

• Updating p2 to 2023-01-30: row becomes hot, then cold again on February 1, 2023.

To keep cold data cold: Avoid updating the custom time column on rows you want to keep archived.

No. Without a value in the custom time column, the system has no basis to classify the row, so it always stays in hot storage.

No. Any update to timestamp-based data refreshes its timestamp, making it hot again.

Cold data archiving is periodic, not instant. When you query with the HOT_ONLY setting or the _l_hot_only_ hint, some data that has crossed the hot-cold boundary may still physically reside in hot storage and be included in results.

Add a time range filter to restrict results to the intended window:

-- Set _l_ts_min_ (start time) and _l_ts_max_ (end time) using consistent units.
SELECT /*+ _l_hot_only_(true), _l_ts_min_(1000), _l_ts_max_(2001) */ * FROM test WHERE p1>1;

This typically happens right after a data migration or after enabling hot and cold data separation on an existing table. The system may not have run an archival pass yet, leaving a large volume of cold data in hot storage. This increases the total data scanned and causes timeouts.

Run a major compaction on the table to force archival. For the syntax, see ALTER TABLE.

The primary table and the index table each have independent archival processes that run on separate periodic schedules. At any given moment, the amount of cold data remaining in hot storage can differ between the two tables, causing inconsistent query results.

Add a time range filter to your query conditions to align results across both tables.

The system checks whether the oldest data file's age exceeds the configured archival period. If it does, a compaction runs immediately. Whether this happens right away depends on how old your oldest file is relative to the archival period you configured.

Yes. Specify a time range in your Scan operation to retrieve cold data.

• For timestamp-based separation, see Query data separated by timestamp.

• For custom time column separation, see Scan and query data separated by a custom time column.

Capacity-optimized and archive storage types have relatively low read IOPS, so Scan queries against cold data are slower than against hot storage.

When you change the storage policy of a table or column family from cold storage (COLD) back to default storage (DEFAULT), the new policy takes effect immediately, but data already archived to cold storage does not become hot right away. That data returns to hot storage only through a major compaction, and during the migration it may be inaccessible or respond more slowly. Perform the change during off-peak hours.

For the storage policy syntax, see ALTER TABLE.

Query only hot storage with HOT_ONLY, then compare the result with the same query without it. The difference is the volume of data held in cold storage.

• In Lindorm-cli:

-- Query hot data only.
SELECT /*+ _l_hot_only_(true) */ * FROM test WHERE p1>1;
-- To bound the hot-data window, set _l_ts_min_ (current time minus the hot-cold data boundary) and _l_ts_max_ (current time) using consistent units.
SELECT /*+ _l_hot_only_(true), _l_ts_min_(1000), _l_ts_max_(2001) */ * FROM test WHERE p1>1;

• In HBase Shell:

scan 'tableName', {HOT_ONLY => true}

Cold data archiving runs periodically, so some data past the hot-cold data boundary may still reside in hot storage. Treat the difference as an approximate estimate of your cold data volume, not an exact figure.

Work through this checklist before you disable cold storage:

1. Estimate the volume of cold data to be moved back using a HOT_ONLY query. See "How do I check the current hot-cold data distribution?" in this topic.

2. Confirm there is no compaction backlog. Check the Table Metrics-Cluster Load and Compaction Queue Size(count) metrics on the Instance Monitoring page of your LindormTable instance: a value consistently above 0 and rising indicates a backlog. For details, see View monitoring information.

3. Estimate the data migration time and resource consumption from your cold data volume and instance specification. A major compaction increases I/O load.

4. Confirm your workload can tolerate cold data being inaccessible or slower during the migration.