ApsaraDB for HBase:Cold storage

Background information

When you purchase an ApsaraDB for HBase cluster, you can select the cold storage medium as an additional storage space, and execute table creation statements to store cold data on the medium. In addition, ApsaraDB for HBase Performance-enhanced Edition allows you to separate cold data from hot data in the same table. The system can automatically store hot data in hot storage with fast read/write speed and store infrequently accessed data in cold storage to reduce costs.

Usage notes

• The read Input/Output Operations Per Second (IOPS) of cold storage is low (up to 25 times/s per node). Therefore, cold storage is applicable only to infrequent queries.

• The write throughput of cold storage equals to the throughput of hot storage for which you use ultra disks.

• Cold storage is not suitable for processing a large number of concurrent read requests. An error may occur if cold storage is used to process a large number of concurrent read requests.

• If you purchase large cold storage capacity, you can adjust the read IOPS based on your business requirements. For more information, submit a ticket for technical support.

• We recommend that you store a maximum of 30 TB cold data in each core node. To increase the storage capacity of each core node, you can submit a ticket for optimization suggestions.

Prerequisites

Cold storage is supported only on ApsaraDB for HBase Performance-enhanced Edition V2.1.8 and later. If your ApsaraDB for HBase Performance-enhanced Edition cluster is of a version earlier than V2.1.8, the cluster is automatically upgraded to the latest version when you activate cold storage for your cluster. The version of the client dependency AliHBase-Connector must be later than V1.0.7 or V2.0.7. The version of HBase Shell must be later than alihbase-2.0.7-bin.tar.gz.

Scenarios

Cold storage is applicable to various cold data scenarios such as data archiving and infrequently accessed data consumption.

Enable cold storage

Method 1: When you create an ApsaraDB for HBase Performance-enhanced Edition cluster, you can choose whether to purchase cold storage and the capacity of cold storage on the cluster buy page. For more information, see Purchase a cluster.

Use cold storage

ApsaraDB for HBase Performance-enhanced Edition allows you to configure storage properties based on column families. You can set the storage type to cold storage for a column family or all column families of a table. Then all data of the column family or all column families in the table is stored in cold storage and does not occupy the Hadoop Distributed File System (HDFS) space of the cluster. You can configure the property of the column family when you create a table or modify the property after you create a table.

You can use Java API or HBase Shell to create a table and modify the table properties. If you use the Java API, you must install the SDK for Java and configure the parameters first. For more information, see Use the ApsaraDB for HBase Java API to access an ApsaraDB for HBase Performance-enhanced Edition instance. If you use HBase Shell, you must install HBase Shell and configure the parameters first. For more information, see Use HBaseue Shell to access an ApsaraDB for HBase Performance-enhanced Edition instance.

Create a table that uses cold storage

hbase(main):001:0> create 'coldTable', {NAME => 'f', STORAGE_POLICY => 'COLD'}

 Admin admin = connection.getAdmin();
 HTableDescriptor descriptor = new HTableDescriptor(TableName.valueOf("coldTable"));
 HColumnDescriptor cf = new HColumnDescriptor("f");
 cf.setValue("STORAGE_POLICY", AliHBaseConstants.STORAGETYPE_COLD);
 descriptor.addFamily(cf);
 admin.createTable(descriptor);

Modify the table property to use cold storage

If you create a table, you can modify the property of a column family in the table to use cold storage. If the column family contains data, the data is archived to cold storage only after a major compaction operation.

hbase(main):011:0> alter 'coldTable', {NAME=>'f', STORAGE_POLICY => 'COLD'}

Admin admin = connection.getAdmin();
TableName tableName = TableName.valueOf("coldTable");
HTableDescriptor descriptor = admin.getTableDescriptor(tableName);
HColumnDescriptor cf = descriptor.getFamily("f".getBytes());
// Set the storage type of the table to cold storage.
cf.setValue("STORAGE_POLICY", AliHBaseConstants.STORAGETYPE_COLD);
admin.modifyTable(tableName, descriptor);

Modify the table property to use hot storage

If the storage type of a column family in a table is cold storage, you can change the type back to hot storage by changing the table property. If the column family in the table contains data, the data is archived to hot storage after a major compaction operation.

hbase(main):014:0> alter 'coldTable', {NAME=>'f', STORAGE_POLICY => 'DEFAULT'}

// Establish a connection to the cluster.
Admin admin = connection.getAdmin();
TableName tableName = TableName.valueOf("coldTable");
HTableDescriptor descriptor = admin.getTableDescriptor(tableName);
HColumnDescriptor cf = descriptor.getFamily("f".getBytes());
// Set the storage type of the table to hot storage, which is the default value.
cf.setValue("STORAGE_POLICY", AliHBaseConstants.STORAGETYPE_DEFAULT);
admin.modifyTable(tableName, descriptor);

View the cold storage status

You can view the cold storage status and expand the capacity of the cold storage by clicking Cold Storage Scaling on the Cold Storage page in the ApsaraDB for HBase console.

You can view the sizes of cold and hot data in a table on the User tables tab of the cluster management system.