Object Storage Service:Use tiered storage to store cold data at a low cost with OSS-HDFS

Usage notes

• If your JindoSDK version is earlier than 6.8.0, you cannot create files in a directory that is set to the Infrequent Access, Archive, Cold Archive, or Deep Cold Archive storage class. To place files in a directory of one of these four storage classes, create and close the file in a Standard storage class directory first, and then use the rename operation to move it to the directory.

  To create files directly in a directory of the Infrequent Access, Archive, Cold Archive, or Deep Cold Archive storage class, upgrade JindoSDK to 6.8.0 or later.

• Converting data to the Archive, Cold Archive, or Deep Cold Archive storage class incurs high system overhead, and retrieval is slow. Choose these storage classes with caution.

• Tiered storage supports the following storage class conversions:

  • Hot-to-cold conversion

    

  • Cold-to-hot conversion

    

Billing

• Data retrieval fees

  Reading OSS-HDFS data of the Infrequent Access, Archive, Cold Archive, or Deep Cold Archive storage class incurs data retrieval fees based on the retrieved capacity. Therefore, avoid using these four storage classes for data that must be accessed frequently. For more information, see Data processing fees.

• Tagging fees

  Configuring a storage policy for OSS-HDFS data involves adding tags to data blocks. Data block tagging fees follow the OSS object tagging billing rules. For more information, see Object tagging fees.

• Storage fees for data stored for less than the minimum duration

  The minimum storage duration is 30 days for the Infrequent Access storage class, 60 days for the Archive storage class, 180 days for the Cold Archive storage class, and 180 days for the Deep Cold Archive storage class. When you convert the storage class of data by using JindoFS, make sure that the minimum storage duration for each storage class is met to avoid extra fees. The following examples describe the details:

  Conversion method	Example	How the minimum storage duration is calculated	Capacity fees already incurred	How to avoid fees for storage shorter than the minimum duration
  Hot-to-cold	Standard (stored for 10 days) --> converted to Infrequent Access	The storage duration is not recalculated after conversion. The number of days stored before the conversion counts toward the minimum storage duration of the new storage class.	Capacity fees for 10 days of Standard storage	Continue to store the data as the Infrequent Access storage class for 20 more days
  	Infrequent Access (stored for 10 days) --> converted to Archive		Capacity fees for 10 days of Infrequent Access storage	Continue to store the data as the Archive storage class for 50 more days
  	Standard (stored for 10 days) --> converted to Cold Archive	The storage duration is recalculated after conversion. The number of days stored before the conversion does not count toward the minimum storage duration of the new storage class.	Capacity fees for 10 days of Standard storage	Continue to store the data as the Cold Archive storage class for 180 more days
  	Standard (stored for 10 days) --> converted to Deep Cold Archive		Capacity fees for 10 days of Standard storage	Continue to store the data as the Deep Cold Archive storage class for 180 more days
  Cold-to-hot	Cold Archive (stored for 10 days) --> converted to Infrequent Access		Capacity fees for 10 days of Cold Archive storage	Continue to store the data as the Cold Archive storage class for 170 more days, and then convert it to Infrequent Access After the data is converted to the Infrequent Access storage class, continue to store it for 30 more days

Procedure

1. Connect to the ECS instance. For more information, see Connect to an instance.

2. Download the JindoFS SDK.

3. Configure the access key and environment variables.

   1. Go to the bin directory in the installed JindoFS JAR package.

      The following example uses jindofs-sdk-x.x.x-linux. If you use another version of JindoSDK, replace the JAR package name accordingly.

      cd jindofs-sdk-x.x.x-linux/bin/

   2. Create a configuration file named jindofs.cfg in the bin directory, and then add the following configuration items to it.

      [client]
      <!-- Configure the AccessKey ID and AccessKey secret used to access OSS-HDFS. -->
      fs.oss.accessKeyId = yourAccessKeyId    
      fs.oss.accessKeySecret = yourAccessKeySecret
      <!-- The following example uses the China (Hangzhou) region. Replace it with the actual region. -->
      fs.oss.endpoint = cn-hangzhou.oss-dls.aliyuncs.com                                     

   3. Set the environment variable.

      Note

      For <JINDOSDK_CONF_DIR>, enter the absolute path of the directory where the jindofs.cfg configuration file is located.

      export JINDOSDK_CONF_DIR=<JINDOSDK_CONF_DIR>

4. Configure a storage policy for the data written to OSS-HDFS.

   Scenario	Command	Result
   Set the storage policy of the data written to OSS-HDFS to Infrequent Access	./jindofs fs -setStoragePolicy -path oss://examplebucket/dir1 -policy CLOUD_IA	The data blocks of the files in the dir1/ directory are tagged with the key transition-storage-class and the value IA.
   Set the storage policy of the data written to OSS-HDFS to Archive	./jindofs fs -setStoragePolicy -path oss://examplebucket/dir2 -policy CLOUD_AR	The data blocks of the files in the dir2/ directory are tagged with the key transition-storage-class and the value Archive.
   Set the storage policy of the data written to OSS-HDFS to Cold Archive	./jindofs fs -setStoragePolicy -path oss://examplebucket/dir3 -policy CLOUD_COLD_AR	The data blocks of the files in the dir3/ directory are tagged with the key transition-storage-class and the value ColdArchive.
   Set the storage policy of the data written to OSS-HDFS to Deep Cold Archive	./jindofs fs -setStoragePolicy -path oss://examplebucket/dir4 -policy CLOUD_DEEP_COLD_AR	The data blocks of the files in the dir4/ directory are tagged with the key transition-storage-class and the value DeepColdArchive.

5. Enable tiered storage.

   1. Log on to the OSS console.

   2. In the left-side navigation pane, click Buckets, and then click the target bucket.

   3. In the left-side navigation pane, choose Data Lake > OSS-HDFS.

   4. On the OSS-HDFS tab, click Settings for tiered storage.

   5. In the Basic Settings section of the Automatic Storage Tiering panel, turn on the status switch.

      A message appears at the top of the page: Tiered storage is implemented by creating a specific lifecycle rule. This rule is preconfigured and cannot be modified. Tiered storage classes include Infrequent Access, Archive, Cold Archive, and Deep Cold Archive.

      To prevent tiered storage from malfunctioning due to configuration errors, OSS automatically creates a lifecycle rule that includes all of the following four policies:

      • Data in the OSS-HDFS data storage directory .dlsdata/ that is tagged with the key transition-storage-class and the value IA is converted to the Infrequent Access storage class after 1 day.

      • Data in the OSS-HDFS data storage directory .dlsdata/ that is tagged with the key transition-storage-class and the value Archive is converted to the Archive storage class after 1 day.

      • Data in the OSS-HDFS data storage directory .dlsdata/ that is tagged with the key transition-storage-class and the value ColdArchive is converted to the Cold Archive storage class after 1 day.

      • Data in the OSS-HDFS data storage directory .dlsdata/ that is tagged with the key transition-storage-class and the value DeepColdArchive is converted to the Deep Cold Archive storage class after 1 day.

      Important

      After tiered storage is enabled, a lifecycle rule is automatically created. This rule includes four policies that convert data to the Infrequent Access, Archive, Cold Archive, and Deep Cold Archive storage classes. Do not edit this rule. Otherwise, OSS-HDFS data and services may become abnormal.

   6. Click OK.

      • OSS-HDFS applies the corresponding conversion policy in the lifecycle rule based on the configured storage policy.

      • OSS loads the rule within 24 hours after the lifecycle rule is created. After the rule is loaded, OSS runs it every day starting at 08:00 (UTC+8). The exact run time depends on the number of files, and data is converted to the specified storage class within 48 hours at the earliest.

Related commands

./jindofs fs -setStoragePolicy -path <path> -policy <policy>

./jindofs fs -getStoragePolicy -path <path>

./jindofs fs -unsetStoragePolicy -path <path>

./jindofs fs -checkStoragePolicy -path <path>

./jindofs fs -setStoragePolicy -path <path> -policy <policy> -restoreDays <restoreDays>

FAQ

• What should I do if I accidentally delete the lifecycle rule for OSS-HDFS tiered storage?

• Does OSS-HDFS support converting Archive or Cold Archive objects to the Standard or Infrequent Access storage class?

References

• Overview of storage classes

• Storage fees