Object Storage Service:Cost-effectively store cold data with automatic storage tiering of the OSS-HDFS service

Usage notes

• If you use a JindoSDK version earlier than 6.8.0, you cannot create files in directories set to Infrequent Access, Archive, Cold Archive, or Deep Cold Archive storage. To work around this, create and close the file in a Standard storage directory, and then move it to the target directory using the rename operation.

  To create files directly in directories set to Infrequent Access, Archive, Cold Archive, or Deep Cold Archive storage, you must upgrade JindoSDK to version 6.8.0 or later.

• Transitioning data to Archive storage, Cold Archive storage, or Deep Cold Archive storage incurs significant system overhead and slow restore times. Choose these storage tiers carefully.

• The following storage tier transitions are supported:

  • Hot to cold:

    

  • Cold to hot:

    

Billing

• Data retrieval fees

  You are charged data retrieval fees when you read data from Infrequent Access, Archive, Cold Archive, or Deep Cold Archive storage. To avoid these fees, do not use these storage tiers for frequently accessed data. For more information, see Data processing fees.

• Object tag fees

  When you set a storage policy for data in the OSS-HDFS service, object tags are added to the data blocks. You are charged for these tags based on the OSS object tag billing rules. For more information, see Object tag fees.

• Fees for storage under the minimum duration

  The minimum storage duration is 30 days for Infrequent Access storage, 60 days for Archive storage, 180 days for Cold Archive storage, and 180 days for Deep Cold Archive storage. When you transition data using jindofs, ensure that the data meets the minimum storage duration of the new storage tier to avoid extra fees. The following table provides examples.

  Transition type	Example	Minimum storage duration	Storage cost	Fee avoidance
  Hot to cold	Standard storage (stored for 10 days) → Infrequent Access storage	The storage duration is not reset; days in the previous tier count toward the new tier's minimum duration.	10 days of Standard storage costs	Keep the data in Infrequent Access storage for at least 20 more days.
  	Infrequent Access storage (stored for 10 days) → Archive storage		10 days of Infrequent Access storage costs	Keep the data in Archive storage for at least 50 more days.
  	Standard storage (stored for 10 days) → Cold Archive storage	The storage duration is reset. The days spent in the previous tier do not count towards the minimum duration of the new tier.	10 days of Standard storage costs	Keep the data in Cold Archive storage for 180 days.
  	Standard storage (stored for 10 days) → Deep Cold Archive storage		10 days of Standard storage costs	Keep the data in Deep Cold Archive storage for 180 days.
  Cold to hot	Cold Archive storage (stored for 10 days) → Infrequent Access storage		10 days of Cold Archive storage costs	Keep the data in Cold Archive storage for 170 more days before you transition the data to Infrequent Access storage. After you transition the data to Infrequent Access storage, keep the data in Infrequent Access storage for 30 days.

Procedure

1. Connect to an ECS instance. For more information, see Connect to an ECS instance.

2. Download the JindoSDK.

3. Configure the access key pair and environment variables.

   1. Go to the bin directory of the installed JindoSDK package.

      The following command uses jindofs-sdk-x.x.x-linux as an example. Replace the package name with the actual name of your JindoSDK package.

      cd jindofs-sdk-x.x.x-linux/bin/

   2. In the bin directory, create a jindofs.cfg file with the following content:

      [client]
      # Specify the AccessKey ID and AccessKey secret used to access the OSS-HDFS service.
      fs.oss.accessKeyId = yourAccessKeyId    
      fs.oss.accessKeySecret = yourAccessKeySecret
      # The following endpoint is for the China (Hangzhou) region. Replace it with the endpoint for your region.
      fs.oss.endpoint = cn-hangzhou.oss-dls.aliyuncs.com                                     

   3. Set the environment variable.

      Note

      Replace with the absolute path of the jindofs.cfg configuration file.

      export JINDOSDK_CONF_DIR=<JINDOSDK_CONF_DIR>

4. Set a storage policy for the data written to the OSS-HDFS service.

   Scenario	Command	Result
   Set the storage policy for data written to the OSS-HDFS service to Infrequent Access storage.	./jindofs fs -setStoragePolicy -path oss://examplebucket/dir1 -policy CLOUD_IA	The service tags the data blocks for files in the dir1/ directory with the key transition-storage-class and value IA.
   Set the storage policy for data written to the OSS-HDFS service to Archive storage.	./jindofs fs -setStoragePolicy -path oss://examplebucket/dir2 -policy CLOUD_AR	The service tags the data blocks for files in the dir2/ directory with the key transition-storage-class and value Archive.
   Set the storage policy for data written to the OSS-HDFS service to Cold Archive storage.	./jindofs fs -setStoragePolicy -path oss://examplebucket/dir3 -policy CLOUD_COLD_AR	The service tags the data blocks for files in the dir3/ directory with the key transition-storage-class and value ColdArchive.
   Set the storage policy for data written to the OSS-HDFS service to Deep Cold Archive storage.	./jindofs fs -setStoragePolicy -path oss://examplebucket/dir4 -policy CLOUD_DEEP_COLD_AR	The service tags the data blocks for files in the dir4/ directory with the key transition-storage-class and value DeepColdArchive.

5. Enable automatic storage tiering.

   1. Log on to the OSS console.

   2. In the left-side navigation pane, click Buckets. On the Buckets page, click the name of the target bucket.

   3. In the left-side navigation pane, choose Data Lake > OSS-HDFS.

   4. On the OSS-HDFS tab, find automatic storage tiering and click Settings.

   5. In the Automatic Storage Tiering panel, go to the Basic Settings section and turn on the Status switch.

      Automatic storage tiering is implemented by a specific, pre-configured, and unmodifiable lifecycle rule. The storage tiers for automatic tiering include Infrequent Access, Archive, Cold Archive, and Deep Cold Archive.

      To prevent misconfigurations from causing automatic storage tiering to fail, OSS automatically creates a lifecycle rule that includes the following four policies:

      • Transitions data in the OSS-HDFS data directory .dlsdata/ that is tagged with the key transition-storage-class and value IA to Infrequent Access storage after one day.

      • Transitions data in the OSS-HDFS data directory .dlsdata/ that is tagged with the key transition-storage-class and value Archive to Archive storage after one day.

      • Transitions data in the OSS-HDFS data directory .dlsdata/ that is tagged with the key transition-storage-class and value ColdArchive to Cold Archive storage after one day.

      • Transitions data in the OSS-HDFS data directory .dlsdata/ that is tagged with the key transition-storage-class and value DeepColdArchive to Deep Cold Archive storage after one day.

      Important

      After enabling automatic storage tiering, a lifecycle rule is automatically created. This rule contains policies to transition objects to Infrequent Access, Archive, Cold Archive, and Deep Cold Archive storage. Do not edit this rule. Otherwise, data in the OSS-HDFS service or the service itself may become unavailable.

   6. Click OK.

      • The OSS-HDFS service applies the corresponding transition policy from the lifecycle rule based on the storage policy that you set.

      • OSS loads a new lifecycle rule within 24 hours. Once loaded, the rule runs daily at 08:00 (UTC+8). The transition completes in as little as 48 hours, although the exact execution time depends on the number of files.

Related commands

./jindofs fs -setStoragePolicy -path <path> -policy <policy>

./jindofs fs -getStoragePolicy -path <path>

./jindofs fs -unsetStoragePolicy -path <path>

./jindofs fs -checkStoragePolicy -path <path>

./jindofs fs -setStoragePolicy -path <path> -policy <policy> -restoreDays <restoreDays>

FAQ

• What do I do if I accidentally delete the lifecycle rule for automatic storage tiering of the OSS-HDFS service?

• Can I transition files from Archive or Cold Archive storage to Standard or Infrequent Access storage in the OSS-HDFS service?

Related topics

• Storage class overview

• Storage fees