All Products
Search

This Product

    Object Storage Service:Automatic storage tiering of OSS-HDFS

    Document Center

    Object Storage Service:Automatic storage tiering of OSS-HDFS

    Last Updated:Sep 25, 2023

    Some data in OSS-HDFS is not frequently accessed but needs to be retained to meet compliance or archiving requirements. To meet these requirements, OSS-HDFS provides the automatic storage tiering feature. This feature automatically moves frequently accessed data to the Standard storage class and rarely accessed data to the Infrequent Access (IA), Archive, or Cold Archive storage class to help reduce storage costs.

    Prerequisites

    • Data is written to OSS-HDFS.

    • The bucket for which you want to enable the automatic storage tiering feature is located in one of the following regions: China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China (Zhangjiakou), China (Hong Kong), Singapore, Germany (Frankfurt), US (Silicon Valley), US (Virginia), and Indonesia (Jakarta).

    • A ticket is submitted to use the automatic storage tiering feature.

    • JindoSDK 4.4.0 or later is installed and configured. For more information, see Connect non-EMR clusters to OSS-HDFS.

    Usage notes

    • You are charged data retrieval fees when you read Archive data in OSS-HDFS. We recommend that you do not store frequently accessed data as IA, Archive, or Cold Archive objects. For more information about the data retrieval fees, see Data processing fees.

    • When you configure a storage policy for data in OSS-HDFS, you must add tags to data blocks. You are charged for the tags based on the Object Storage Service (OSS) object tagging billing rules. For more information, see Object tagging fees.

    • You cannot directly create an object in an IA, Archive, or Cold Archive directory. If you need to create an object in an IA, Archive, or Cold Archive directory, you can create an object and close it in a Standard directory and move the object to the IA, Archive, or Cold Archive directory by using the rename operation.

    • When you convert the storage class of objects to Archive or Cold Archive, additional system overheads are generated and data restoration is slow. Proceed with caution.

    • You cannot convert Archive objects to Cold Archive objects and Cold Archive objects to Archive objects.

    Procedure

    1. Specify a storage policy for the data that is written to OSS-HDFS. The following table describes the storage policies.

      Scenario

      Command

      Result

      IA

      jindo fs -setStoragePolicy -path oss://examplebucket/dir1 -policy CLOUD_IA

      Objects in the dir1/ directory contain a tag whose key is transition-storage-class and whose value is IA.

      Archive

      jindo fs -setStoragePolicy -path oss://examplebucket/dir2 -policy CLOUD_AR

      Objects in the dir2/ directory contain a tag whose key is transition-storage-class and whose value is Archive.

      Cold Archive

      jindo fs -setStoragePolicy -path oss://examplebucket/dir3 -policy CLOUD_COLD_AR

      Objects in the dir3/ directory contain a tag whose key is transition-storage-class and whose value is ColdArchive.

    2. Enable the automatic storage tiering feature.

      1. Log on to the OSS console.

      2. In the left-side navigation pane, click Buckets. On the Buckets page, click the name of the bucket for which you want to enable the automatic storage tiering feature.

      3. In the left-side navigation tree, choose Data Lake > OSS-HDFS.

      4. On the OSS-HDFS tab, click Configure.

      5. In the Basic Settings section of the Automatic Storage Tiering panel, turn on Status.

        1

        To prevent the automatic storage tiering feature from not running due to incorrect configurations, OSS automatically creates a lifecycle rule to convert the storage class of data in OSS-HDFS that contains a specific tag:

        • The lifecycle rule specifies that the storage class of the data that contains a tag whose key is transition-storage-class and whose value is IA in the .dlsdata/ directory is converted to IA one day after the data is last modified.

        • The lifecycle rule specifies that the storage class of the data that contains a tag whose key is transition-storage-class and whose value is Archive in the .dlsdata/ directory is converted to Archive one day after the data is last modified.

        • The lifecycle rule specifies that the storage class of the data that contains a tag whose key is transition-storage-class and whose value is ColdArchive in the .dlsdata/ directory is converted to Cold Archive one day after the data is last modified.

      6. Click OK.

        • OSS-HDFS applies a storage class conversion policy based on the storage policy that you specify in Step 1.

        • After a lifecycle rule is created, OSS loads the rule within 24 hours. After the rule is loaded, OSS starts to execute the rule at 08:00 (UTC+8) every day. The specific execution time varies based on the number of objects. The objects are converted to the specified storage class within at least 48 hours.

    Related commands

    Syntax

    Description

    jindo fs -setStoragePolicy -path <path> -policy <policy>

    Specifies a storage policy for data in a path.

    • -path: specifies the path to the object or directory.

    • -policy: specifies a storage policy. Valid values:

      • CLOUD_STD: the Standard storage class.

      • CLOUD_IA: the IA storage class.

      • CLOUD_AR: the Archive storage class.

      • CLOUD_COLD_AR: the Cold Archive storage class.

    If you do not specify the storage class for an object or a subdirectory, the storage class of the object or the subdirectory inherits the storage class of the directory. For example, if the storage class of the oss://examplebucket/dir directory is CLOUD_STD and you do not specify a storage class for the oss://examplebucket/dir/subdir subdirectory, the storage class of the oss://examplebucket/dir/subdir subdirectory is also CLOUD_STD. 

    jindo fs -getStoragePolicy -path <path>

    Obtains the storage policy of data in a specific path. 

    jindo fs -unsetStoragePolicy -path <path>

    Deletes the storage policy of data in a specific path. 

    jindo fs -checkStoragePolicy -path <path>

    Obtains the status of storage class conversion for data in a specific path based on the storage policy. Valid values:

    • Pending: The conversion task is to be submitted.

    • Submitted: The conversion task is submitted.

    • Finalized: The conversion task is complete.

    FAQ

    What do I do if I accidentally delete a lifecycle rule after the automatic storage tiering feature is enabled?

    After the automatic storage tiering feature is enabled, OSS automatically creates a lifecycle rule to convert the storage class of data to IA, Archive, or Cold Archive. If you delete the lifecycle rule, the automatic storage tiering feature may not work as expected.

    Solution:

    1. Disable the automatic storage tiering feature.

      1. Click Configure.

        lifecycle

      2. In the Basic Settings section of the Automatic Storage Tiering panel, turn off Status and click OK.

        setting

      3. In the message that appears, click OK.

    2. Re-create a lifecycle rule.

      1. On the OSS-HDFS tab, click Configure.

      2. In the Basic Settings section of the Automatic Storage Tiering panel, turn on Status.

      3. Click OK.

        OSS re-creates a lifecycle rule to convert the storage class of data to IA, Archive, or Cold Archive.

    Can I convert the storage class of Archive or Cold Archive objects to Standard or IA?

    Yes. For example, if you want to convert an Archive object to a Standard object, set -policy to CLOUD_STD.