All Products
Search

This Product

    Object Storage Service:Migrate data from HDFS to OSS-HDFS

    Document Center

    Object Storage Service:Migrate data from HDFS to OSS-HDFS

    Last Updated:May 01, 2026

    To back up HDFS data or expand your storage space, you can use the Jindo DistCp tool in an Alibaba Cloud EMR cluster to migrate data from HDFS to OSS-HDFS. OSS-HDFS is compatible with the Hadoop ecosystem, allowing you to use its various tools and frameworks for data processing and analysis after the migration.

    Prerequisites

    • You have created an Alibaba Cloud EMR cluster of version EMR-5.6.0 or later, or EMR-3.40.0 or later. For more information, see Create a cluster.

    • Your self-managed cluster built on ECS instances has a Hadoop 2.7+ or 3.x environment and can run MapReduce jobs. Before you migrate data, you must deploy JindoData, which includes JindoSDK and JindoFSx. Download the latest version.

    • You have enabled the OSS-HDFS service and granted the required permissions. For more information, see Enable OSS-HDFS.

    Background

    Alibaba Cloud Jindo DistCp is a distributed tool for copying files within or between large-scale clusters. Jindo DistCp uses MapReduce jobs to distribute files, handle errors, and recover from failures. It takes a list of files and directories as input for MapReduce tasks, where each task copies a subset of files from the source list. Jindo DistCp fully supports copying data between HDFS, OSS, and OSS-HDFS. It provides various parameters and strategies to customize copy jobs.

    Jindo DistCp provides the following benefits:

    • High efficiency: Achieves up to 1.59x speedup in test scenarios.

    • Comprehensive features: Provides multiple copy methods and scenario-based optimization strategies.

    • Deep integration with OSS: Supports file archiving and compression.

    • No-rename copy: Ensures data consistency by avoiding temporary file renames during the copy process.

    • Broad compatibility: Supports Hadoop 2.7+ and Hadoop 3.x, and can be used as a replacement for Hadoop DistCp.

    Procedure

    1. Log on to the EMR cluster.

      1. Log on to the EMR on ECS console.

      2. Click the name of the target EMR cluster.

      3. Click the Nodes tab, and then click the +.jpg icon to the left of a node group.

      4. Click the ECS ID. On the ECS instance page, click Connect next to the instance ID.

        To connect using SSH, see Log on to a cluster.

    2. Migrate data from a specified HDFS path to OSS-HDFS.

      1. Run the following command to view the data in HDFS:

        hdfs dfs -ls /

        Sample response:

        Found 8 items
drwxrwxrwx - admin supergroup 0 2023-10-26 10:55 /.sysinfo
drwxrwxrwx - hadoop supergroup 0 2023-10-26 10:55 /apps
drwxrwxrwx - root supergroup 0 2022-08-03 15:54 /data
-rw-r----- 1 root supergroup 13 2022-08-25 11:45 /examplefile.txt
drwxrwxrwx - spark supergroup 0 2023-10-26 14:49 /spark-history
drwx-wx-wx - hive supergroup 0 2023-10-26 13:35 /tmp
drwxrwxrwx - hive supergroup 0 2023-10-26 14:48 /user
drwxrwxrwx - hadoop supergroup 0 2023-10-26 14:48 /yarn

      2. Navigate to the directory that contains the jindo-distcp-tool-${version}.jar package.

        cd /opt/apps/JINDOSDK/jindosdk-current/tools

      3. View the version of the Jindo DistCp tool automatically deployed with the cluster.

        ls

      4. Migrate data from HDFS to OSS-HDFS.

        Full migration

        To perform a full migration or copy of data from the /tmp directory in HDFS to the oss://examplebucket/dir destination path in OSS-HDFS, run the following command. This example uses jindo-distcp-tool-6.1.0.jar. Replace the JAR version number to match your environment.

        hadoop jar jindo-distcp-tool-6.1.0.jar --src /tmp/ --dest oss://examplebucket.cn-hangzhou.oss-dls.aliyuncs.com/dir/ --hadoopConf fs.oss.accessKeyId=LTAI************************  --hadoopConf fs.oss.accessKeySecret=yourAccessKeySecret --parallelism 10

        The following table describes the parameters and options in the command.

        Parameter

        Description

        Example

        --src

        The source path of the data in HDFS.

        /tmp/

        --dest

        The destination path in OSS-HDFS.

        oss://destbucket.cn-hangzhou.oss-dls.aliyuncs.com/dir/

        --hadoopConf

        The AccessKey ID and AccessKey Secret for accessing the OSS-HDFS service.

        • AccessKey ID

          LTAI************************

        • AccessKey Secret

          yourAccessKeySecret

        --parallelism

        The degree of parallelism for the copy job. Adjust this value based on your cluster resources.

        10

        Incremental migration

        To copy only the data added to the source path since the last full migration, use the --update option for an incremental copy.

        For example, to incrementally migrate or copy data from the /data/ directory in HDFS to the oss://destbucket/dir destination path in OSS-HDFS, run the following command:

        hadoop jar jindo-distcp-tool-6.1.0.jar --src /data/ --dest oss://destbucket.cn-hangzhou.oss-dls.aliyuncs.com/dir/ --hadoopConf fs.oss.accessKeyId=LTAI************************  --hadoopConf fs.oss.accessKeySecret=yourAccessKeySecret --update --parallelism 10

    Related documents

    • If you do not want to manually enter the Endpoint and AccessKey each time you run a migration command, you can configure the Endpoint and AccessKey in the Hadoop configuration file core-site.xml. For more information, see Connect a non-EMR cluster to the OSS-HDFS service.

    • For more information about other use cases for Jindo DistCp, see Jindo DistCp usage guide.