Migrate data from HDFS to OSS or OSS-HDFS

Use Jindo DistCp to copy data from Hadoop Distributed File System (HDFS) to Alibaba Cloud Object Storage Service (OSS) or OSS-HDFS.

Prerequisites

Before you begin, ensure that you have:

An EMR cluster with Jindo DistCp downloaded. For setup instructions, see Use Jindo DistCp
An AccessKey pair with access to the destination OSS bucket or OSS-HDFS path

Network requirement

JindoSDK 4.4.0 and later routes OSS and OSS-HDFS traffic through different domain names. By default, read and write operations use the OSS internal endpoint.

If you run the distcp command from outside Alibaba Cloud's internal network, add the OSS public endpoint to the core-site.xml file of the Hadoop-Common service:

<configuration>
    <property>
        <name>fs.oss.data.endpoint</name>
        <value>oss-cn-xxx.aliyuncs.com</value>
    </property>
</configuration>

Copy data from HDFS to OSS or OSS-HDFS

Step 1: Configure your AccessKey

Add your AccessKey to core-site.xml in Hadoop-Common so you do not have to pass credentials on every command.

Log on to the E-MapReduce (EMR) console.
Find the Hadoop-Common service for your cluster, go to the Configure tab, and open core-site.xml.
Add the following configuration items. For detailed steps, see Manage configuration items.
Configuration item Description
fs.oss.accessKeyId AccessKey ID used to access OSS or OSS-HDFS
fs.oss.accessKeySecret AccessKey secret used to access OSS or OSS-HDFS

Configuration item	Description
`fs.oss.accessKeyId`	AccessKey ID used to access OSS or OSS-HDFS
`fs.oss.accessKeySecret`	AccessKey secret used to access OSS or OSS-HDFS

Step 2: Run the copy command

Log on to your cluster in SSH mode. For instructions, see Log on to a cluster.
Go to the Jindo DistCp tools directory and note the JAR filename:
```
/opt/apps/JINDOSDK/jindosdk-current/tools
```
The JAR follows the naming pattern jindo-distcp-tool-${version}.jar, for example, jindo-distcp-tool-4.6.11.jar.

Run the following command to copy data from HDFS to OSS or OSS-HDFS:

hadoop jar jindo-distcp-tool-${version}.jar --src /data --dest oss://destBucket.cn-xxx.oss-dls.aliyuncs.com/dir/ --parallelism 10

Parameter	Description	Example
`--src`	Source path in HDFS	`/data`
`--dest`	Destination path in OSS or OSS-HDFS	OSS: `oss://destBucket/`; OSS-HDFS: `oss://destBucket.cn-xxx.oss-dls.aliyuncs.com/`
`--parallelism`	Number of concurrent copy tasks. Adjust based on cluster resources.	`10`

To access OSS-HDFS, specify the destination path in the format oss://<Bucket>.<Endpoint>/<Object>, for example, oss://mydlsbucket.cn-shanghai.oss-dls.aliyuncs.com/Test. For other ways to configure the OSS-HDFS endpoint, see Configure an endpoint to access OSS-HDFS (JindoFS).

Advanced operations

Copy incremental files

Use --update to handle two scenarios:

Resume an interrupted job: Copy files that failed in a previous Jindo DistCp job.
Sync new files: Copy files added to the source directory since the last run.

hadoop jar jindo-distcp-tool-${version}.jar --src /data --dest oss://destBucket.cn-xxx.oss-dls.aliyuncs.com/dir/ --update --parallelism 20

Control resource usage with YARN queue and bandwidth

To limit the impact of a migration job on production workloads, assign it to a dedicated YARN queue and cap the per-node bandwidth:

hadoop jar jindo-distcp-tool-${version}.jar --src /data --dest oss://destBucket.cn-xxx.oss-dls.aliyuncs.com/dir/ --hadoopConf mapreduce.job.queuename=yarnQueue --bandWidth 100 --parallelism 10

Parameter	Description	Example
`--hadoopConf mapreduce.job.queuename`	Name of the YARN queue to submit the job to	`yarnQueue`
`--bandWidth`	Bandwidth cap per Elastic Compute Service (ECS) node, in MB/s.	`100`

What's next

If you encounter errors during migration, see FAQ about Jindo DistCp for troubleshooting guidance.