E-MapReduce:JindoDistCp FAQ

Why is object listing taking so long?

When the Jindo DistCp client runs low on heap memory, it cannot buffer listing results efficiently, causing object listing to slow dramatically. In normal conditions, OSS lists 1,000 objects per second. If you see a log line like the following, where 315 objects took roughly 100 seconds to list, the client is out of heap memory:

Successfully list objects with prefix xxx/yyy/ in bucket xxx recursive 0 result 315 dur 100036.615031MS

Increase the JVM heap size before running the job:

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Xmx4096m"

Why do I get "Failed to get checksum store"?

OSS-HDFS uses COMPOSITE_CRC as its default checksum algorithm. If your HDFS cluster has dfs.checksum.combine.mode set to MD5MD5CRC, OSS-HDFS cannot reconcile the difference, and the job fails with this error.

Set fs.oss.checksum.combine.mode to match your HDFS configuration:

hadoop jar jindo-distcp-${version}.jar --src /data --dest oss://destBucket/ --hadoopConf fs.oss.checksum.combine.mode=MD5MD5CRC

Why do I get "verify checksum failed" when copying from OSS to OSS-HDFS?

Jindo DistCp verifies data integrity using checksums written during the original transfer. If the objects in OSS are not migrated from HDFS to OSS by using Jindo DistCp, no matching checksum records exist, and verification fails.

Add --disableChecksum to skip checksum verification for these objects:

hadoop jar jindo-distcp-${version}.jar --src oss://ossBucket/ --dest oss://dlsBucket/ --disableChecksum

How do I verify that a Jindo DistCp job completed successfully?

The answer depends on whether you used --ignore.

Without --ignore: Jindo DistCp stops immediately on any error and stops the copy operation.

With --ignore: The job runs to completion regardless of individual file errors. Check the job counters after the run and confirm that both COPY_FAILED and CHECKSUM_DIFF are zero. If either counter is non-zero, some files were not transferred correctly. For a full list of counters and their meanings, see Jindo DistCp counters in the Use Jindo DistCp topic.