Use Hadoop Shell commands to access OSS or OSS-HDFS - E-MapReduce

This topic describes how to use Hadoop Shell commands to access Object Storage Service (OSS) or OSS-HDFS.

Environment preparation

In the E-MapReduce (EMR) environment, JindoSDK is installed by default and can be directly used.
Note To access OSS-HDFS, you must create a cluster of EMR V3.42.0 or a later minor version, or EMR V5.8.0 or a later minor version.
In a non-EMR environment, install JindoSDK first. For more information, see Deploy JindoSDK in an environment other than EMR.
Note To access OSS-HDFS, you must install JindoSDK V4.X or later.

Commands and examples

You can use Hadoop Shell commands to access OSS and OSS-HDFS in the same way. The difference lies in the endpoints in paths.

Upload a file
Run the following command to upload a file named examplefile.txt in the local root directory to a bucket named examplebucket in OSS-HDFS:
```
hadoop fs -put examplefile.txt oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/
```
Create a directory
Run the following command to create a directory named dir/ in a bucket named examplebucket in OSS-HDFS:
```
hadoop fs -mkdir oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/dir/
```
Query files or directories
Run the following command to query the files and directories in a bucket named examplebucket in OSS-HDFS:
```
hadoop fs -ls oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/
```
Query the sizes of files or directories
Run the following command to query the sizes of all files and directories in a bucket named examplebucket in OSS-HDFS:
```
hadoop fs -du oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/
```
Query the content of a file
Run the following command to query the content of a file named localfile.txt in a bucket named examplebucket in OSS-HDFS:
```
hadoop fs -cat oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/localfile.txt
```
Important The content of the queried file is displayed in plain text. If the content is encoded, use HDFS API for Java to read and decode the content.
Copy files or directories
Run the following command to copy a root directory named subdir1 to a directory named subdir2 in a bucket named examplebucket in OSS-HDFS. The location of the subdir1 directory, the files in the subdir1 directory, and the structure and content of subdirectories in the subdir1 directory remain unchanged.
```
hadoop fs -cp oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/subdir1  oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/subdir2/subdir1
```
Move files or directories
Run the following command to move a root directory named srcdir in a bucket named examplebucket and the files and subdirectories in the root directory to another root directory named destdir in OSS-HDFS:
```
hadoop fs -mv oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/srcdir  oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/destdir
```
Download a file
Run the following command to download a file named exampleobject.txt from a bucket named examplebucket in OSS-HDFS to a directory named /tmp in the root directory of your computer:
```
hadoop fs -get oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/exampleobject.txt  /tmp/
```
Delete files or directories
Run the following command to delete a directory named destfolder/ and all files in the directory from a bucket named examplebucket in OSS-HDFS:
```
hadoop fs -rm oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/destfolder/
```