Use POSIX commands to access OSS or OSS-HDFS - E-MapReduce

This topic describes how to use Portable Operating System Interface (POSIX) commands to access Object Storage Service (OSS) or OSS-HDFS.

Environment preparation

In the E-MapReduce (EMR) environment, JindoSDK is installed by default and can be directly used.
Note To access OSS-HDFS, you must create a cluster of EMR V3.42.0 or a later minor version, or EMR V5.8.0 or a later minor version.
In a non-EMR environment, install JindoSDK first. For more information, see Deploy JindoSDK in an environment other than EMR.
Note To access OSS-HDFS, you must install JindoSDK V4.X or later.

Install the required dependencies

Note By default, the required dependencies are installed for the following clusters:

Clusters of EMR V3.44.0 or a later minor version or clusters of EMR V5.10.0 or a later minor version
Clusters in which JindoSDK V4.6.2 or later is deployed

If your clusters use JindoSDK 4.5.0 or earlier, you must install the following dependencies:

# CentOS
sudo yum install -y fuse3 fuse3-devel
# Debian
sudo apt install -y fuse3 libfuse3-dev

If your clusters use JindoSDK 4.5.1 or later, you must install libfuse 3.7 or later.

For example, run the following commands to install fuse-3.11:

# build fuse required meson & ninja, for debian: apt install -y pkg-config meson ninja-build
sudo yum install -y meson ninja-build

# compile fuse required newer g++ (only CentOS)
sudo yum install -y scl-utils
sudo yum install -y alinux-release-experimentals
sudo yum install -y devtoolset-8-gcc devtoolset-8-gdb devtoolset-8-binutils devtoolset-8-make devtoolset-8-gcc-c++
sudo su -c "echo 'source /opt/rh/devtoolset-8/enable' > /etc/profile.d/g++.sh"
source /opt/rh/devtoolset-8/enable
sudo ln -s /opt/rh/devtoolset-8/root/bin/gcc /usr/local/bin/gcc
sudo ln -s /opt/rh/devtoolset-8/root/bin/g++ /usr/local/bin/g++

# compile & install libfuse
wget https://github.com/libfuse/libfuse/releases/download/fuse-3.11.0/fuse-3.11.0.tar.xz
xz -d fuse-3.11.0.tar.xz
tar xf fuse-3.11.0.tar
cd fuse-3.11.0/
mkdir build; cd build
meson ..
sudo ninja install

Mount JindoFuse

After you configure JindoSDK, you can perform the following operations to mount JindoFuse.

Run the following command to create a mount point:
```
sudo mkdir -p <mountpoint>
```
Run the following command to mount JindoFuse:
```
jindo-fuse <mount_point> -ouri=[<osspath>]
```
After you run the command, a daemon process starts in the background to mount the specified OSS or OSS-HDFS path to the specified mount point of the local file system.
Replace the value of <mount_point> with a local path. Replace the value of <oss_path> with an OSS or OSS-HDFS path to be mapped. The path can be the root directory or a subdirectory of the OSS or OSS-HDFS bucket. Example: oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/subdir/.
Note The methods to mount OSS and OSS-HDFS paths are basically the same. Only the endpoints in the paths are different. For more information, see Path description.
Run the following command to check whether JindoFuse is mounted:
```
ps -ef | grep jindo-fuse
```
If the jindo-fuse process is running and the startup parameters are as expected, JindoFuse is mounted.

Access JindoFuse

If JindoFS is mounted to the local path /mnt/oss/, perform the following operations to access JindoFuse:

Run the following command to view all directories in the /mnt/oss/ path:
```
ls /mnt/oss/
```
Run the following commands to create a directory named dir1 and view the directory:
```
mkdir /mnt/oss/dir1
ls /mnt/oss/
```
Run the following command to write data to a file named hello.txt:
```
echo "hello world" > /mnt/oss/dir1/hello.txt
```
Run the following command to read data from the hello.txt file:
```
cat /mnt/oss/dir1/hello.txt
```
hello world is displayed.
Run the following command to delete the dir1 directory:
```
rm -rf /mnt/oss/dir1/
```

Manually uninstall JindoFuse

Run the following command to unmount the mount point:

umount <mount_point>

Automatically uninstall JindoFuse

You can use the -oauto_unmount parameter to automatically unmount the mount point.

If you use this parameter, you can run the killall -9 jindo-fuse command to send a SIGINT signal to the jindo-fuse process. The mount point is automatically unmounted before the process exits.

For more information, see User guide of JindoFuse.