E-MapReduce:Use POSIX commands to access OSS or OSS-HDFS

Prerequisites

Before you begin, ensure that you have:

• In an EMR cluster: JindoSDK is pre-installed. No additional setup is needed.

  • To access OSS-HDFS, use a cluster running EMR V3.42.0 or later, or EMR V5.8.0 or later.

• Outside an EMR cluster: Install JindoSDK first. See Deploy JindoSDK in an environment other than EMR.

  • To access OSS-HDFS, install JindoSDK V4.X or later.

Install dependencies

Dependencies are pre-installed on:

• EMR clusters running V3.44.0 or later, or V5.10.0 or later

• Environments with JindoSDK V4.6.2 or later

If your environment does not meet the above criteria, install the required dependencies manually based on your JindoSDK version.

# CentOS
sudo yum install -y fuse3 fuse3-devel
# Debian
sudo apt install -y fuse3 libfuse3-dev

# Build fuse requires meson and ninja. On Debian: apt install -y pkg-config meson ninja-build
sudo yum install -y meson ninja-build

# Compile fuse requires a newer g++ (CentOS only)
sudo yum install -y scl-utils
sudo yum install -y alinux-release-experimentals
sudo yum install -y devtoolset-8-gcc devtoolset-8-gdb devtoolset-8-binutils devtoolset-8-make devtoolset-8-gcc-c++
sudo su -c "echo 'source /opt/rh/devtoolset-8/enable' > /etc/profile.d/g++.sh"
source /opt/rh/devtoolset-8/enable
sudo ln -s /opt/rh/devtoolset-8/root/bin/gcc /usr/local/bin/gcc
sudo ln -s /opt/rh/devtoolset-8/root/bin/g++ /usr/local/bin/g++

# Compile and install libfuse
wget https://github.com/libfuse/libfuse/releases/download/fuse-3.11.0/fuse-3.11.0.tar.xz
xz -d fuse-3.11.0.tar.xz
tar xf fuse-3.11.0.tar
cd fuse-3.11.0/
mkdir build; cd build
meson ..
sudo ninja install

Mount JindoFuse

1. Create a mount point:

   sudo mkdir -p <mountpoint>

   Replace <mountpoint> with the local directory path where you want to mount the storage.

2. Mount JindoFuse:

   jindo-fuse <mount_point> -ouri=[<osspath>]

   Replace <osspath> with the OSS or OSS-HDFS path to map — either a bucket root or a subdirectory. For example: oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/subdir/. After the command runs, a daemon process starts in the background and mounts the specified path to the local file system.

   Note OSS and OSS-HDFS use the same mount command. Only the endpoint in the path differs. For path format details, see Path description.

3. Verify the mount:

   ps -ef | grep jindo-fuse

   If the jindo-fuse process appears and the startup parameters match what you specified, the mount is successful.

Access files with POSIX commands

With JindoFuse mounted at /mnt/oss/, use standard shell commands to work with your data:

# List all contents
ls /mnt/oss/

# Create a directory
mkdir /mnt/oss/dir1

# Write data to a file
echo "hello world" > /mnt/oss/dir1/hello.txt

# Read data from a file
cat /mnt/oss/dir1/hello.txt
# Output: hello world

# Delete a directory
rm -rf /mnt/oss/dir1/

For the full list of supported operations and known limitations, see the JindoFuse user guide.

Unmount JindoFuse

umount <mount_point>