Mount multiple data sources to a unified namespace with JindoCache - E-MapReduce

When your EMR jobs need to read data from multiple storage backends—Object Storage Service (OSS) buckets, OSS-HDFS buckets, and on-premises File Storage NAS directories—you typically have to manage separate paths and clients for each backend. JindoCache solves this by mapping all backends into a single Jindo file system namespace. Jobs access every data source through consistent jindo:// paths without application-level changes, and JindoCache accelerates reads with local caching.

Prerequisites

Before you begin, ensure that you have:

JindoCache configured. See Use the transparent caching feature of JindoCache to accelerate access to OSS-HDFS and Use the transparent caching feature of JindoCache to accelerate access to OSS

Mount data sources

Each mount command maps a remote storage path to a local path in the Jindo file system. You can mount multiple buckets or directories of the same type to different paths.

Mount an OSS data source

Mount a directory in an OSS bucket to the /oss path in the Jindo file system:

jindocache -mount /oss oss://<bucket>/<dir>

Mount an OSS-HDFS data source

Mount a directory in an OSS-HDFS bucket to the /oss-hdfs path in the Jindo file system:

jindocache -mount /oss-hdfs oss://<bucket>/<dir>

Mount an on-premises NAS data source

Mount a local NAS directory to the /nas path in the Jindo file system:

jindocache -mount /nas local:///mnt/nas

Access data through the unified namespace

After mounting, all data sources appear under a single jindo:// root. The following example shows the Jindo file system after an OSS directory is mounted to /oss.

List the root directory of the Jindo file system:

hadoop fs -ls jindo://master-1-1:8101/

The /oss mount point is listed:

drwxrwxr-x   - root root          0 1970-01-01 00:00 jindo:///oss

List files and directories under the /oss mount point:

hadoop fs -ls jindo://master-1-1:8101/oss/

The contents of the mounted OSS directory are returned:

drwxrwxr-x   - root root          0 1970-01-01 00:00 jindo:///oss/test_a_dir
-rwxrwxr-x   0 root root   19564111 2022-04-11 10:25 jindo:///oss/tmp.txt

View all data sources in a unified namespace

To see all mounted sources in one view, mount OSS, OSS-HDFS, and NAS simultaneously and then list the root:

# Mount all three backends
jindocache -mount /oss oss://<oss-bucket>/<dir>
jindocache -mount /oss-hdfs oss://<oss-hdfs-bucket>/<dir>
jindocache -mount /nas local:///mnt/nas

# List the unified namespace root
hadoop fs -ls jindo://master-1-1:8101/

Expected output:

drwxrwxr-x   - root root          0 1970-01-01 00:00 jindo:///nas
drwxrwxr-x   - root root          0 1970-01-01 00:00 jindo:///oss
drwxrwxr-x   - root root          0 1970-01-01 00:00 jindo:///oss-hdfs

All three backends appear as sibling directories under the same root, accessible from a single path.

View all mount points

Run the following command to list all active mount points on the current system:

jindocache -mount

For the full list of jindocache command options, see JindoCache CLI usage notes.