This topic describes how to use JindoFileSystem (JindoFS) as the storage back end of HBase.

Overview

In the Hadoop ecosystem, HBase is a database with high write performance that is suitable for real-time data query. HBase clusters of E-MapReduce V3.22.0 or later support JindoFS or JindoFS-based OssFileSystem as the storage back end. Compared with Hadoop Distributed File System (HDFS), JindoFS and JindoFS-based OssFileSystem are more flexible.

JindoFS configuration

For example, a namespace named emr-jfs is created with the following configuration:

  • jfs.namespaces=emr-jfs
  • jfs.namespaces.emr-jfs.uri=oss://oss-bucket/oss-dir
  • jfs.namespaces.emr-jfs.mode=block

Specify the storage location for an HBase cluster

JindoFS and JindoFS-based OssFileSystem in E-MapReduce V3.22.0 do not support data synchronization. You can set the hbase.rootdir parameter in the hbase-site file to a directory in JindoFS or JindoFS-based OssFileSystem, and the hbase.wal.dir parameter to a local directory in HDFS. Then, you can store write-ahead logging (WAL) files through HDFS in an HBase cluster. To release an HBase cluster, disable tables first and make sure that the updates in WAL files have been written to HFile.

Configuration file Parameter Description Example
hbase-site hbase.rootdir The root directory of the HBase cluster in JindoFS. jfs://emr-jfs/hbase-root-dir
hbase.wal.dir The local directory in which the HBase cluster stores WAL files in HDFS. hdfs://emr-cluster/hbase

Create an E-MapReduce cluster

You can add custom configuration when creating an E-MapReduce cluster, as shown in the following figure.

config_sel

Use JindoFS

For example, to use JindoFS as the storage back end of an HBase cluster, use the following custom configuration and replace the Object Storage Service (OSS) bucket and relevant directories:

[   
  {
       "ServiceName":"BIGBOOT",
       "FileName":"bigboot",
       "ConfigKey":"jfs.namespaces",
       "ConfigValue":"emr-jfs"
  },
  {
       "ServiceName":"BIGBOOT",
       "FileName":"bigboot",
       "ConfigKey":"jfs.namespaces.emr-jfs.uri",
       "ConfigValue":"oss://oss-bucket/jindoFS"
  },
  {
       "ServiceName":"BIGBOOT",
       "FileName":"bigboot",
       "ConfigKey":"jfs.namespaces.emr-jfs.mode",
       "ConfigValue":"block"
  },
  {
       "ServiceName":"HBASE",
       "FileName":"hbase-site",
       "ConfigKey":"hbase.rootdir",
       "ConfigValue":"jfs://emr-jfs/hbase-root-dir"
  },
  {
       "ServiceName":"HBASE",
       "FileName":"hbase-site",
       "ConfigKey":"hbase.wal.dir",
       "ConfigValue":"hdfs://emr-cluster/hbase"
  }
]