Accessing OSS-HDFS in EMR Hive or Spark - Object Storage Service

You can use OSS-HDFS (JindoFS) as a data store for E-MapReduce (EMR) clusters running version 3.42 or later, or version 5.8.0 or later. OSS-HDFS provides features such as cache acceleration and Ranger authorization to improve performance and simplify migration from HDFS in big data ETL scenarios that use Hive or Spark. This topic describes how to access data in OSS-HDFS by using Hive or Spark on EMR.

Prerequisites

OSS-HDFS is enabled for a bucket and permissions are granted to a RAM role to access OSS-HDFS. For more information, see Enable OSS-HDFS and grant access permissions.
By default, an Alibaba Cloud account has the permissions to connect EMR clusters to OSS-HDFS and perform common operations related to OSS-HDFS. A RAM user that is granted the required permissions is created. If you want to use a RAM user to connect EMR clusters to OSS-HDFS, the RAM user must have the required permissions. For more information, see Grant a RAM user permissions to connect EMR clusters to OSS-HDFS.

Background

OSS-HDFS is a cloud-native data lake storage service. With its unified metadata management capability, OSS-HDFS is fully compatible with the HDFS file system interface and provides comprehensive POSIX support. This makes OSS-HDFS an ideal solution for a variety of data lake computing scenarios in the big data and AI fields. For more information, see What is OSS-HDFS?.

Procedure

Note This topic uses Hive to demonstrate how to work with data in OSS-HDFS. The same method applies to Spark.

Log on to the cluster. For more information, see Log on to a cluster.
Create a Hive table that points to OSS-HDFS.
1. Run the following command to enter the Hive CLI.
```
hive
```
2. Run the following command to create a database that points to OSS-HDFS.
```
CREATE DATABASE if not exists dw LOCATION 'oss://{yourHdfsBucketDomain}/{path}';
```
  Parameter description:
  - dw: The database name. You can customize this name.
  - {path}: The path in OSS-HDFS to store the database. You can customize this path.
  - {yourHdfsBucketDomain}: The bucket domain name for the OSS-HDFS service.
    - To get the endpoint, log on to the OSS console. Navigate to the target bucket.On the Overview tab, in the Access Ports section, copy the full bucket endpoint for the OSS-HDFS.
  Note
  This example uses the OSS-HDFS domain name as the path prefix. If you want to use only the bucket name to point to OSS-HDFS, you can configure a bucket-level endpoint or a global endpoint. For more information, see Appendix 1: Other ways to configure an endpoint.
3. Run the following command to use the newly created database.
```
use dw;
```
4. Run the following command to create a table in the new database.
```
CREATE TABLE IF NOT EXISTS employee(eid int, name String,salary String,destination String)
COMMENT 'Employee details';
```
Verify the table data.
```
SELECT * FROM employee WHERE eid = 1;
```

Verify the table data.

SELECT * FROM employee WHERE eid = 1;

The response includes the inserted data.

OK
1       liu hua 100.0
Time taken: 12.379 seconds, Fetched: 1 row(s)