Accessing OSS-HDFS in EMR Hive or Spark - Object Storage Service

E-MapReduce (EMR) clusters running version 3.42 or later, or version 5.8.0 or later support OSS-HDFS (JindoFS) as a data store. OSS-HDFS provides cache acceleration and Ranger authentication, which helps improve performance and simplifies HDFS migration for big data ETL scenarios that use Hive or Spark. This topic describes how to use Hive or Spark in an EMR cluster to access data in OSS-HDFS.

Prerequisites

OSS-HDFS is enabled for a bucket and permissions are granted to a RAM role to access OSS-HDFS. For more information, see Enable OSS-HDFS and grant access permissions.
By default, an Alibaba Cloud account has the permissions to connect EMR clusters to OSS-HDFS and perform common operations related to OSS-HDFS. A RAM user that is granted the required permissions is created. If you want to use a RAM user to connect EMR clusters to OSS-HDFS, the RAM user must have the required permissions. For more information, see Grant a RAM user permissions to connect EMR clusters to OSS-HDFS.

Background information

OSS-HDFS is a cloud-native data lake storage service. With its unified metadata management, it is fully compatible with the HDFS file system interface and provides comprehensive POSIX support. This makes it ideal for a wide range of data lake computing scenarios in big data and AI. For more information, see What is OSS-HDFS?.

Procedure

Note This topic uses Hive as an example to demonstrate how to operate on data in OSS-HDFS. You can follow similar steps to do the same with Spark.

Log on to the cluster. For more information, see Log on to a cluster.
Create a Hive table that points to OSS-HDFS.
1. Run the following command to enter the Hive command line.
```
hive
```
2. Run the following command to create a database that points to OSS-HDFS.
```
CREATE DATABASE if not exists dw LOCATION 'oss://{yourHdfsBucketDomain}/{path}';
```
  Parameter description:
  - dw: The database name. You can customize this name.
  - {path}: The path in OSS-HDFS to store the database. You can customize this path.
  - {yourHdfsBucketDomain}: The bucket domain name for the OSS-HDFS service.
    - To get the endpoint, log on to the OSS console. Navigate to the target bucket.On the Overview tab, in the Access Ports section, copy the full bucket endpoint for the OSS-HDFS.
  Note
  This example uses the OSS-HDFS domain name as the path prefix. If you want to use only the bucket name to point to OSS-HDFS, you can configure a bucket-level endpoint or a global endpoint. For more information, see Appendix 1: Other ways to configure an endpoint.
3. Run the following command to use the new database.
```
use dw;
```
4. Run the following command to create a table in the new database.
```
CREATE TABLE IF NOT EXISTS employee(eid int, name String,salary String,destination String)
COMMENT 'Employee details';
```
Insert data into the table.
Use the INSERT INTO statement to write data to the table. This statement generates a MapReduce job.
```
INSERT INTO employee(eid, name, salary, destination) values(1, 'John Doe', '100.0', '');
```

Verify the table data.

SELECT * FROM employee WHERE eid = 1;

The output includes the data that you inserted.

OK
1       John Doe 100.0
Time taken: 12.379 seconds, Fetched: 1 row(s)