Integrate OSS-HDFS with EMR Hive and Spark for Big Data Pipelines - E-MapReduce - Alibaba Cloud - E-MapReduce

E-MapReduce (EMR) clusters running version 3.42 or later, or version 5.8.0 or later support OSS-HDFS (JindoFS) as a data store. OSS-HDFS provides cache acceleration and Ranger authentication, which helps improve performance and simplifies HDFS migration for big data ETL scenarios that use Hive or Spark. This topic describes how to use Hive or Spark in an EMR cluster to access data in OSS-HDFS.

Background information

OSS-HDFS is a cloud-native data lake storage service. With its unified metadata management, it is fully compatible with the HDFS file system interface and provides comprehensive POSIX support. This makes it ideal for a wide range of data lake computing scenarios in big data and AI. For more information, see What is OSS-HDFS?.

Prerequisites

You have created an EMR cluster. For more information, see Create a cluster.

Step 1: Enable OSS-HDFS

Enable OSS-HDFS and grant the required access permissions. For more information, see Enable OSS-HDFS.

Step 2: Obtain the OSS-HDFS endpoint

On the Overview page of your bucket in the OSS console, in the Access Domain Names section, copy the endpoint from the HDFS Service row. You need this endpoint to create a Hive table in Step 3: Use OSS-HDFS in an EMR cluster. The endpoint is in the format of <Bucket-name>.<region>.oss-dls.aliyuncs.com.

Step 3: Use OSS-HDFS in an EMR cluster

Note This topic uses Hive as an example to demonstrate how to operate on data in OSS-HDFS. You can follow similar steps to do the same with Spark.

Log on to the cluster. For more information, see Log on to a cluster.
Create a Hive table that points to OSS-HDFS.
1. Run the following command to enter the Hive command line.
```
hive
```
2. Run the following command to create a database that points to OSS-HDFS.
```
CREATE DATABASE if not exists dw LOCATION 'oss://<your-oss-hdfs-endpoint>/<path>';
```
  Note
  
  In the preceding command, dw is the database name, <path> is an arbitrary path, and <your-oss-hdfs-endpoint> is the OSS-HDFS endpoint that you obtained in Step 2: Obtain the OSS-HDFS endpoint.
  
  This example uses the OSS-HDFS endpoint as a path prefix. If you want to point to OSS-HDFS by using only the bucket name, you can configure a bucket-level or global endpoint. For more information, see Connect to OSS-HDFS from a non-EMR cluster.
3. Run the following command to use the new database.
```
use dw;
```
4. Run the following command to create a table in the new database.
```
CREATE TABLE IF NOT EXISTS employee(eid int, name String,salary String,destination String)
COMMENT 'Employee details';
```
5. Run the following command to view the table information.
```
desc formatted employee;
```
  The following output is returned. The Location field shows that the table points to a path on OSS-HDFS.
```
# col_name              data_type               comment
eid                     int
name                    string
salary                  string
destination             string
# Detailed Table Information
Database:               dw
Owner:                  root
CreateTime:             Fri May 06 16:40:06 CST 2022
LastAccessTime:         UNKNOWN
Retention:              0
Location:               oss://****.cn-hangzhou.oss-dls.aliyuncs.com/dw/employee
Table Type:             MANAGED_TABLE
```
Insert data into the table.
Use the INSERT INTO statement to write data to the table. This statement generates a MapReduce job.
```
INSERT INTO employee(eid, name, salary, destination) values(1, 'John Doe', '100.0', '');
```

Verify the table data.

SELECT * FROM employee WHERE eid = 1;

The output includes the data that you inserted.

OK
1       John Doe 100.0
Time taken: 12.379 seconds, Fetched: 1 row(s)

Authorize the EMR cluster

If your EMR cluster does not use the default AliyunECSInstanceForEMRRole instance RAM role, you must grant permissions to the cluster.

The AliyunECSInstanceForEMRRole instance RAM role is associated with the AliyunECSInstanceForEMRRolePolicy policy, which includes the oss:PostDataLakeStorageFileOperation permission by default. Therefore, clusters that use the default role can access OSS-HDFS without additional authorization.