How to use JindoSDK with Impala to query data stored in OSS-HDFS - Object Storage Service

JindoSDK is an OSS client developed for the Hadoop and Spark ecosystems. JindoSDK implements a highly optimized Hadoop file system based on OSS. Compared with the Hadoop community OSS client, JindoSDK delivers better performance when you use Impala to query data stored in OSS-HDFS (JindoFS).

Prerequisites

An Alibaba Cloud account has the required permissions by default. If you use a Resource Access Management (RAM) user to access the OSS-HDFS service, you must grant the required permissions to the RAM user. For more information, see Grant a RAM user permissions to connect a non-EMR cluster to the OSS-HDFS service.
For example, if you use Alibaba Cloud ECS for your deployment, you need to purchase an ECS instance.
A Hadoop environment is created. For more information, see Create a Hadoop runtime environment.
The OSS-HDFS service is enabled for a bucket and access to the service is authorized. For more information, see Enable the OSS-HDFS service.

Procedure

Connect to the ECS instance. For more information, see Connect to an ECS instance.
Configure JindoSDK.
1. Download the latest version of the JindoSDK JAR package. To download JindoSDK, visit GitHub.
2. Decompress the JindoSDK JAR package.
  The following example shows how to decompress a package named jindosdk-x.x.x-linux.tar.gz. Replace the package name with the actual version that you downloaded.
```
tar zxvf jindosdk-x.x.x-linux.tar.gz
```
  Note
  x.x.x indicates the version number of the JindoSDK JAR package.
3. Copy the JindoSDK JAR package to the classpath path of Hive.
```
cp jindosdk-x.x.x-linux/lib/*.jar  $HIVE_HOME/lib/
```
4. Copy the downloaded JindoSDK JAR package to the path specified by classpath.
```
cp jindosdk-x.x.x-linux/lib/*.jar  $HIVE_HOME/lib/
```

Configure the implementation class of OSS-HDFS and specify the AccessKey pair that you want to use to access the bucket.

Configure the implementation class of OSS-HDFS in the core-site.xml file of Impala.

<configuration>
    <property>
        <name>fs.AbstractFileSystem.oss.impl</name>
        <value>com.aliyun.jindodata.oss.JindoOSS</value>
    </property>

    <property>
        <name>fs.oss.impl</name>
        <value>com.aliyun.jindodata.oss.JindoOssFileSystem</value>
    </property>
</configuration>

In the core-site.xml file of Impala, specify the AccessKey ID and AccessKey secret of the account that you use to access the OSS-HDFS-enabled bucket.

<configuration>
    <property>
        <name>fs.oss.accessKeyId</name>
        <value>LTAI********</value>
    </property>

    <property>
        <name>fs.oss.accessKeySecret</name>
        <value>KZo1********</value>
    </property>
</configuration>

Configure the endpoint of OSS-HDFS.

You must specify the endpoint of OSS-HDFS if you want to use OSS-HDFS to access buckets in Object Storage Service (OSS). We recommend that you configure the path that is used to access OSS-HDFS in the oss://{yourBucketName}.{yourBucketEndpoint}/{path} format (example: oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/exampleobject.txt). After you configure the access path, JindoSDK calls the corresponding OSS-HDFS operation based on the specified endpoint in the access path.

You can also configure the endpoint of OSS-HDFS by using other methods. The endpoints that are configured by using different methods have different priorities. For more information, see Quick start: Connect a non-EMR cluster to the OSS-HDFS service.

Use Impala to query data stored in OSS-HDFS.

Create a table.

CREATE EXTERNAL TABLE customer_demographics (
 `cd_demo_sk` INT,
 `cd_gender` STRING,
 `cd_marital_status` STRING,
 `cd_education_status` STRING,
 `cd_purchase_estimate` INT,
 `cd_credit_rating` STRING,
 `cd_dep_count` INT,
 `cd_dep_employed_count` INT,
 `cd_dep_college_count` INT)
STORED AS PARQUET
LOCATION 'oss://bucket.endpoint/dir';

Query data in the table.
```
select * from customer_demographics;
```