How to use JindoSDK with Spark to query data stored in OSS-HDFS - Object Storage Service

JindoSDK is an OSS client for the Hadoop and Spark ecosystems that provides a highly optimized Hadoop FileSystem implementation. Compared with Hadoop OSS clients, JindoSDK delivers better performance when you use Spark to query data in OSS-HDFS.

Prerequisites

To use Elastic Compute Service (ECS) as your deployment environment, you need to create an instance.
A Hadoop environment is created. For more information, see Step 2: Set up a Hadoop runtime environment.
Apache Spark is deployed. For more information, visit Apache Spark.
OSS-HDFS is enabled for a bucket and permissions are granted to access OSS-HDFS. For more information, see Enable OSS-HDFS.

Procedure

Connect to the ECS instance. For more information, see Connect to an ECS instance.
Configure JindoSDK.
1. Download the latest version of the JindoSDK JAR package. To download JindoSDK, visit GitHub.
2. Decompress the JindoSDK JAR package.
  
  The following example shows how to decompress a package named jindosdk-x.x.x-linux.tar.gz. If you use another version of JindoSDK, replace the package name with the name of the corresponding JAR package.
```
tar zxvf jindosdk-x.x.x-linux.tar.gz
```
  Note
  x.x.x indicates the version number of the JindoSDK JAR package.
3. Copy the downloaded JindoSDK JAR files to the classpath.
```
cp jindosdk-x.x.x-linux/lib/*.jar  $SPARK_HOME/jars/
```

Configure the implementation class of OSS-HDFS and specify the AccessKey pair that you want to use to access the bucket.

Configure the settings in the core-site.xml file

Configure the implementation class of OSS-HDFS in the core-site.xml file of Spark.

<configuration>
    <property>
        <name>fs.AbstractFileSystem.oss.impl</name>
        <value>com.aliyun.jindodata.oss.JindoOSS</value>
    </property>

    <property>
        <name>fs.oss.impl</name>
        <value>com.aliyun.jindodata.oss.JindoOssFileSystem</value>
    </property>
</configuration>

Configure the AccessKey ID and AccessKey secret used to access the OSS-HDFS-enabled bucket in the core-site.xml configuration file of Spark.

<configuration>
    <property>
        <name>fs.oss.accessKeyId</name>
        <value>LTAI********</value>
    </property>

    <property>
        <name>fs.oss.accessKeySecret</name>
        <value>KZo1********</value>
    </property>
</configuration>

Configure the settings when you submit Spark jobs

The following example shows how to configure the implementation class of OSS-HDFS and specify the AccessKey pair used to access a bucket when you submit Spark jobs:

spark-submit --conf spark.hadoop.fs.AbstractFileSystem.oss.impl=com.aliyun.jindodata.oss.OSS --conf spark.hadoop.fs.oss.impl=com.aliyun.jindodata.oss.JindoOssFileSystem --conf spark.hadoop.fs.oss.accessKeyId=LTAI********  --conf spark.hadoop.fs.oss.accessKeySecret=KZo149BD9GLPNiDIEmdQ7d****

Configure the OSS-HDFS service Endpoint.
To access an OSS bucket using the OSS-HDFS service, you must configure an Endpoint. We recommend that you use an access path in the oss://<Bucket>.<Endpoint>/<Object> format, such as oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/exampleobject.txt. After you complete the configuration, JindoSDK uses the Endpoint in the access path to connect to the correct OSS-HDFS service API.
You can also configure the OSS-HDFS service Endpoint in other ways. Endpoints that are configured in different ways have different priorities. For more information, see Appendix 1: Other ways to configure an Endpoint.

Use Spark to access OSS-HDFS.

Create a table.

create table test_oss (c1 string) location "oss://examplebucket.cn-hangzhou.oss-dls.aliyuncs.com/dir/";

Insert data into the table.

insert into table test_oss values ("testdata");

Query data in the table.
```
select * from test_oss;
```