Quick start: Connect a non-EMR cluster to the OSS-HDFS service - Object Storage Service

The Object Storage Service (OSS)-Hadoop Distributed File System (HDFS) service, also known as JindoFS, is fully compatible with HDFS interfaces and supports directory-level operations. The Jindo software development kit (SDK) allows Apache Hadoop computing and analysis applications, such as MapReduce, Hive, Spark, and Flink, to access the OSS-HDFS service. This topic describes how to deploy the JindoSDK on an Elastic Compute Service (ECS) instance and perform common operations to get started with the OSS-HDFS service.

Note

If you use an Alibaba Cloud E-MapReduce (EMR) cluster, you can connect the cluster to the OSS-HDFS service. For more information, see Connect EMR clusters to OSS-HDFS.

Prerequisites

An Alibaba Cloud account has the required permissions by default. If you use a Resource Access Management (RAM) user to access the OSS-HDFS service, you must grant the required permissions to the RAM user. For more information, see Grant a RAM user permissions to connect a non-EMR cluster to the OSS-HDFS service.
For example, if you use Alibaba Cloud ECS for your deployment, you need to purchase an ECS instance.
A Hadoop environment is created. For more information, see Create a Hadoop runtime environment.
The OSS-HDFS service is enabled for a bucket and access to the service is authorized. For more information, see Enable the OSS-HDFS service.

Video tutorial

The following video shows how to quickly connect a non-EMR cluster to the OSS-HDFS service and perform common operations.

Procedure

Connect to the ECS instance. For more information, see Connect to an ECS instance.
Download and decompress the JindoSDK JAR package. To download the package, visit GitHub.
Run the following command to decompress the JindoSDK JAR package.

The following example shows how to decompress jindosdk-x.x.x-linux.tar.gz. If you use a different version of the JindoSDK, replace the package name with the actual file name.
```
tar zxvf jindosdk-x.x.x-linux.tar.gz
```
Note
x.x.x represents the version number of the JindoSDK JAR package.
Configure environment variables.
1. Configure JINDOSDK_HOME.
  The following example assumes the package is decompressed to the /usr/lib/jindosdk-x.x.x-linux directory:
```
export JINDOSDK_HOME=/usr/lib/jindosdk-x.x.x-linux
```
2. Configure HADOOP_CLASSPATH.
```
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:${JINDOSDK_HOME}/lib/*
```
  Important
  Deploy the installation directory and environment variables to all required nodes.

Configure the OSS-HDFS service implementation class and AccessKey pair.

Run the following command to open the core-site.xml configuration file of Hadoop.
```
vim /usr/local/hadoop/etc/hadoop/core-site.xml
```

In the core-site.xml file, configure the JindoSDK DLS implementation class.

<configuration>
    <property>
        <name>fs.AbstractFileSystem.oss.impl</name>
        <value>com.aliyun.jindodata.oss.JindoOSS</value>
    </property>

    <property>
        <name>fs.oss.impl</name>
        <value>com.aliyun.jindodata.oss.JindoOssFileSystem</value>
    </property>
</configuration>

In the core-site.xml file, configure the AccessKey pair of your Alibaba Cloud account or a RAM user that has the required permissions.

For more information about the permissions required by a RAM user in this scenario, see Grant a RAM user permissions to connect a non-EMR cluster to the OSS-HDFS service.
```
<configuration>
    <property>
        <name>fs.oss.accessKeyId</name>
        <value>xxx</value>
    </property>

    <property>
        <name>fs.oss.accessKeySecret</name>
        <value>xxx</value>
    </property>
</configuration>
```

Configure the OSS-HDFS service Endpoint.

You must configure an Endpoint to access an OSS bucket using the OSS-HDFS service. The recommended format for the access path is oss://<Bucket>.<Endpoint>/<Object>. For example, oss://examplebucket.cn-hangzhou.oss-dls.aliyuncs.com/exampleobject.txt. After you complete the configuration, the JindoSDK uses the Endpoint in the access path to access the corresponding OSS-HDFS service API.
Use HDFS Shell commands to perform common OSS-HDFS operations.
- Upload a file
  
  To upload the examplefile.txt file from the local root directory to examplebucket, run the following command:
```
hdfs dfs -put examplefile.txt oss://examplebucket.cn-hangzhou.oss-dls.aliyuncs.com/
```
- Download a file
  
  To download the exampleobject.txt file from examplebucket to the local /tmp directory, run the following command:
```
hdfs dfs -get oss://examplebucket.cn-hangzhou.oss-dls.aliyuncs.com/exampleobject.txt  /tmp/
```
For more information about operations, see Access OSS-HDFS using Hadoop shell commands.

Appendix 1: Performance tuning

You can add the following configuration items to the core-site.xml file of Hadoop to tune performance. These configuration items are supported only in JindoSDK 4.0 and later.

<configuration>

    <property>
          <!-- The directories where the client writes temporary files. You can specify multiple directories separated by commas (,). In multi-user environments, grant read and write permissions. -->
        <name>fs.oss.tmp.data.dirs</name>
        <value>/tmp/</value>
    </property>

    <property>
          <!-- The number of retries when an attempt to access OSS fails. -->
        <name>fs.oss.retry.count</name>
        <value>5</value>
    </property>

    <property>
          <!-- The timeout for OSS requests, in milliseconds. -->
        <name>fs.oss.timeout.millisecond</name>
        <value>30000</value>
    </property>

    <property>
          <!-- The timeout for OSS connections, in milliseconds. -->
        <name>fs.oss.connection.timeout.millisecond</name>
        <value>3000</value>
    </property>

    <property>
          <!-- The number of concurrent threads to upload a single file to OSS. -->
        <name>fs.oss.upload.thread.concurrency</name>
        <value>5</value>
    </property>

    <property>
          <!-- The size of the queue for concurrent OSS upload tasks. -->
        <name>fs.oss.upload.queue.size</name>
        <value>5</value>
    </property>

    <property>
          <!-- The maximum number of concurrent OSS upload tasks per stream in a process. -->
        <name>fs.oss.upload.max.pending.tasks.per.stream</name>
        <value>16</value>
    </property>

    <property>
          <!-- The size of the queue for concurrent OSS download tasks. -->
        <name>fs.oss.download.queue.size</name>
        <value>5</value>
    </property>

    <property>
          <!-- The number of concurrent threads for downloading from OSS. -->
        <name>fs.oss.download.thread.concurrency</name>
        <value>16</value>
    </property>

    <property>
          <!-- The buffer size for pre-reading data from OSS. -->
        <name>fs.oss.read.readahead.buffer.size</name>
        <value>1048576</value>
    </property>

    <property>
          <!-- The number of buffers for concurrently pre-reading data from OSS. -->
        <name>fs.oss.read.readahead.buffer.count</name>
        <value>4</value>
    </property>

</configuration>