Deploy JindoSDK in a non-EMR cluster - E-MapReduce - Alibaba Cloud Documentation Center

This topic describes how to deploy JindoSDK in a non-EMR cluster. It covers the steps to download JindoSDK, configure environment variables, and modify configuration files.

Deploy JindoSDK

You can deploy JindoSDK in a non-EMR environment, such as on an Alibaba Cloud ECS instance, other Alibaba Cloud services, or your on-premises servers. This topic uses an Alibaba Cloud ECS instance as an example. For more information about how to connect to an ECS instance, see Connect to an ECS instance.

Run the following command to download the TAR package for a specific JindoSDK version.
This topic uses version 6.3.4 for the Linux x86 platform as an example.
```
wget https://jindodata-binary.oss-cn-shanghai.aliyuncs.com/release/6.3.4/jindosdk-6.3.4-linux.tar.gz
```
Run the following command to decompress the JindoSDK JAR package.
```
tar zxvf jindosdk-6.3.4-linux.tar.gz
```
Configure environment variables.
Important
You must deploy the installation folder and configure the environment variables on all required nodes.
For example, you can execute the following command to extract the installation package to the /usr/lib/jindosdk-6.3.4 directory.
```
export JINDOSDK_HOME=/usr/lib/jindosdk-6.3.4-linux
export JINDOSDK_CONF_DIR=${JINDOSDK_HOME}/conf
export PATH=${PATH}:${JINDOSDK_HOME}/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${JINDOSDK_HOME}/lib/native
```

Modify configurations

Hadoop configuration file

Configure the implementation class and AccessKey for OSS or OSS-HDFS.

Edit the Hadoop core-site.xml configuration file.
```
vim <HADOOP_HOME>/etc/hadoop/core-site.xml
```

Add the implementation class for OSS or OSS-HDFS to the core-site.xml file.

<configuration>
    <property>
        <name>fs.AbstractFileSystem.oss.impl</name>
        <value>com.aliyun.jindodata.oss.JindoOSS</value>
    </property>

    <property>
        <name>fs.oss.impl</name>
        <value>com.aliyun.jindodata.oss.JindoOssFileSystem</value>
    </property>
</configuration>

Add the AccessKey ID and AccessKey secret for the OSS or OSS-HDFS bucket to the core-site.xml file.

<configuration>
    <property>
        <name>fs.oss.accessKeyId</name>
        <value>xxx</value>
    </property>

    <property>
        <name>fs.oss.accessKeySecret</name>
        <value>xxx</value>
    </property>
</configuration>

Configure the Endpoint for the OSS or OSS-HDFS service.
You must configure an Endpoint to access an OSS or OSS-HDFS bucket. Use the following format for the access path: oss://<Bucket>.<Endpoint>/<Object>. For example: oss://examplebucket.cn-hangzhou.oss-dls.aliyuncs.com/exampleobject.txt. After the configuration is complete, JindoSDK uses the Endpoint in the access path to call the corresponding OSS or OSS-HDFS service API. You can also configure a default Endpoint to simplify the access path to the oss://<Bucket>/<Object> format. For example: oss://examplebucket/exampleobject.txt.
```
<configuration>
    <property>
        <name>fs.oss.endpoint</name>
        <value>xxx</value>
    </property>
</configuration>
```
For more information about configuration methods, see the following topics:

Non-Hadoop configuration file

When you use components outside the Hadoop ecosystem, such as JindoFuse and the Jindo command-line interface (CLI), JindoSDK reads the configuration file from the folder specified by the JINDOSDK_CONF_DIR environment variable.

Configuration file

Use an INI-style configuration file named jindosdk.cfg. The following code provides an example.

[common]
logger.dir = /tmp/jindosdk-log

[jindosdk]
# The Endpoint of the created OSS bucket. For example, for the China (Hangzhou) region, enter oss-cn-hangzhou.aliyuncs.com.
# The Endpoint of the created OSS-HDFS bucket. For example, for the China (Hangzhou) region, enter cn-hangzhou.oss-dls.aliyuncs.com.
fs.oss.endpoint = <your_Endpoint>
# The AccessKey ID and AccessKey secret used to access OSS. The AccessKey of an Alibaba Cloud account has permissions to access all APIs. This poses a high security threat. Create and use a Resource Access Management (RAM) user for API calls or routine O&M. To create a RAM user, log on to the RAM console.
fs.oss.accessKeyId = <your_AccessKey_id>
fs.oss.accessKeySecret = <your_AccessKey_secret>

Passwordless access

To use passwordless access, you must use an Alibaba Cloud ECS instance that has an instance RAM role attached. For more information, see Instance RAM roles.

The following code provides an example.

[common]
logger.dir = /tmp/jindosdk-log

[jindosdk]
# The Endpoint of the created OSS bucket. For example, for the China (Hangzhou) region, enter oss-cn-hangzhou.aliyuncs.com.
# The Endpoint of the created OSS-HDFS bucket. For example, for the China (Hangzhou) region, enter cn-hangzhou.oss-dls.aliyuncs.com.
fs.oss.endpoint = <your_Endpoint>
fs.oss.provider.endpoint = ECS_ROLE
fs.oss.provider.format = JSON