Enable password-free access to a JindoFS file system using the SDK - E-MapReduce

This topic describes how to use the JindoFS SDK to enable password-free access to a JindoFS file system on E-MapReduce (EMR) from a client outside an EMR cluster.

Prerequisites

This procedure applies to an ECS instance that is outside an E-MapReduce (EMR) cluster and has Hadoop and a Java SDK installed.

Background information

When you use the JindoFS SDK, you must remove related Jindo packages from the environment, such as jboot.jar and smartdata-aliyun-jfs-*.jar. If you use Spark, you must also remove the packages from the /opt/apps/spark-current/jars/ directory to ensure proper functionality.

Step 1: Create an instance RAM role

Log on to the Resource Access Management (RAM) console by using your Alibaba Cloud account.
In the navigation pane on the left, choose Identities > Roles.
Click Create Role. For Alibaba Cloud Service, select Alibaba Cloud Service.
Click Next.
Enter a Role Name. From the Select Trusted Service list, select Elastic Compute Service.
Click Complete.

Step 2: Grant permissions to the RAM role

Log on to the RAM console by using your Alibaba Cloud account.
(Optional) If a system policy does not meet your requirements, create a custom policy. For more information, see Create a custom policy.
In the navigation pane on the left, choose Identities > Roles.
Click the name of the role you created.
Click Input and Attach.
For System Policy, select System Policy or Custom Policy.

For example, you can use the AliyunOSSReadOnlyAccess policy. Select a policy based on your needs.
Search for the policy by name, and then click it to add it to the right-side panel.
Click OK.

Step 3: Attach the RAM role to an instance

Log on to the ECS console.
In the navigation pane on the left, choose Instances & Images > Instance.
In the top navigation bar, select the region of your instance.
Find the target ECS instance. In the Actions column, choose > Instance Settings > Attach/Detach RAM Role.
In the dialog box, select the instance RAM role you created and click OK.

Step 4: Set environment variables

Run the following command on the ECS instance to set the CLASSPATH environment variable. Replace the placeholder path with the path to your JindoFS SDK file.

export CLASSPATH=/path/to/jindofs-x.y.z-sdk.jar

Alternatively, run the following command to update the HADOOP_CLASSPATH variable. Replace the placeholder path with the path to your JindoFS SDK file.

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/path/to/jindofs-x.y.z-sdk.jar

Step 5: Verify password-free access

Access Object Storage Service (OSS) from the shell.
```
hdfs dfs -ls oss://<your-bucket-name>/
```

Access OSS using the Hadoop file system API.

The JindoFS SDK lets you access OSS using the Hadoop file system API. The following code is an example. In the example, replace ossPath with your OSS path, such as oss://your-bucket-name/your-directory/.

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.LocatedFileStatus;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.RemoteIterator;
import java.net.URI;
public class test {
    public static void main(String[] args) throws Exception {
        String ossPath = "oss://your-bucket-name/your-directory/";
        FileSystem fs = FileSystem.get(new URI(ossPath), new Configuration());
        RemoteIterator<LocatedFileStatus> iterator = fs.listFiles(new Path(ossPath), false);
        while (iterator.hasNext()){
            LocatedFileStatus fileStatus = iterator.next();
            Path fullPath = fileStatus.getPath();
            System.out.println(fullPath);
        }
    }
}