E-MapReduce:Access OSS-HDFS through RootPolicy

Step 1: Set up JindoSDK configuration

1. Connect to an ECS instance. See Connect to an ECS instance.

2. Go to the bin directory of the installed JindoSDK JAR file.

   cd jindosdk-x.x.x/bin/

   Replace x.x.x with the actual version number of your JindoSDK JAR file.

3. Create a configuration file named jindosdk.cfg with the following content.

   [common]
   logger.dir = /tmp/jindo/
   logger.sync = false
   logger.consolelogger = false
   logger.level = 0
   logger.verbose = 0
   logger.cleaner.enable = true
   hadoopConf.enable = false
   
   [jindosdk]
   # Replace with the endpoint for your region. This example uses China (Hangzhou).
   fs.oss.endpoint = cn-hangzhou.oss-dls.aliyuncs.com
   # AccessKey credentials for accessing OSS-HDFS.
   fs.oss.accessKeyId = <your-access-key-id>
   fs.oss.accessKeySecret = <your-access-key-secret>

4. Export the JINDOSDK_CONF_DIR environment variable, pointing to the directory that contains jindosdk.cfg.

   export JINDOSDK_CONF_DIR=<absolute-path-to-jindosdk-cfg-directory>

Step 2: Run SetRootPolicy

Run the SetRootPolicy command to register an hdfs:// namespace for a bucket:

jindo admin -setRootPolicy oss://<bucket_name>.<dls_endpoint>/ hdfs://<your_ns_name>/

Optional: avoid repeating the endpoint in every command

Add fs.oss.endpoint to core-site.xml so you don't need to include <dls_endpoint> each time.

Method 1 — applies to all buckets:

<configuration>
  <property>
    <name>fs.oss.endpoint</name>
    <value><dls_endpoint></value>
  </property>
</configuration>

Method 2 — applies to a specific bucket:

<configuration>
  <property>
    <name>fs.oss.bucket.<bucket_name>.endpoint</name>
    <value><dls_endpoint></value>
  </property>
</configuration>

To enable access policies for multiple buckets, separate the oss:// addresses with commas in the fs.accessPolicies.discovery value.

Step 3: Configure access policies in core-site.xml

Add the following properties to core-site.xml:

<configuration>
    <property>
        <name>fs.accessPolicies.discovery</name>
        <value>oss://<bucket_name>.<dls_endpoint>/</value>
    </property>
    <property>
        <name>fs.AbstractFileSystem.hdfs.impl</name>
        <value>com.aliyun.jindodata.hdfs.HDFS</value>
    </property>
    <property>
        <name>fs.hdfs.impl</name>
        <value>com.aliyun.jindodata.hdfs.JindoHdfsFileSystem</value>
    </property>
</configuration>

Step 4: Verify the configuration

Run the following command to confirm that RootPolicy is working:

hadoop fs -ls hdfs://<your_ns_name>/

A successful configuration returns output similar to:

drwxr-x--x   - hdfs  hadoop          0 2023-01-05 12:27 hdfs://<your_ns_name>/apps
drwxrwxrwx   - spark hadoop          0 2023-01-05 12:27 hdfs://<your_ns_name>/spark-history
drwxrwxrwx   - hdfs  hadoop          0 2023-01-05 12:27 hdfs://<your_ns_name>/tmp
drwxrwxrwx   - hdfs  hadoop          0 2023-01-05 12:27 hdfs://<your_ns_name>/user

Step 5: Restart dependent services

Restart services such as Hive and Spark. After the restart, jobs can access OSS-HDFS using the hdfs://<your_ns_name>/ prefix.

Scenario 1: Notebook session

1. On the EMR Serverless Spark page, click Session Manager in the left-side navigation pane.

2. On the Notebook Sessions page, click Create Notebook Session.

3. On the Create Notebook Session page, add the Spark configuration above.

4. In a Data Development Notebook task, use the hdfs://<ns_name>/ prefix directly in your SQL statements.

   # Create table
   spark.sql("""CREATE TABLE default.my_orc_table (
       id INT,
       name STRING,
       age INT
   )
   location 'hdfs://<ns_name>/user/hive/warehouse/ads_user_info_1d_emr/my_orc_table/'""")
   
   # Insert data
   spark.sql("""INSERT INTO table default.my_orc_table(id, name, age) VALUES (1, 'Alice', 30)""")
   
   # Query
   spark.sql("SELECT * FROM default.my_orc_table").show()

   Replace <ns_name> with the custom namespace you set in the SetRootPolicy command.

   

Scenario 2: SQL session

1. On the EMR Serverless Spark page, click Sessions in the left-side navigation pane.

2. On the SQL Session page, click Create SQL Session.

3. On the Create SQL Session page, add the Spark configuration above.

4. In a Data Development SparkSQL task, use the hdfs://<ns_name>/ prefix directly in your SQL statements.

   -- Create table
   CREATE TABLE default.my_orc_table1 (
       id INT,
       name STRING,
       age INT
   )
   location "hdfs://<ns_name>/user/hive/warehouse/ads_user_info_1d_emr/my_orc_table1/";
   
   -- Insert data
   INSERT INTO table default.my_orc_table1(id, name, age)
   VALUES (1, 'Alice', 30);
   
   -- Query
   SELECT * FROM default.my_orc_table1;

   Replace <ns_name> with the custom namespace you set in the SetRootPolicy command.

   

Scenario 3: Batch job

1. Upload the SQL file. This example uses the sample SQL file test.sql.

   1. On the EMR Serverless Spark page, click Artifacts in the left-side navigation pane.

   2. On the Managed File Directory page, click Upload File.

   3. In the Upload File dialog box, click the upload area to select a local file, or drag and drop the file directly.

2. Create and configure a batch task.

   1. On the EMR Serverless Spark page, click Development in the left-side navigation pane.

   2. On the Catalog tab, click the (Create) icon.

   3. In the dialog box, enter a Name, select SQL under the Application (Batch) type, and then click OK.

   4. On the Task Configuration page, add the Spark configuration above.

3. Click Run. The results appear as shown below.