Data Lake Formation:Access DLF Data Using Files on EMR on ECS

Step 1: Grant RAM permissions to the EMR role

Skip this step after EMR productization integration is complete.

1. Log on to the Resource Access Management (RAM) console as an Alibaba Cloud account or RAM administrator.

2. Click Identity Management > Roles. Find the AliyunECSInstanceForEMRRole role.

3. In the Actions column, click Add Authorization.

4. In the Access Policy field, search for and select AliyunDLFFullAccess, then click Confirm Authorization.

   

Step 2: Grant catalog permissions to the EMR role

1. Log on to the Data Lake Formation console.

2. On the Catalog list page, click the catalog name to open the catalog details page.

3. Click the Permissions tab, then click Authorize.

4. On the Authorize page, configure the following parameters and click OK.

   If AliyunECSInstanceForEMRRole is not in the drop-down list, go to the user management page and click Synchronize.

   Parameter	Value
   Principal	Select RAM-User/RAM-Role
   Select Principal	Select AliyunECSInstanceForEMRRole from the drop-down list
   Preset Permission Type	Select Data Editor

Step 1: Upload packages to OSS

Upload the four JAR files to an OSS bucket and set the file permissions to public-read. For details, see Simple upload.

Step 2: Create and upload the installation script

Create a shell script with the following content and upload it to OSS.

#!/bin/bash

echo 'prepare paimon-vfs in hadoop classpath'
cd /opt/apps/JINDOSDK/jindosdk-current/lib
rm -rf paimon-*
wget ${paimon_vfs_jar}
wget ${paimon_jindo_jar}
wget ${paimon_bundle_jar}

echo 'prepare paimon-spark in spark classpath'
rm -rf /opt/apps/PAIMON/paimon-dlf
rm -rf /opt/apps/PAIMON/paimon-dlf.tar.gz.*
mkdir -p /opt/apps/PAIMON/paimon-dlf/lib/spark3
cd /opt/apps/PAIMON/paimon-dlf/lib/spark3
wget ${paimon_spark_jar}
rm -f /opt/apps/PAIMON/paimon-current
ln -sf /opt/apps/PAIMON/paimon-dlf /opt/apps/PAIMON/paimon-current

Step 3: Run the script as a bootstrap action on the cluster

Run the script as a bootstrap action of the EMR cluster. For details, see Manually run a script.

1. In the EMR console, go to your cluster page and choose Script Operations > Manual Execution. Click Create and Execute.

2. Configure the following parameters and click OK.

   Parameter	Value
   Name	Enter a custom name for the script
   Script Location	Select the script you uploaded to OSS. The path must follow the oss://**/**.sh format
   Execution Scope	Select Cluster

Step 4: Restart Spark and Hive

After the script finishes, restart the Spark and Hive services for the changes to take effect.

Access using Hadoop shell

Upload a CSV file to the Object Table and verify the upload.

# Upload the CSV file
echo "James,Sales,3000" >> employee.csv
hadoop fs -put employee.csv pvfs://catalog_name/default/object_table/

# Query all files in the object_table
hadoop fs -ls pvfs://catalog_name/default/object_table/

# View the CSV file
hadoop fs -cat pvfs://catalog_name/default/object_table/employee.csv

Access using Hive

Create a temporary table mapped to the Object Table path and query it.

-- Connect to Hive
beeline -u jdbc:hive2://localhost:10000

-- Create a temporary table to map the CSV file
CREATE TEMPORARY TABLE temp_table (
    employee_name STRING,
    department STRING,
    salary INT
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS TEXTFILE
LOCATION 'pvfs://catalog_name/default/object_table/';

-- Query the temporary table
SELECT * FROM temp_table;

Access using Spark

Create a temporary view mapped to the Object Table path and query it.

-- Start spark-sql
spark-sql

-- Create a temporary view to map the CSV file
CREATE TEMPORARY VIEW temp_table
USING csv
OPTIONS (
path 'pvfs://catalog_name/default/object_table',
inferSchema 'false',
schema 'employee_name STRING, department STRING, salary INT'
);

-- Query the temporary view
SELECT * FROM temp_table;