How to upgrade the EMR-HOOK component in an EMR gateway - Data Lake Formation

Background

You must manually upgrade the EMR-HOOK component if you run computing tasks in an EMR gateway that was custom-deployed with EMR-CLI, and you use Data Lake Formation (DLF) features such as the Data Overview access count metrics and Lifecycle Management data access frequency rules.

Note

The upgrade process does not affect running computing tasks. The changes automatically apply to new computing tasks started after the upgrade is complete.

Prerequisites

Your EMR cluster uses Data Lake Formation (DLF) for metadata management.
This procedure applies only to EMR gateways custom-deployed with EMR-CLI.

Procedure

EMR-5.10.1, EMR-3.44.1 and later

Step 1: Upgrade the JAR files

Log on to the gateway as root by using SSH. Replace ${region} with your region ID, such as cn-hangzhou, and then run the following script.

sudo mkdir -p /opt/apps/EMRHOOK/upgrade/
sudo wget https://dlf-repo-${region}.oss-${region}-internal.aliyuncs.com/emrhook/latest/emrhook.tar.gz -P /opt/apps/EMRHOOK/upgrade
sudo tar -p -zxf /opt/apps/EMRHOOK/upgrade/emrhook.tar.gz -C /opt/apps/EMRHOOK/upgrade/
sudo cp -p /opt/apps/EMRHOOK/upgrade/emrhook/* /opt/apps/EMRHOOK/emrhook-current/

Step 2: Modify the Hive configuration

Important

Select the correct ${hive-jar} based on your Hive version. Use hive-hook-hive23.jar for Hive 2 and hive-hook-hive31.jar for Hive 3.

Configuration file	Parameter	Value
hive-site.xml (/etc/taihao-apps/hive-conf/hive-site.xml)	`hive.aux.jars.path`	Append `,/opt/apps/EMRHOOK/emrhook-current/${hive-jar}` to the existing value. Note Use a comma (,) as the separator.
hive-site.xml (/etc/taihao-apps/hive-conf/hive-site.xml)	`hive.exec.post.hooks`	Append `,com.aliyun.emr.meta.hive.hook.LineageLoggerHook` to the existing value.
hive-env.sh (/etc/taihao-apps/hive-conf/hive-env.sh)	`hive_aux_jars_path`	Append `,/opt/apps/EMRHOOK/emrhook-current/${hive-jar}` to the existing value. Note Use a comma (,) as the separator.

Step 3: Modify the Spark configuration

Important

Select the correct ${spark-jar} based on your Spark version. Use spark-hook-spark24.jar for Spark 2 and spark-hook-spark30.jar for Spark 3.

Configuration file	Parameter	Value
spark-defaults.conf (/etc/taihao-apps/spark-conf/spark-defaults.conf)	`spark.driver.extraClassPath`	Append `:/opt/apps/EMRHOOK/emrhook-current/${spark-jar}` to the existing value. Note Use a colon (:) as the separator.
	`spark.executor.extraClassPath`	Append `:/opt/apps/EMRHOOK/emrhook-current/${spark-jar}` to the existing value. Note Use a colon (:) as the separator.
	`spark.sql.queryExecutionListeners`	Append `,com.aliyun.emr.meta.spark.listener.EMRQueryLogger` to the existing value.

EMR-5.10.1, EMR-3.44.1 and earlier

Step 1: Upgrade the JAR files

Log on to the gateway as root by using SSH. Replace ${region} with your region ID, such as cn-hangzhou.

Run the following script to download and extract the latest EMR-HOOK JAR files.

sudo mkdir -p /opt/apps/EMRHOOK/upgrade/
sudo wget https://dlf-repo-${region}.oss-${region}-internal.aliyuncs.com/emrhook/latest/emrhook.tar.gz -P /opt/apps/EMRHOOK/upgrade
sudo tar -p -zxf /opt/apps/EMRHOOK/upgrade/emrhook.tar.gz -C /opt/apps/EMRHOOK/upgrade/

Different E-MapReduce (EMR) releases may ship with different minor versions of the EMR-HOOK component. You must therefore rename the downloaded JAR files to match the version of the existing JAR files before replacing them.

For example, EMR-3.43.1 includes EMR-HOOK component version 1.1.4, and the JAR file naming convention is hive-hook-${version}-hive20.jar. You must rename the extracted JAR files to match this convention.

cd /opt/apps/EMRHOOK/upgrade/emrhook
mv hive-hook-hive20.jar hive-hook-1.1.4-hive20.jar
mv hive-hook-hive23.jar hive-hook-1.1.4-hive23.jar
mv hive-hook-hive31.jar hive-hook-1.1.4-hive31.jar
mv spark-hook-spark24.jar spark-hook-1.1.4-spark24.jar
mv spark-hook-spark30.jar spark-hook-1.1.4-spark30.jar

[root@master-1-1 emrhook-current]# ll
total 636
-rw-r--r-- 1 502 games 122460 Sep 22 2022 hive-hook-1.1.4-hive20.jar
-rw-r--r-- 1 502 games 122762 Sep 22 2022 hive-hook-1.1.4-hive23.jar
-rw-r--r-- 1 502 games 121183 Sep 22 2022 hive-hook-1.1.4-hive31.jar
-rw-r--r-- 1 502 games 158682 Sep 22 2022 spark-hook-1.1.4-spark24.jar
-rw-r--r-- 1 502 games 121298 Sep 22 2022 spark-hook-1.1.4-spark30.jar

After you rename the files, run the following command to replace the old files:

sudo cp -p /opt/apps/EMRHOOK/upgrade/emrhook/* /opt/apps/EMRHOOK/emrhook-current/

Step 2: Modify the Hive configuration

Important

For ${hive-jar}, select the JAR file that matches your Hive version: hive-hook-${emrhook-version}-hive23.jar for Hive 2, or hive-hook-${emrhook-version}-hive31.jar for Hive 3. Replace ${emrhook-version} with the version number of your EMR-HOOK component. For example, the full filename might be hive-hook-1.1.4-hive23.jar.

Configuration file	Parameter	Value
hive-site.xml (/etc/taihao-apps/hive-conf/hive-site.xml)	`hive.aux.jars.path`	Append `,/opt/apps/EMRHOOK/emrhook-current/${hive-jar}` to the existing value. Note Use a comma (,) as the separator.
hive-site.xml (/etc/taihao-apps/hive-conf/hive-site.xml)	`hive.exec.post.hooks`	Append `,com.aliyun.emr.meta.hive.hook.LineageLoggerHook` to the existing value.
hive-env.sh (/etc/taihao-apps/hive-conf/hive-env.sh)	`hive_aux_jars_path`	Append `,/opt/apps/EMRHOOK/emrhook-current/${hive-jar}` to the existing value. Note Use a comma (,) as the separator.

Step 3: Modify the Spark configuration

Important

For ${spark-jar}, select the JAR file that matches your Spark version: spark-hook-${emrhook-version}-spark24.jar for Spark 2, or spark-hook-${emrhook-version}-spark30.jar for Spark 3. Replace ${emrhook-version} with the version number of your EMR-HOOK component. For example, the full filename might be spark-hook-1.1.4-spark24.jar.

Configuration file	Parameter	Value
spark-defaults.conf (/etc/taihao-apps/spark-conf/spark-defaults.conf)	`spark.driver.extraClassPath`	Append `:/opt/apps/EMRHOOK/emrhook-current/${spark-jar}` to the existing value. Note Use a colon (:) as the separator.
	`spark.executor.extraClassPath`	Append `:/opt/apps/EMRHOOK/emrhook-current/${spark-jar}` to the existing value. Note Use a colon (:) as the separator.
	`spark.sql.queryExecutionListeners`	Append `,com.aliyun.emr.meta.spark.listener.EMRQueryLogger` to the existing value.