E-MapReduce:Hive extension: Record data lineage and access history

Prerequisites

Create a DataLake cluster or custom cluster and select the Hive service. For more information, see Create a cluster.

Limitations

• EMR-HOOK does not support collecting SQL information from jobs in a gateway environment custom-deployed with EMR-CLI.

• In EMR versions earlier than 5.16.0 and 3.50.0, the hive.exec.post.hooks (for Hive) and spark.sql.queryExecutionListeners (for Spark) parameter settings are not synchronized to gateway nodes. In EMR versions 5.16.0 and later and 3.50.0 and later, these settings are synchronized to gateway nodes. These versions also introduce the hive_aux_jars_path_gateway_only parameter, which lets you independently use custom extension JAR files on gateway nodes.

Usage notes

• EMR-HOOK is enabled by default in EMR versions earlier than 5.14.0 and 3.48.0.

  For custom clusters of EMR 3.44 where EMR-HOOK is not enabled by default, you can manually enable it. For more information, see the FAQ section.

• EMR-HOOK is disabled by default in EMR versions 5.14.0 and later and 3.48.0 and later. You must enable it manually.

Procedure

1. Go to the Cluster services page.

   1. Log on to the E-MapReduce console.

   2. In the top navigation bar, select a region and a resource group.

   3. On the EMR on ECS page, find your cluster and click Services in the Actions column.

2. Configure EMR-HOOK.

   1. On the Services page, in the Hive service section, click Configure.

   2. On the Configure page, add or modify the following parameters.

      Tab	Parameter	Description
      hive-site.xml	hive.exec.post.hooks	Specifies the hook that listens for SQL statements from the Hive service. This hook collects information for data lineage and access frequency. To enable EMR-HOOK, set this parameter to com.aliyun.emr.meta.hive.hook.LineageLoggerHook. To disable EMR-HOOK, leave this parameter empty.
      	dlf.emrhook.webtracking	Specifies whether to enable access frequency reporting. Valid values: true: Enabled. false: Disabled.
      hivemetastore-site.xml	hive.metastore.event.listeners	Specifies the listener for events related to Hive metadata changes. This listener collects information for data lineage. To enable EMR-HOOK, set this parameter to com.aliyun.emr.meta.hive.listener.MetaStoreListener. To disable EMR-HOOK, leave this parameter empty.
      	hive.metastore.pre.event.listeners	Specifies the listener for events that occur before Hive metadata changes. This listener collects information for data lineage. To enable EMR-HOOK, set this parameter to com.aliyun.emr.meta.hive.listener.MetaStorePreAuditListener. To disable EMR-HOOK, leave this parameter empty.

      Note

      If you disable EMR-HOOK, the Data Overview page for tables in the Data Lake Formation (DLF) console will no longer display data for Access Count (Today), Access Count (Last 7 Days), and Access Count (Last 30 Days).

   3. Save the configuration.

      1. On the Configure page, click Save.

      2. In the dialog box that appears, enter an Execution Reason and click Save.

3. Restart Hive.

   1. On the Configure page, choose More > Restart.

   2. In the dialog box that appears, enter an Execution Reason and click OK.

   3. In the Confirm dialog box, click OK.

4. View the data overview and data lineage.

   • View the Data Overview in Data Lake Formation (DLF). For more information, see Data overview of data tables.

   • View data lineage in DataWorks. For more information, see Data lineage analysis.

FAQ

Related topics

To learn how to configure EMR-HOOK for the Spark service, see SparkSQL: Use an extension to record data lineage and access history.