Initialize the metadata warehouse using Amazon EMR as the metadata warehouse engine - Dataphin

The Dataphin metadata warehouse manages internal business metadata and compute engine metadata. It resides in a Dataphin project within the metadata tenant (OPS tenant) and consists of periodic data integration nodes, SQL script nodes, and Shell nodes. Metadata warehouse initialization configures the compute engine type and initializes metadata. This topic describes how to initialize the metadata warehouse with Amazon EMR as the compute engine.

Limits

Only accounts with the metadata tenant super administrator or system administrator role can initialize the system.

Important

Keep the account and password of the metadata tenant super administrator or system administrator secure. Exercise caution when performing operations with the metadata tenant super administrator account.

Procedure

In the top menu bar of the Dataphin homepage, select Management Hub > System Settings.
In the navigation pane on the left, select System O&M > Metadata Warehouse Settings.
On the Metadata Warehouse Settings configuration wizard page, click Start.
In the Select Initialization Engine Type step, select the Amazon EMR engine type.

Important
If the metadata warehouse has already been initialized, the previously successful metadata warehouse is selected by default. Switching to an incompatible compute engine will cause the administration features to become unavailable.
Click Next.

On the Parameter Checking page, configure the following parameters.

Parameter	Description
Primary Node Public DNS	The public DNS of the primary node, used to obtain the VPC private DNS. Both Hive and Spark connect through the private DNS. The format is `ec2-{public_ip}.{region}.compute.amazonaws.com`.
*Key File (.pem)**	The key pair for accessing the primary node EC2 instance, as specified when creating the EMR cluster.
core-site.xml	Upload the cluster configuration files manually, or click Get Cluster Configuration to download them from the primary node. To use this option, first provide the primary node public DNS and upload the key file.
yarn-site.xml
hive-site.xml
hdfs-site.xml
Cluster Storage	Currently, only HDFS is supported.
Metadata Retrieval Method	Select HMS or Amazon Glue. HMS: Selected by default. Amazon Glue: If you select Amazon Glue, configure the Glue Region Code, Glue AccessKey ID, and Glue AccessKey Secret. Glue Region Code: Enter the Region Code for Amazon Glue, such as ap-northeast-3, us-east-1, or us-west-1. Glue AccessKey ID, Glue AccessKey Secret: Enter the AccessKey ID and AccessKey Secret for accessing Amazon Glue.
Engine Type	Select Hive or Spark. If you select Hive, enter the Hive JDBC URL. If you select Spark, enter the Spark JDBC URL. Hive JDBC URL: Enter the JDBC connection address for Hive, or click Automatically Retrieve to obtain it. To use this option, first provide the primary node public DNS and upload the key file. The format is `jdbc:hive2//host1:port1,host2:post2/`. The database name is not required. Spark JDBC URL: Enter the JDBC connection address for Spark. The format is `jdbc:hive2//host1:port1/` or `jdbc:kyuubi://host1:port1/`. The database name is not required.
Username	The username for Hive or Spark, used as the `username` for the JDBC connection.
Database	The database name for the Amazon EMR compute engine.
Metadata Production Project	The name of the metadata warehouse project in Dataphin, used for metadata production and processing.

Click Test Connection. After the connection test passes, click Next.
On the initialization page, click Start.

Note
System initialization takes approximately 15 minutes.
After the page indicates successful execution, click Finish to complete the configuration.

What to do next

After the metadata warehouse is initialized, configure the compute engine for the Dataphin instance. If the metadata warehouse engine is Amazon EMR, you can set the business tenant engine to any engine type except MaxCompute. For more information, see Compute settings.