Dataphin:Create TDH Inceptor compute source

Prerequisites

• The following requirements must be met if you use TDH Inceptor as the metadata warehouse or as the metadatabase retrieval method during metadata warehouse init:

  • A project named dataphin_meta is created in TDH Inceptor.

  • The user configured for TDH Inceptor during metadata warehouse init must have permissions to write data to tables and create tables in the dataphin_meta project.

  • If you use TDH Inceptor as the metadatabase retrieval method during metadata warehouse init, the user in the Inceptor metadata configuration must have permissions to write data to tables and create tables in the dataphin_meta project.

    

Limits

• Dataphin does not support registering user-defined functions (UDFs): Dataphin does not allow you to register UDFs when using the TDH Inceptor DPI engine. This is because if you add a JAR package with a duplicate name to register a UDF, the Inceptor service may stop and fail to restart. Similarly, if you add a JAR package with a different name that contains the same class file, the UDF execution result may be unpredictable. To add a UDF, use the TDH Inceptor client. Ensure that the UDF name is unique and the class name is consistent across the cluster.

• When you retrieve metadata from a MySQL metadatabase or Hive Metastore (HMS), you cannot retrieve the following information:

  • Data volume information for the asset overview, data plates, and projects.

  • Table data volume, partition data volume, and number of partition records in the asset directory.

  • Storage-related metric information in resource administration.

  • Data volume and number of records for dim_dataphin_table and dim_dataphin_partition in the shared model of the metadata warehouse.

• When you retrieve metadata from the TDH Inceptor System library, you cannot retrieve the following information:

  • Number of partition records in the asset directory.

  • Number of records for dim_dataphin_table and dim_dataphin_partition in the shared model of the metadata warehouse.

• The TDH Inceptor DPI engine does not support setting node priorities.

  To assign different resources to nodes with different priorities, you can set different usernames for different priority queues. After you set different priorities for Inceptor SQL nodes, Dataphin submits these nodes to the TDH Inceptor DPI engine using the corresponding users. The users that you set must have the Submit permission on the Inceptor resource queue. For more information, see Inceptor User Manual.

Procedure

1. In the top menu bar of the Dataphin home page, choose Planning > Compute Source.

2. On the Compute Source page, click Add Compute Source and then select TDH Inceptor Compute Source.

3. On the Create TDH Inceptor Compute Source page, configure the parameters.

   1. In the Basic Information section, configure the parameters.

      Parameter	Description
      Compute Source Type	Select the compute source type as TDH Inceptor.
      Compute Source Name	The naming convention is as follows: The name can contain only Chinese characters, digits, letters, underscores (_), and hyphens (-). The name can be up to 64 characters long.
      Compute Source Description	A brief description of the compute source. The description can be up to 128 characters long.

   2. In the Cluster Basic Information section, configure the parameters.

      Parameter	Description
      nameNode	Defaults to the NameNode value configured during system initialization. This value cannot be changed.
      Configuration File	Upload the HDFS configuration file. The file must include core-site.xml, hdfs-site.xml, hive-site.xml, yarn-site.xml, and other configuration files. You can contact Transwarp O&M engineers or log on to the Transwarp cluster O&M interface, then choose HDFS Service > More Operations > Download Service Configuration to get the configuration file.
      Authentication Type	If the TDH Inceptor cluster uses Kerberos authentication, select Kerberos as the authentication method. Kerberos is a symmetric-key-based authentication protocol that supports single sign-on (SSO), allowing an authenticated client to access multiple services such as HBase and HDFS. After you select Kerberos authentication, upload a Krb5 authentication file or configure the KDC Server address: Krb5 authentication file: Upload the Krb5 file for Kerberos authentication. KDC Server address: The address of the KDC server, which helps complete Kerberos authentication. You can configure multiple KDC Server addresses. Separate them with commas (,).

   3. In the HDFS Connection Information section, configure the parameters.

      Parameter	Description
      Execution Username, Password	The username and password for the compute execution machine. These credentials are used to run MapReduce nodes and to read from and write to HDFS. Important Make sure that you have the permissions to submit MapReduce nodes.
      Authentication Type	If HDFS uses Kerberos authentication, select Kerberos as the authentication method. Kerberos is a symmetric-key-based authentication protocol that supports SSO, allowing an authenticated client to access multiple services such as HBase and HDFS. After you select Kerberos authentication, upload a Keytab File authentication file and configure the Principal address: Keytab File: Upload the Keytab File for Kerberos authentication. Principal: The corresponding Kerberos authentication username. If you select no authentication, configure the username to access HDFS.

   4. In the Inceptor Configuration section, configure the parameters.

      Parameter	Description
      JDBC URL	Configure the HiveServer2 endpoint. The format is jdbc:hive2://{endpoint}:{port}/{database_name}.
      Authentication Type	Select the authentication type for Inceptor. The available options are No Authentication, LDAP, and Kerberos. No Authentication: No authentication is required. LDAP: Configure the username and password for access. Kerberos: Upload the HDFS Kerberos authentication file and configure the Hive Principal.
      Execution User For Development Environment Tasks	The execution username for nodes in the development environment.
      Execution User For Periodic Scheduling Tasks	The execution username for periodic scheduling nodes.
      Priority Task Queue	Select how to assign users for priority tasks. You can select Use Default Execution User or Custom. If you select Custom, you must configure the usernames for executing nodes with different priorities. Note Priority queues allocate resources by creating different YARN queues on the Hadoop cluster. Nodes are submitted to the corresponding YARN queue based on their priority.

   5. In the Inceptor Metadata Connection Information section, configure the parameters.

      Parameter	Description
      Metadata Retrieval Method	Supports Metadatabase and HMS as metadata retrieval methods. The required configuration varies based on the method you select. For the Metadatabase method, configure: Database type, Database version (only for MySQL), JDBC URL, Authentication type, Username, and Password (only for MySQL). For the HMS method, configure: Authentication type. Note If you select the Metadatabase retrieval method, first upload the core-site.xml and hdfs-site.xml configuration files. If you select the HMS retrieval method, first upload the core-site.xml, hdfs-site.xml, and hive-site.xml configuration files.
      Database Type	Select Inceptor or MySQL. If you select MySQL, you must also select a database Version. Valid values include MySQL 5.6/5.7, MySQL 8, and MySQL 5.1.43.
      JDBC URL	Enter the metadatabase endpoint. If you select MySQL as the database type, use the format jdbc:mysql://{endpoint}:{port}/{database_name}. If you select Inceptor, use the format jdbc:postgresql://{endpoint}:{port}/{database_name}.
      Authentication Type	Select No Authentication or LDAP.
      Username, Password	Enter the username and password for the metadatabase.

4. Click Test Connection to verify the connection to the compute source.

5. After the connection is successfully tested, click Submit.