All Products
Search

This Product

    Dataphin:Set the compute engine of a Dataphin instance to Hadoop

    Document Center

    Dataphin:Set the compute engine of a Dataphin instance to Hadoop

    Last Updated:Jan 21, 2025

    Before initiating data development in a project space, it is necessary to configure the Dataphin instance's compute engine. Once configured, the system allows the addition of corresponding compute resources to the project space, thus providing essential computing and storage capabilities. This topic outlines the steps to set Hadoop as the compute engine in the Dataphin system.

    Prerequisites

    The system metadata initialization must be complete. For more information, see using Hadoop as the metadata warehouse compute engine for metadata warehouse initialization.

    Procedure

    1. Log on to the Dataphin console with a super administrator account.

    2. Navigate to Management Center > System Settings from the top menu bar on the Dataphin home page.

    3. In the Compute Settings section, select a Hadoop type compute engine and configure its parameters.

      Options for Hadoop-type compute engines include E-MapReduce 3.x Hadoop, E-MapReduce 5.x Hadoop, CDH 5.x Hadoop, CDH6.x Hadoop, Cloudera Data Platform 7.x, Huawei FusionInsight 8.x Hadoop, and AsiaInfo DP5.3 Hadoop.

      Hadoop, CDH6.x, Cloudera Data Platform 7.x, and Huawei FusionInsight 8.x

      Hadoop-based compute engines encompass E-MapReduce3.x Hadoop, E-MapReduce5.x Hadoop, CDH5.x Hadoop, CDH6.x Hadoop, Cloudera Data Platform 7.x, Huawei FusionInsight 8.x Hadoop, and AsiaInfo DP5.3 Hadoop.

      Note

      When the compute engine is selected as E-MapReduce3.x Hadoop, E-MapReduce5.x Hadoop, CDH5.x Hadoop, CDH6.x Hadoop, Cloudera Data Platform 7.x, AsiaInfo DP5.3 Hadoop, or Huawei FusionInsight 8.x Hadoop, you only need to select the offline compute engine type. No need to configure the following compute engine parameters. After saving the compute engine type, you can click Configure Compute Cluster to add or configure a Hadoop cluster on the Planning > Compute Source > Manage Hadoop Cluster page.

      Parameter

      Description

      NameNode

      Hadoop allows the addition of multiple HDFS connection URLs within the same cluster. Metadata acquisition is successful as long as one URL is verified. For example, host=192.xxx.xx.xxx,webUiPort=500xxx,ipcPort=80xx.

      Execution Engine

      Choose the compute execution engine that best fits your business needs.

    4. Click Save.

    What to do next

    Once the compute engine for the Dataphin instance is configured, the system enables the addition of corresponding compute resources to the newly created project space, thereby providing necessary computing and storage capabilities. For guidance on creating a project space and incorporating compute resources, see creating a general project.