Obtain cluster information and configure network - DataWorks

DataWorks connects to Cloudera’s Distribution Including Apache Hadoop (CDH) and Cloudera Data Platform (CDP) clusters. You can register a CDH or CDP cluster in DataWorks for data development and governance operations, such as task development, scheduling, metadata management in Data Map, and Data Quality checks. Before you register a cluster, you must obtain the required configuration information and establish network connectivity between the cluster and a resource group. This topic uses a CDH cluster as an example to show how to obtain cluster information and configure network connectivity.

Background information

CDH is Cloudera's open-source platform distribution. It includes out-of-the-box features like cluster management, monitoring, and diagnostics, and supports various components for running end-to-end big data workflows.
CDP is a public data platform that collects and consolidates customer data from multiple platforms. It gathers real-time data to create individual user data profiles.

You can register CDH and CDP clusters in DataWorks for task development, scheduling, metadata management in Data Map, and Data Quality checks.

Prerequisites

A CDH cluster is deployed.

DataWorks supports CDH deployed in environments other than Alibaba Cloud ECS, but you must ensure network connectivity between the deployment environment and an Alibaba Cloud VPC. You can typically use services like Express Connect or a VPN to establish this connection.
A new DataWorks serverless resource group (recommended) or a legacy exclusive resource group for scheduling is required.

By default, a DataWorks resource group is network-isolated from other cloud products. To connect to CDH, you must first establish network connectivity between the CDH cluster and the resource group.
Note
- A serverless resource group (recommended) is a general-purpose resource group that supports various task types, such as data synchronization and task scheduling. For purchasing details, see Use serverless resource groups. New users (those who have not activated any edition of DataWorks in the current region) can only purchase new-version resource groups.
- You can also use a legacy exclusive resource group for scheduling to run CDH or CDP tasks. For more information, see Use exclusive resource groups for scheduling.

Obtain CDH cluster configuration information

Follow these steps to obtain the CDH configuration information required to register the cluster in DataWorks.

Obtain the CDH version information.

Log on to Cloudera Manager. On the home page, find the CDH version information, such as Cluster 1 (CDH 6.1.1, Parcel), in the cluster panel on the left. Also, confirm the running status of services like HBase, HDFS, Hive, Hue, Impala, Oozie, Solr, Spark, YARN, and ZooKeeper.
Obtain the host and component address information needed to configure the cluster connection during registration.
Find information in Cloudera Manager

Log on to Cloudera Manager. In the top navigation bar, click Hosts > Roles. The Roles page groups hosts by their assigned roles. For example, the cdh-header-1-cn-shanghai host is assigned roles such as HMS, HS2, and RM, while the cdh-worker-[1-2]-cn-shanghai hosts are assigned the ID role.

Where:
- HS2: HiveServer2
- HMS: Hive Metastore
- ID: Impala Daemon
- RM: YARN ResourceManager
Obtain the configuration files to be uploaded during CDH cluster registration.
1. Log on to Cloudera Manager.
2. On the Status page, click the drop-down menu for the cluster and select View Client Configuration URLs.
3. Download the configuration package for the required service. For example, to download the YARN configuration, find YARN in the dialog box. The dialog box displays download links for client configurations of services such as YARN (MR2 Included), HBase, Solr, HDFS, and Hive. Click the download icon for the target service, such as YARN (MR2 Included), to get the client configuration file.
Obtain the CDH cluster's network information to configure its connection to the DataWorks resource group.
1. Log on to the ECS console where the CDH cluster is deployed.
2. In the instance list, find the ECS instance where the CDH cluster is deployed and click the instance name to open the Instance Details page. View and record the Security Group, VPC, and vSwitch information.

Configure network connectivity

Serverless resource group

This section uses a serverless resource group as an example to describe how to configure network connectivity between the resource group and the CDH cluster.

By default, DataWorks serverless resource groups are network-isolated from other cloud products. To connect to a CDH cluster, you must bind the resource group to the VPC where the cluster is located. This establishes network connectivity between the CDH cluster and the resource group.

Go to the network settings page for the serverless resource group.
1. Log on to the DataWorks console.
2. In the left-side navigation pane, click Resource Group. The Resource Groups page opens to the Exclusive Resource Group tab by default.
3. Find your resource group and click Network Settings in the Actions column.
Bind the VPC.

On the VPC Binding tab, in the Data Scheduling & Data Integration section, click Add Binding. On the configuration page, select the VPC, zone, and vSwitch of the CDH cluster that you recorded in Step 4 of Obtain CDH cluster configuration information.
Configure hosts.

Go to the Alibaba Cloud DNS console. Use Private DNS (PrivateZone) to authoritatively resolve the host addresses you recorded in Step 2 of Obtain CDH cluster configuration information.
1. Activate Private DNS. For details, see Activate Private DNS.
  
  Note
  You can skip this step if you have already activated Private DNS.
2. Add a built-in authoritative domain name. For details, see Add a built-in authoritative domain name.
  Note
  - This topic uses the hostname cdh-header-1-cn-shanghai obtained from Manually find the information in the Cloudera Manager UI as an example for authoritative resolution. Adjust the parameters based on your actual hostname configuration.
  - The resolved IP address is the private IP address of the ECS instance where your CDH cluster is deployed.
3. Set the scope for the zone. For details, see Set the Scope for a Zone.
  
  Note
  When setting the VPC scope for the zone, select the VPC to which the resource group is bound.

Exclusive scheduling resource group

This section explains how to configure network connectivity between the exclusive resource group for scheduling and the CDH cluster.

By default, a DataWorks exclusive resource group for scheduling is network-isolated from other cloud products. To connect to a CDH cluster, you must bind the resource group to the VPC where the cluster is located. This establishes network connectivity between the CDH cluster and the exclusive resource group for scheduling.

Go to the network settings page for the exclusive resource group.
1. Log on to the DataWorks console.
2. In the left-side navigation pane, click Resource Group. The Resource Groups page opens to the Exclusive Resource Group tab by default.
3. Find your exclusive resource group for scheduling and click Network Settings in the Actions column.
Bind the VPC.

On the VPC Binding tab, click Add Binding. On the configuration page, select the VPC, zone, vSwitch, and security group of the CDH cluster that you recorded in Step 4 of Obtain CDH cluster configuration information.
Configure hosts.

Click the Hostname-to-IP Mapping tab. Click Batch Edit and enter the host address information that you recorded in Step 2 of Obtain CDH cluster configuration information in the dialog box.

Next steps

After completing the preparations in this topic, you can register the CDH cluster in DataWorks to begin development. For more information, see DataStudio (legacy): Bind a CDH compute engine.