DataWorks:DataStudio (legacy): Register a CDH computing resource

Background information

• CDH is Cloudera's open-source platform distribution. It includes out-of-the-box features like cluster management, monitoring, and diagnostics, and supports various components for running end-to-end big data workflows.

• CDP is a public data platform that collects and consolidates customer data from multiple platforms. It gathers real-time data to create individual user data profiles.

You can register CDH and CDP clusters in DataWorks for task development, scheduling, metadata management in Data Map, and Data Quality checks.

Prerequisites

• To register a CDH or CDP cluster, you must have one of the following roles or permissions in the current workspace:

  • You are using an Alibaba Cloud account.

  • You are a workspace member with the Workspace Administrator role. For more information, see Add workspace members and manage their roles and permissions.

  • You are a workspace member with the AliyunDataWorksFullAccess policy. To grant this policy, see Grant permissions to a RAM user and Grant permissions to a RAM role. To add a member to the workspace, see Add workspace members and manage their roles and permissions.

• You have deployed the CDH or CDP cluster and obtained the configuration information required to register the cluster. For more information, see Preparations: Obtain CDH or CDP cluster information and configure network connectivity.

Limits

• You can run CDH or CDP cluster tasks only using the new serverless resource group (recommended) or a legacy exclusive resource group for scheduling.

  Note

  • A serverless resource group is a general-purpose resource group that can be used for various scenarios, such as data synchronization and task scheduling. For more information about how to purchase a serverless resource group, see Use a serverless resource group. If you have already purchased a legacy exclusive resource group for scheduling, you can also use it to run CDH or CDP tasks. For more information, see Use an exclusive resource group for scheduling.

  • New users can purchase only new serverless resource groups.

  • If you use a cluster of a Custom Version to register with DataWorks, you can use only a legacy exclusive resource group for scheduling. For more information about cluster versions, see Step 2: Register a CDH or CDP cluster.

• You can register CDH or CDP clusters only in the following regions: China (Beijing), China (Shanghai), China (Hangzhou), China (Shenzhen), China (Zhangjiakou), China (Chengdu), and Germany (Frankfurt).

Step 1: Go to the registration page

1. Log on to the DataWorks console. In the target region, click More > Management Center in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Management Center.

2. In the left-side navigation pane, click Clusters to go to the Clusters page. Click Register Cluster, select CDH for Open Source Cluster Type, and go to the cluster registration page.

Step 2: Register a CDH or CDP cluster

1. Configure basic cluster information.

   Parameter	Description
   Display Name of Cluster	The display name of the cluster in DataWorks. The name must be unique.
   Cluster Version	Select the version of the cluster that you want to register. DataWorks provides the following versions: CDH 5.16.2, CDH 6.1.1, CDH 6.2.1, CDH 6.3.2, and CDP 7.1.7. The component versions associated with these cluster versions are fixed. For more information, see Cluster connection information. If these cluster versions do not meet your business requirements, you can select Custom Version and configure component versions based on your needs. Note Refer to the UI for the most current information. If you use a cluster of a Custom Version to register with DataWorks, you can use only a legacy exclusive resource group for scheduling. After the registration is complete, you must submit a ticket to request that technical support initialize the environment.
   Cluster Name	Specifies the source of the configuration information for the cluster that you are registering. You can select a cluster that is already registered in another workspace or create a new cluster. Registered cluster: Reuses the configuration of a cluster that is already registered in another workspace. New cluster: You must configure the information for the cluster that you are registering.

2. Configure cluster connection information.

   Based on your use case, select the component versions for your cluster and enter the component endpoint information you obtained. For more information, see Preparations: Obtain CDH or CDP cluster information and configure network connectivity. On the head node of the CDH cluster, run the java -jar dw-tools.jar admin admin command to obtain the host and URL information for each cluster component. Then, use the output connection addresses to fill in the following fields: HiveServer2 (format: jdbc:hive2://<host>:<port>/<database>), Metastore (format: thrift://<host>:<port>), Impala JDBC URL (format: jdbc:impala://<host>:<port>/<schema>), Yarn.ResourceManager.Address (format: http://<host>:<port>), Jobhistory.Webapp.Address (change the port of the Yarn ResourceManager address to 8088), and Presto JDBC URL (format: jdbc:presto://<host>:<port>/<catalog>/<schema>). Presto is not a default CDH component. Specify the access address based on your deployment.

   Note

   If you access CDH components by domain name from a serverless resource group, you must configure authoritative DNS resolution for the CDH component domain names in Alibaba Cloud DNS PrivateZone. For more information, see Add a built-in authoritative domain and Set a scope for a domain name.

3. Add cluster configuration files.

   You can upload configuration files for the required components. To obtain the configuration files, see Preparations: Obtain CDH or CDP cluster information and configure network connectivity.

   The following table describes the configuration files.

   Configuration file	Description	Scenarios
   core-site.xml	Contains global configurations for the Hadoop Core library, such as common I/O settings for HDFS and MapReduce.	This file must be uploaded to run Spark or MapReduce tasks.
   hdfs-site.xml	Contains HDFS-related configurations, such as data block size, number of replicas, and path names.
   mapred-site.xml	Used to configure MapReduce-related parameters, such as the execution method and scheduling behavior of MapReduce jobs.	This file must be uploaded to run MapReduce tasks.
   yarn-site.xml	Contains all configurations related to YARN daemons, such as the resource manager, node manager, and application runtime environment.	This file must be uploaded to run Spark or MapReduce tasks, or when Kerberos is selected as the identity mapping type.
   hive-site.xml	Contains various parameters for configuring Hive, such as database connection information, Hive Metastore settings, and the execution engine.	This file must be uploaded when Kerberos is selected as the identity mapping type.
   spark-defaults.conf	Specifies the default configurations applied when a Spark job is executed. You can use the spark-defaults.conf file to pre-set properties, such as memory size and the number of CPU cores. Spark applications use these settings at runtime.	This file must be uploaded to run Spark tasks.
   config.properties	Contains configurations for the Presto server, such as global properties for coordinator and worker nodes in a Presto cluster.	This file must be uploaded when you use the Presto component and select OPEN LDAP or Kerberos as the identity mapping type.
   presto.jks	A Java KeyStore (JKS) file that stores security certificates, including private keys and public key certificates issued to applications. In Presto, the presto.jks file is used to enable SSL/TLS encrypted communication for Presto processes and secure data transfer.

4. Configure the default access identity.

   This setting specifies the account used to access the CDH cluster when tasks are run from DataWorks. The supported account types vary by environment.

   Note

   When you register a cluster, if you set the Default Access Identity to a non-cluster account and no identity mapping is configured (or if the mapping type is set to no authentication), all tasks fail.

   Environment	Default access identity	Related information
   Development environment	Cluster account: A specified cluster account is used to access the CDH cluster, regardless of who runs the CDH task in DataWorks, such as an Alibaba Cloud account or a RAM user with only development permissions. Mapped account: When a user runs a CDH task, you must configure an identity mapping between the user's account and a cluster account. When the task runs, the mapped account is used to access the CDH cluster.	To configure an identity mapping, see Configure cluster identity mappings.
   Production environment	Cluster account: A specified cluster account is used to access the CDH cluster, regardless of who runs the CDH task in DataWorks, such as the task owner, an Alibaba Cloud account, or a RAM user. Mapped account: When the task owner, an Alibaba Cloud account, or a RAM user runs a CDH task, you must configure an identity mapping between the corresponding account and a cluster account. When the task runs, the mapped account is used to access the CDH cluster.

5. Click Complete Registration to register the cluster in DataWorks.

Step 3: Initialize the resource group

You must initialize the resource group when you first register a cluster or after you change the cluster service configuration, or after you upgrade a component version, for example, by modifying the core-site.xml file. Initialization ensures that the resource group can access the CDH cluster and is correctly configured to run CDH cluster tasks. On the Clusters page, find the registered CDH cluster and click Initialize Resource Group in the upper-right corner. Select the required resource group and initialize it.

(Optional) Configure a YARN resource queue

The primary purpose of a YARN resource queue is to partition and isolate cluster resources. This ensures that different types of tasks can use computing resources in a reasonable and fair manner, preventing interference. To set up a dedicated YARN resource queue for tasks in a specific module, find your registered CDH cluster on the Clusters page. On the YARN Resource Queue tab, click Edit YARN resource queue to configure the settings.

(Optional) Configure Spark properties

Set dedicated Spark properties for tasks in different modules.

1. On the Clusters page, find your registered CDH cluster.

2. On the Spark-related Parameter tab, click the Edit Spark properties button to go to the page for editing Spark properties for the CDH cluster.

3. Click the Add button under a module, and enter the Spark Property Name and the corresponding Spark Property Value to set the Spark properties.

Next steps

• Configure cluster identity mappings: If the default access identity for the CDH cluster is not a specific cluster account (meaning access is through your Alibaba Cloud account), you must configure an identity mapping between the Alibaba Cloud account and a cluster account. This allows the Alibaba Cloud account to access the CDH cluster by using the mapped cluster identity, which enables data permission isolation and control.

• After registering the CDH computing resource, you can use CDH-related nodes in DataStudio to perform data development operations.