Data Studio: Associate an EMR computing resource

Prerequisites

• A DataWorks workspace is created, and the RAM user who performs the operation is added to the workspace and assigned the Workspace Administrator role.

• An EMR cluster is created.

  • Supportedcluster types:

    • DataLake cluster (new data lake): EMR on ECS

    • Custom cluster: EMR on ECS

    • Hadoop cluster (legacy data lake): EMR on ECS

    • Spark cluster: EMR on ACK

  • You can associate this computing resource only with a workspace that uses Use Data Studio (New Version).

    Note

    For workspaces that do not use Use Data Studio (New Version), you can associate the resource in Clusters. For more information, see Associate an EMR computing resource (legacy version).

• A resource group is associated with the workspace, and network connectivity is in place.

  • If you use a Serverless resource group, ensure that the EMR computing resource can connect to the Serverless resource group.

  • If you use a legacy exclusive resource group, ensure that the EMR computing resource can connect to the exclusive scheduling resource group for the corresponding use case.

Limitations

• Product limitations:

  • For EMR clusters with Kerberos authentication enabled, you must add an inbound rule to the security group of the cluster to allow UDP traffic from the CIDR block of the vSwitch that is associated with the resource group.

    Note

    Click the icon next to Cluster Security Group in the Basic information section of the EMR cluster to open the Security Group Details tab. On the Inbound tab of the Access Rule page, click Added Manually. Set Protocol Type to Custom UDP. For the Port Range, check the KDC port in the /etc/krb5.conf file on the EMR cluster. Set Authorized object to the CIDR block of the vSwitch that is associated with the resource group.

  • To manage metadata for a DataLake or custom cluster in DataWorks, you must configure EMR-HOOK on the cluster or when you set Spark parameters. Without EMR-HOOK, DataWorks cannot display real-time metadata, generate audit logs, or show data lineage, and EMR-related governance tasks cannot run. Currently, only the EMR Hive and EMR Spark SQL services support EMR-HOOK. For more information, see Configure EMR-HOOK for Hive and Configure EMR-HOOK for Spark SQL.

    Note

    • You can configure EMR-HOOK for Hive in the E-MapReduce console. You do not need to reinitialize the resource group after the configuration.

    • You can configure EMR-HOOK for Spark SQL in one of the following ways:

      • Configure it in the E-MapReduce console. You must re-initialize the resource group after the configuration.

      • Configure it by setting Spark properties in the computing resource. You do not need to re-initialize the resource group.

• Region limitations: China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China (Chengdu), China (Hong Kong), Japan (Tokyo), Singapore, Malaysia (Kuala Lumpur), Indonesia (Jakarta), Germany (Frankfurt), UK (London), US (Silicon Valley), and US (Virginia).

• Permission limitations:

  Operator	Required permissions
  Alibaba Cloud account	No additional permissions are required.
  RAM user/RAM role	Only workspace members with the O&M and Workspace Administrator roles, or members with the AliyunDataWorksFullAccess permission, can create computing resources. For more information, see Grant a user the permissions of a workspace administrator.

Usage notes

• DataWorks supports Hadoop clusters (legacy data lake) of the following EMR versions:

  EMR-3.38.2, EMR-3.38.3, EMR-4.9.0, EMR-5.6.0, EMR-3.26.3, EMR-3.27.2, EMR-3.29.0, EMR-3.32.0, EMR-3.35.0, EMR-4.3.0, EMR-4.4.1, EMR-4.5.0, EMR-4.5.1, EMR-4.6.0, EMR-4.8.0, EMR-5.2.1, EMR-5.4.3.

• Hadoop clusters (legacy data lake) are deprecated. We recommend that you migrate to DataLake clusters as soon as possible. For more information, see Migrate from a Hadoop cluster to a DataLake cluster.

Go to the computing resource list page

1. Log on to the DataWorks console. In the navigation pane on the left, switch to the target region and click More > Management Center. Select your workspace from the drop-down list and click Go to Management Center.

2. In the navigation pane on the left, click Computing Resources to open the computing resource list page.

Associate an EMR computing resource

On the Computing Resources page, configure and associate an EMR computing resource.

1. Select the computing resource type.

   1. Click Associate Computing Resources to go to the Associate Computing Resources page.

   2. On the Associate Computing Resources page, select EMR as the computing resource type to open the Bind EMR Computing Resource configuration page.

2. Configure the EMR computing resource.

   On the Associate EMR Computing Resource configuration page, configure the following parameters:

   Parameter	Description
   Alibaba Cloud Account to Which Cluster Belongs	You can select Current Alibaba Cloud Account or Another Alibaba Cloud Account. Note If you select Another Alibaba Cloud Account, follow the instructions in Scenario: Register a cross-account EMR cluster to grant the required permissions, and then enter the parameters as prompted on the page.
   Cluster Type	Select the cluster type that you want to use based on your business requirements. Note Supported cluster types: DataLake cluster (new data lake): EMR on ECS Custom cluster: EMR on ECS Hadoop cluster (legacy data lake): EMR on ECS Spark cluster: EMR on ACK
   Cluster	Select the EMR cluster that you want to use under the corresponding cluster type.
   Default Access Identity	development environment: You can use the cluster account hadoop or the cluster account that is mapped to the task executor. production environment: You can use the cluster account hadoop, or the cluster account that is mapped to the task owner, an Alibaba Cloud account, or a RAM user. Note If you select a default access identity that is mapped to a task owner, an Alibaba Cloud account, or a RAM user, you can manually configure the mapping between DataWorks users and EMR cluster accounts. For more information, see Configure cluster identity mapping. If this mapping is not configured, DataWorks handles task execution as follows: If a RAM user runs the task: By default, the task is run as an EMR cluster system account that has the same name as the RAM user. If the cluster has LDAP or Kerberos authentication enabled, the task fails. If an Alibaba Cloud account runs the task: The DataWorks task fails.
   Pass Proxy User Information	Specifies whether to pass proxy user information. Note When an authentication method such as LDAP or Kerberos is enabled, the cluster issues a credential to each standard user. To centrally manage permissions, a superuser (real user) can act on behalf of standard users (proxy users). When you access the cluster as a proxy user, the credentials of the real user are used for authentication. You only need to add the user as a proxy user. Pass: When tasks are run on the EMR cluster, data access permissions are verified and controlled based on the proxy user. Data Studio and Data Analysis: The Alibaba Cloud account name of the task executor is dynamically passed as the proxy user information. Operation Center: The Alibaba Cloud account name of the default access identity that is configured during cluster registration is always passed as the proxy user information. Do Not Pass: When tasks are run on the EMR cluster, data access permissions are verified and controlled based on the authentication method that is configured during cluster registration. The method for passing proxy user information varies based on the EMR task type: EMR Kyuubi tasks: passed by using the hive.server2.proxy.user parameter. EMR Spark tasks and EMR Spark SQL tasks in non-JDBC mode: passed by using the -proxy-user parameter.
   Configuration file	If you set Cluster type to HADOOP, go to the EMR console to obtain the configuration files. For more information, see Export and import service configurations. After you export the files, rename them based on the upload requirements on the page. Alternatively, you can log on to the EMR cluster and obtain the files from the following paths: /etc/ecm/hadoop-conf/core-site.xml /etc/ecm/hadoop-conf/hdfs-site.xml /etc/ecm/hadoop-conf/mapred-site.xml /etc/ecm/hadoop-conf/yarn-site.xml /etc/ecm/hive-conf/hive-site.xml /etc/ecm/spark-conf/spark-defaults.conf /etc/ecm/spark-conf/spark-env.sh
   Computing Resource Instance Name	Enter a custom name for the computing resource instance. You can select the computing resource by this name when you run a task.

3. Click Confirm to complete the configuration.

Initialize resource group

Initialize the resource group when you register a cluster for the first time, change cluster service configurations, or upgrade component versions (for example, after you modify the core-site.xml file). This ensures that the resource group can access the EMR cluster after you configure network connectivity.

1. On the Computing Resources page, find the EMR computing resource that you created. In the upper-right corner, click Initialize Resource Group.

2. Find the required resource group and click Initialize next to it. After the resource group is initialized, click Determine.

(Optional) Configure YARN resource queue

On the Computing Resources page, find the associated EMR cluster. On the YARN Resource Queue tab, click Edit YARN Resource Queue to set a global YARN resource queue for tasks in different modules.

(Optional) Set Spark parameters

Set dedicated Spark properties for tasks in different modules.

1. On the Computing Resources page, find the associated EMR cluster.

2. On the Spark-related Parameter tab, click Edit Spark Parameters to open the page for editing Spark parameters for the EMR cluster.

3. By clicking the Add button below a module and entering a Spark Property Name and its corresponding Spark Property Value, you can set global Spark parameters for the task in different modules.

Next steps

• Configure Kyuubi connection information: To use a custom account and password to run Kyuubi tasks, follow this document to customize the Kyuubi connection information.

• After you configure the EMR computing resource, you can use EMR-related nodes in Data Studio to develop tasks.