All Products
Search

This Product

    DataWorks:Data Studio: Associate an EMR computing resource

    Document Center

    DataWorks:Data Studio: Associate an EMR computing resource

    Last Updated:Jul 17, 2026

    Associate your E-MapReduce (EMR) cluster with DataWorks as an EMR computing resource to enable data synchronization, development, and management of EMR tasks.

    Prerequisites

    Limitations

    • Product limitations:

      • For EMR clusters with Kerberos authentication enabled, the security group must allow inbound UDP traffic from the vSwitch CIDR block associated with the resource group.

        Note

        Click the image icon next to Cluster Security Group in the Basic information section of the EMR cluster to go to the Security Group Details tab. Click Access Rule > Inbound, select Added Manually, set Protocol Type to Custom UDP. For the Port Range, check the KDC port in the /etc/krb5.conf file on the EMR cluster. Set Authorized object to the vSwitch CIDR block associated with the resource group.

      • To manage metadata in DataWorks for DataLake or Custom clusters, configure EMR-HOOK on the cluster side or when you configure Spark parameters. Without EMR-HOOK, DataWorks cannot display metadata in real time, generate audit logs, or display lineage, and EMR-related governance tasks cannot run. Only EMR Hive and EMR Spark SQL services support EMR-HOOK. For more information, see Configure EMR-HOOK for Hive and Configure EMR-HOOK for Spark SQL.

        Note

    • Region limitations: China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China (Chengdu), China (Hong Kong), Japan (Tokyo), Singapore, Malaysia (Kuala Lumpur), Indonesia (Jakarta), Germany (Frankfurt), UK (London), US (Silicon Valley), and US (Virginia).

    • Permission limitations:

      Operator

      Required permissions

      Alibaba Cloud account

      No additional authorization is required.

      RAM user or RAM role

      Only workspace members who are assigned the operator or workspace administrator role, or who have the AliyunDataWorksFullAccess permission, can create compute resources. For more information, see Grant a user the workspace administrator role.

    Notes

    • The following EMR versions of Hadoop clusters (legacy data lake) are supported in DataWorks:

      EMR-3.38.2, EMR-3.38.3, EMR-4.9.0, EMR-5.6.0, EMR-3.26.3, EMR-3.27.2, EMR-3.29.0, EMR-3.32.0, EMR-3.35.0, EMR-4.3.0, EMR-4.4.1, EMR-4.5.0, EMR-4.5.1, EMR-4.6.0, EMR-4.8.0, EMR-5.2.1, and EMR-5.4.3.

    • Hadoop clusters (legacy data lake) are no longer recommended. Migrate to DataLake clusters as soon as possible. For more information, see Migrate from a Hadoop cluster to a DataLake cluster.

    Go to the computing resource list page

    1. Log on to the DataWorks console. In the navigation pane on the left, switch to the target region and click More > Management Center. Select your workspace from the drop-down list and click Go to Management Center.

    2. In the navigation pane on the left, click Computing Resources to open the computing resource list page.

    Associate an EMR computing resource

    On the computing resource list page, configure and associate the EMR computing resource.

    1. Select the type of computing resource to associate.

      1. Click Associate Computing Resources to go to the Associate Computing Resources page.

      2. On the Associate Computing Resources page, set the computing resource type to EMR to go to the Associate EMR Computing Resource configuration page.

    2. Configure the EMR computing resource.

      On the Associate EMR Computing Resource configuration page, configure the following parameters.

      Parameter

      Description

      Alibaba Cloud Account to Which Cluster Belongs

      You can select Current Alibaba Cloud Account or Another Alibaba Cloud Account.

      Note

      When you select Another Alibaba Cloud Account, you can follow the instructions in Use an EMR cluster across Alibaba Cloud accounts to authorize the relevant accounts and then configure the required parameters as prompted.

      Cluster Type

      Select the cluster type based on your business requirements.

      Cluster

      Select the EMR cluster you want to use under the corresponding cluster type.

      Default Access Identity

      • Development environment: You can use the cluster account hadoop, or the cluster account mapped to the task executor .

      • Production environment: You can use the cluster account hadoop, or the cluster account mapped to the task owner, Alibaba Cloud account, or RAM user.

        Note

        When the default access identity is set to the cluster account mapped to the task owner, Alibaba Cloud account, or RAM user, see Configure account mapping between DataWorks and EMR to configure the mapping between DataWorks tenant members and EMR cluster accounts. EMR tasks run under the mapped cluster account. If no mapping is configured, DataWorks handles this as follows:

        • If a RAM user (sub-account) executes the task: DataWorks uses the EMR cluster system account with the same name as the current operator by default. If LDAP or Kerberos authentication is enabled for the cluster, the task fails.

        • If an Alibaba Cloud account executes the task: The DataWorks task returns an error.

      Pass Proxy User Information

      Whether to pass Proxy User information when tasks run on the EMR cluster.

      Note

      When LDAP, Kerberos, or another authentication method is enabled, the cluster issues an authentication credential for each regular user. You can use a super user (real user) to proxy regular users (proxy users) for authentication, so that proxy users access the cluster with the super user's credentials. You only need to add the user as a proxy user.

      • Pass: When tasks are run on the EMR cluster, data access permissions are verified and controlled based on the Proxy User.

        • Data Studio and Data Analysis: The Alibaba Cloud account name of the task executor is dynamically passed as the Proxy User information.

        • Operation Center: The Alibaba Cloud account name of the default access identity configured when the cluster was registered is passed as the fixed Proxy User information.

      • Do not pass: When tasks are run on the EMR cluster, data access permissions are verified and controlled based on the account authentication method configured when the cluster was registered.

      The Proxy User information is passed differently depending on the EMR task type:

      • EMR Kyuubi tasks: The information is passed through the hive.server2.proxy.user configuration item.

      • EMR Spark tasks and EMR Spark SQL tasks in non-JDBC mode: The information is passed through the -proxy-user configuration item.

      Configuration files

      When the cluster type is set to HADOOP, obtain the configuration files from the EMR console. For more information, see Export EMR cluster configuration files. After exporting, rename the files as prompted on the configuration page.

      You can also log on to the EMR cluster and obtain the configuration files from the following paths.

      /etc/ecm/hadoop-conf/core-site.xml
/etc/ecm/hadoop-conf/hdfs-site.xml
/etc/ecm/hadoop-conf/mapred-site.xml
/etc/ecm/hadoop-conf/yarn-site.xml
/etc/ecm/hive-conf/hive-site.xml
/etc/ecm/spark-conf/spark-defaults.conf
/etc/ecm/spark-conf/spark-env.sh

      Computing Resource Instance Name

      Customize the name of the computing resource instance. When a task runs, you can select the computing resource by name.

    3. Click Confirm to complete the EMR computing resource configuration.

    Resource group initialization

    After you register a cluster for the first time, modify the cluster service configuration, or upgrade a component version (for example, modify core-site.xml), initialize the resource group to ensure it can access the EMR cluster through network connectivity configuration.

    1. On the Computing Resources list page, find the EMR computing resource you created. Click Initialize Resource Group in the upper-right corner.

    2. Click Initialize next to the desired resource group. After the resource group is initialized, click Determine.

    (Optional) Configure YARN resource queues

    On the Computing Resources list page, find the EMR cluster you associated. On the YARN Resource Queue tab, click Edit YARN Resource Queue to configure YARN resource queues for tasks in different modules.

    (Optional) Configure Spark parameters

    Configure dedicated Spark property parameters for tasks in different modules.

    1. On the Computing Resources list page, find the EMR cluster you associated.

    2. Click the Spark-related Parameter tab and then click Edit Spark Parameters to go to the Spark parameter editing page for the EMR cluster.

    3. Click Add below a module, enter the Spark Property Name and the corresponding Spark Property Value to configure Spark property parameters for tasks in different modules.

    Next steps