DataWorks:Register a cross-account EMR cluster

Prerequisites

• You have created two Alibaba Cloud accounts, referred to as Account A and Account B. For more information, see Register an account.

  • Alibaba Cloud account A: The account used to bind the EMR cluster from Account B in DataWorks.

  • Alibaba Cloud account B: The account that owns the EMR cluster.

• You have created an EMR cluster in Alibaba Cloud account B. For more information, see Create a cluster.

Limitations

• Cross-account binding is supported only for Hadoop clusters that do not use DLF to manage metadata.

• Kerberos authentication is not supported.

• For Spark SQL tasks, only table lineage is supported. Field lineage is not supported.

Account B: Create a RAM role and grant access

In Account B, create a RAM role with a trust policy that allows Account A to assume the role. This gives Account A access to the EMR resources in Account B.

1. Create a RAM role.

   Log on to the RAM console with Account B to create a RAM role. Specify Account A as the trusted account to allow it to assume the role and access authorized resources. For detailed instructions, see Create a RAM role for a trusted Alibaba Cloud account.

   The following are the key parameter settings:

   • RAM Role Name: Enter EMRRole.

   • Select Trusted Alibaba Cloud Account: Select Other Alibaba Cloud Account. In the account ID field, enter the ID of Account A. To find the ID, log on to the console with Account A and hover over the profile avatar in the top navigation bar.

   After the configuration is complete, Account A can assume the EMRRole role to access the authorized resources.

2. Modify the trust policy.

   On the details page for the EMRRole role, update the trust policy to grant Account A access to the EMR cluster in Account B. For more information, see Modify the trust policy of a RAM role. Use the following policy:

   {
     "Statement": [
       {
         "Action": "sts:AssumeRole",
         "Effect": "Allow",
         "Principal": {
           "Service": [
             "<UID of Account A>@emr.dataworks.aliyuncs.com"
           ]
         }
       }
     ],
     "Version": "1"
   }

   Note

   Replace <UID of Account A> with the UID of Alibaba Cloud account A.

3. Grant the AliyunDataWorksAccessingEMRReadOnlyPolicy permission to the EMRRole role.

   In the RAM console, go to the details page of the EMRRole role. On the Permissions tab, click Grant Permission. Search for and select the AliyunDataWorksAccessingEMRReadOnlyPolicy policy, and then click OK.

Account A: Register the cross-account cluster

1. Log on to the DataWorks console. In the target region, click More > Management Center in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Management Center.

2. In the left-side navigation pane, click Computing Resources.

3. Configure the cluster information.

   1. Configure basic information.

      Workspaces in standard mode require you to bind computing resources for the development and production environments separately. Follow the on-screen instructions to configure the cluster. For more information about workspace modes, see Differences between workspace modes.

      For Cloud Account for Cluster, select Other Alibaba Cloud Account. Upload the seven configuration files (core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml, hive-site.xml, spark-defaults.conf, and spark-env.sh). Set the Default Access Identity and Pass Proxy User Information. Enter a Compute Resource Instance Name and a Description, and then click OK.

      Key parameters are described as follows:

      • UID of Other Account: Enter the UID of Account B, which owns the EMR cluster. You must obtain the UID from the owner of Account B.

      • RAM Role of Other Account: Enter the name of the RAM role that you created in Account B. In this example, the name is EMRRole.

      • EMR Cluster of Other Account: Select the EMR cluster that you want to bind to DataWorks. You can select only Hadoop clusters of version 3.38.3 or 3.38.2 that do not use DLF to manage metadata.

      For more information about how to bind a cluster, see DataStudio: Bind EMR computing resources.

   2. Resource group initialization

      Initialize the resource group when you bind EMR computing resources for the first time, change cluster service configurations, or upgrade components (for example, by modifying core-site.xml). This process ensures that the resource group can access the EMR cluster after you configure network connectivity.

      Important

      • If the initialization fails, use the connectivity diagnosis tool to troubleshoot the cause.

      • Initialization can cause running tasks to fail. Unless immediate re-initialization is necessary, for example, to prevent widespread task failures after a configuration change, we recommend initializing the resource group during off-peak hours.

Next steps

After registering the cluster in DataWorks, you can perform the following operations:

• Configure cluster identity mapping: If the default access identity of the EMR cluster is not a Hadoop account, configure cluster identity mapping. This prevents RAM users from accessing unauthorized DataWorks resources.

• Use Data Integration to run data synchronization tasks on the cluster. For more information, see Data Integration.

• Go to Operation Center and Data Map to view more details about the cluster. For more information, see Operation Center and Data Map.