E-MapReduce (EMR) allows you to configure a custom ECS application role in the Advanced Settings section of the Basic Settings step when you create a cluster in the EMR console. You can use this role to access other cloud resources, such as Object Storage Service (OSS) and Log Service, in your Alibaba Cloud account in password-free mode. This topic describes how to create and use a custom ECS application role.

Background information

To manage access to external resources from an EMR cluster, you can attach policies to the custom ECS application role that is configured when you create the cluster. For example, you can impose the following limits:
  • A cluster can access only the OSS data directories that you specify.
  • A cluster can access the external resources that you specify.

Prerequisites

An OSS bucket is created in the OSS console. Make sure that the bucket resides in the region where you want to create an EMR cluster. For more information about how to create a bucket, see Create buckets.

Procedure

  1. Step 1: Create a policy
  2. Step 2: Create a RAM role
  3. Step 3: Create a cluster and access external resources

Step 1: Create a policy

  1. Go to the Create Policy page.
    1. Log on to the RAM console by using an Alibaba Cloud account.
    2. In the left-side navigation pane, choose Permissions > Policies.
    3. On the Policies page, click Create Policy.
  2. On the Create Policy page, click the JSON tab.
  3. Enter a policy document and click Next: Edit Basic Information.
    Enter the following policy document in the script editor:
    {
        "Version": "1",
        "Statement": [
            {
                "Action": [
                    "oss:GetObject",
                    "oss:ListObjects"
                ],
                "Resource": [
                    "acs:oss:*:*:emr-logs2",
                    "acs:oss:*:*:emr-logs2/*"
                ],
                "Effect": "Allow"
            }
        ]
    }
    Note Elements in the policy:
    • Action: the actions on resources. In this example, the permissions to read and query OSS data are granted.
    • Resource: the resources on which the permissions are granted. In this example, the OSS bucket emr-logs2 and the objects in this OSS bucket are used as the resources.
    For more information about the basic elements of a policy, see Policy elements.
  4. Enter a policy name.
    In this example, the policy name is test-emr.
  5. Click OK.

Step 2: Create a RAM role

  1. In the left-side navigation pane of the RAM console, choose Identities > Roles.
  2. On the Roles page, click Create Role.
  3. Create a RAM role.
    1. In the Create Role panel, select Alibaba Cloud Service.
    2. Click Next.
    3. In the Configure Role step, configure the parameters.
      Parameter Description
      RAM Role Name In this example, the policy name is test-emr.
      Select Trusted Service Select Elastic Compute Service from the drop-down list.
    4. Click OK.
  4. Optional:Change the trusted service.
    Notice If you want to create a cluster in EMR V5.X or later, a minor version later than EMR V4.5, or a minor version later than EMR V3.32, skip this step.
    1. On the Roles page, click the name of the RAM role that you created.
    2. Click the Trust Policy Management tab.
    3. Click Edit Trust Policy.
    4. Change ecs.aliyuncs.com to emr.aliyuncs.com.
      Policy
    5. Click OK.
  5. Add the required permissions.
    1. On the Roles page, find the RAM role that you created and click Add Permissions in the Actions column.
    2. In the Add Permissions panel, click Custom Policy and select the policy that you created in Step 1.
    3. Click OK.
    4. Click Complete.

Step 3: Create a cluster and access external resources

  1. Log on to the Alibaba Cloud EMR console.
  2. In the top navigation bar, select the region where your cluster resides and select a resource group based on your business requirements.
  3. Click Cluster Wizard to create a cluster. In the Advanced Settings section of the Basic Settings step, enter the name of the RAM role that you created in Step 2: Create a RAM role in the ECS Role field. For more information about how to create a cluster, see Create a cluster.
    ECS_authority
  4. After the cluster is created, log on to the master node of the cluster by using SSH. For more information, see Log on to a cluster.
    Run the following command to check whether you can access the OSS bucket that is specified in the policy:
    hdfs dfs -ls oss://<yourBucketName>/
    Note Replace <yourBucketName> in the command with the name of the OSS bucket.
    • If the following information is returned, access to the OSS bucket is denied. No permission
    • If the following information is returned, access to the OSS bucket is allowed. emr-logs2

FAQ

  • Q: What do I do if the NoPermission message appears when I create a cluster?
    A: Perform the following operations in sequence to resolve the issue:
    1. Check whether the RAM user that you used to create the cluster is authorized to create clusters and change ECS application roles. If the RAM user is granted the AliyunEMRDevelopAccess permission, you must change the permission to AliyunEMRFullAccess.
    2. Check whether the name of the ECS application role that you specified when you create the cluster is valid.
    3. Check whether the trust policy is changed to emr.aliyuncs.com.
  • Q: What do I do if I cannot access an OSS bucket from Hadoop Distributed File System (HDFS)? hdfs_oss
    A: Perform the following operations in sequence to resolve the issue:
    1. Check whether the OSS bucket that you want to access resides in the same region as your cluster. If the OSS bucket and your cluster do not reside in the same region, you must add the endpoint of the OSS bucket to the access links.
    2. Check whether the OSS bucket that you want to access is one of the resources specified in the created policy. If the OSS bucket is not one of the resources, modify the policy.
    3. Check whether the related permissions on the OSS bucket are configured in the OSS console. If the permissions are configured, you must cancel the settings in the OSS console and specify Action in the policy to configure the permissions.