E-MapReduce:Use EMR-CLI to deploy a custom gateway environment

Step 1: Create an ECS instance

Create an ECS instance in the ECS console. See Create an instance using the wizard.

The instance does not require public network access. Use the following settings:

Step 2: Create a RAM role for the gateway

1. Log on to the RAM console as a Resource Access Management (RAM) administrator.

2. In the left navigation pane, choose Identity Management > Roles.

3. On the Roles page, click Create Role.

4. In the Create Role panel, set Trusted Entity Type to Alibaba Cloud Service and Trusted Service to ECS, then click OK.

5. Enter a Role Name — for example, ECSForEMRGatewayRole — then click OK.

Step 3: Grant permissions to the RAM role

1. On the Permission Management tab, click Grant Permission.

2. In the Grant Permission panel, select the following policies from System Policy, then click OK:

   • AliyunEMRFullAccess

   • AliyunOSSFullAccess

   • AliyunDLFFullAccess

3. Click Close.

Step 4: Attach the RAM role to the ECS instance

1. Log on to the ECS console.

2. Log on to the ECS console.

3. In the left navigation pane, choose Instances & Images > Instances.

4. In the top navigation bar, select the region of your EMR cluster.

5. Find the ECS instance and choose > Instance Settings > Grant/Revoke RAM Role.

6. In the dialog box, select ECSForEMRGatewayRole and click OK.

Step 5: Connect to the ECS instance

Connect to the ECS instance. See Connect to an ECS instance.

Step 6: Install EMR-CLI

Run the following command to install EMR-CLI:

regionId=`curl http://100.100.100.200/latest/meta-data/region-id`; curl https://ecm-repo-${regionId}.oss-${regionId}-internal.aliyuncs.com/emrcli/emrcli.sh -o /tmp/emrcli.sh; chmod 755 /tmp/emrcli.sh; sh /tmp/emrcli.sh install ${regionId}

A successful installation returns:

install emrcli success

Step 7: Deploy the gateway client

emrcli gateway deploy \
  --clusterId <ClusterId> \
  --appNames <ApplicationName>

Replace the following placeholders:

A successful deployment returns:

deployGateway success

Step 8: Re-log on to the ECS instance

Log on to the ECS instance again to apply the updated environment variables.

Step 9 (optional): Configure domain name resolution

1. Add a zone. See Add a built-in authoritative domain name.

2. Add a DNS record. See Add a DNS record. Use the following settings:

   Parameter	Setting
   Record Type	Use the default value A
   Host Record	Enter the hostname of the gateway machine — run the hostname command to get it, for example, iZ2zea8r0aht2vzbqci****
   Record Value	Enter the private IP address of the gateway machine
   TTL Value	Use the default value

Update client components

The deploy command overwrites configurations for installed applications and incrementally installs new ones. To add a new service client — for example, Flink — to a gateway that already has HDFS and YARN:

# Add the FLINK client to an environment that already has HDFS and YARN
emrcli gateway deploy \
  --clusterId <ClusterId> \
  --appNames HDFS,YARN,FLINK

A successful update returns:

deployGateway success

Sync configurations from the EMR cluster

When service configurations change in the EMR cluster — for example, if you modify core-site.xml in the EMR console — sync the changes to the gateway:

emrcli gateway refreshConfigs \
  --clusterId <ClusterId> \
  --appNames <ApplicationName>   # Optional. Omit to sync all applications.

A successful sync returns:

refreshConfiguration success

Check the EMR-CLI version

emrcli version

Example output:

2.0.0

Upgrade EMR-CLI

To upgrade EMR-CLI to the latest version, repeat Step 6: Install EMR-CLI. The install command overwrites the existing version in place.