E-MapReduce:Use EMR-CLI to deploy a custom gateway environment

Gateway deployment options

A Gateway is an isolation layer for job submission provided by EMR. Its core benefits include:

• Decouple client workloads from core cluster services

  Offloads client operations such as spark-submit, hive -f, and yarn application from the master nodes.

• Enable isolation in a multi-tenant environment

  Supports separate runtime environments for users or departments.

• Improve cluster stability and maintainability

  Prevents frequent job submissions, script debugging, environment conflicts, or resource contention from affecting critical services like YARN ResourceManager and HDFS NameNode.

EMR offers three Gateway options to suit different cluster types, versions, and architectural requirements.

Prerequisites

An E-MapReduce compute cluster for DataLake, Dataflow, OLAP, or Custom scenarios must be running. For instructions on creating a cluster, see Create a cluster.

Limitations

• Cluster type: This solution applies only to deploying a gateway environment for DataLake, Dataflow, OLAP, and Custom clusters. If the cluster type and version are compatible, using a gateway node group is recommended.

  For information about deploying a gateway environment for existing Hadoop and Kafka clusters, see Create a gateway cluster.

  Note

  Only Alibaba Cloud accounts that created Hadoop or Kafka clusters before 17:00 (UTC+8) on December 19, 2022, can continue to create them.

• Overwrite installation: EMR-CLI deploys gateway clients in overwrite mode. Redeploying on an ECS instance with an existing gateway overwrites the old client and installs the new client in the same directory.

• Isolated deployment: Do not use an existing ECS instance from your EMR cluster (such as a Master, Core, or Task node) as the gateway node. This prevents the client environment from interfering with normal cluster operations.

• Supported services: This deployment method supports clients for the following services: HDFS, YARN, HBase, Hive, Spark 2, Spark 3, JindoSDK, Flink, Sqoop, Impala, Presto, Hudi, Iceberg, Tez, and Delta Lake.

First-time deployment

1. On the ECS console, create an instance. For more information, see Create an instance by using the wizard.

   Note

   The ECS instance does not require public network access.

   The following parameter settings are recommended.

   Parameter	Description
   Region and zone	Same region and zone as the EMR cluster.
   Image	Same operating system as the EMR cluster's instances.
   System disk	An ESSD of at least 60 GiB is recommended.
   Network	Same Virtual Private Cloud (VPC) as the EMR cluster.
   Security group	Use the same security group as the EMR cluster's master node group. This ensures network connectivity between the ECS instance and the EMR cluster.

2. Create a dedicated ECS RAM role for the EMR gateway.

   1. Log on to the RAM console as a RAM administrator.

   2. In the left-side navigation pane, choose Identities > Role.

   3. On the Role page, click Create Role.

   4. In the Create Role panel, set Principal Type to Cloud Service and Principal Name to Elastic Compute Service, and then click OK.

   5. Enter a Role Name (for example, ECSForEMRGatewayRole), and then click OK.

3. Grant permissions to the RAM role.

   1. On the Permission Settings tab, click Add Authorization.

   2. In the Add Authorization panel, select System Policy. Search for and select AliyunEMRFullAccess, AliyunOSSFullAccess, and AliyunDLFFullAccess. Then, click OK.

   3. Click Close.

4. Attach the RAM role to the ECS instance.

   1. Log on to the ECS console.

   2. In the left navigation bar, select Instances & Images > Instance.

   3. In the top menu bar, select the region.

   4. Find the newly created ECS instance and choose > Instance Settings > Attach/Detach RAM Role.

   5. In the dialog box that appears, select the ECSForEMRGatewayRole role and click OK.

5. Connect to the ECS instance. For more information, see Connect to an instance.

6. Run the following command to install EMR-CLI.

   regionId=`curl http://100.100.100.200/latest/meta-data/region-id`; curl https://ecm-repo-${regionId}.oss-${regionId}-internal.aliyuncs.com/emrcli/emrcli.sh -o /tmp/emrcli.sh; chmod 755 /tmp/emrcli.sh; sh /tmp/emrcli.sh install ${regionId}

   A successful installation returns the following message:

   install emrcli success

7. Run the following command to deploy the EMR gateway client.

   emrcli gateway deploy \
     --clusterId <ClusterId> \
     --appNames <ApplicationName>

   Replace the following parameters with your actual values.

   Parameter	Required	Description
   clusterId	Yes	The E-MapReduce cluster ID.
   appNames	No	The application names. Separate multiple names with a comma (,), for example, HDFS,YARN. If omitted, clients for all supported applications in the cluster are installed by default.

   A successful deployment returns the following message:

   deployGateway success

   Important

   After the gateway is installed, the JAVA_HOME system environment variable is set to /usr/lib/jvm/java-1.8.0. You can modify it in the /etc/profile.d/emr_env.sh file, but this may affect gateway functionality. Proceed with caution.

8. Log in to the ECS instance again to apply the new environment variables.

9. Optional: Configure DNS resolution for the gateway node.

   Important

   This step is required if the gateway includes the Spark service.

   1. Add a zone. For more information, see Add a built-in authoritative zone.

   2. Add a DNS record. For more information, see Add DNS records.

      The following table describes the required parameters.

      Parameter	Description
      Record type	Use the default value A.
      Hostname	Enter the hostname of the gateway node, for example, iZ2zea8r0aht2vzbqci****. You can obtain the hostname by running the hostname command.
      Record value	Enter the internal IP address of the gateway node. You can view this address on the node management page.
      TTL value	Use the default value.

Gateway management

After the gateway is created, you can run commands to update client components or synchronize configurations when services are added or modified in the associated cluster.

# Example: Add the FLINK client to an existing HDFS and YARN setup
emrcli gateway deploy \
  --clusterId <ClusterId> \
  --appNames HDFS,YARN,FLINK

deployGateway success

# Run the synchronization command
emrcli gateway refreshConfigs \
  --clusterId <ClusterId> \
  --appNames <ApplicationName> # Optional. Specify the applications to synchronize.

refreshConfiguration success

EMR-CLI management

emrcli version

2.0.0

FAQ

Q: How do I switch compute clusters?

A: To switch compute clusters, follow these steps:

1. To prevent data loss, use the mv command to manually back up files from the old cluster. Back up the /opt/apps directory, the /etc/taihao-apps directory, and the /etc/profile.d/yarn.sh file.

2. Follow the steps in this topic to redeploy the gateway environment for the new cluster.