E-MapReduce:Cluster management FAQ

Can I upgrade an EMR cluster?

No. You cannot upgrade an EMR cluster or its services. To use a newer version, release the existing cluster and create a new one.

What services do EMR clusters support?

The supported services vary by cluster type and version. For more information, see Release versions.

Can I add Zeppelin in the console?

No. You cannot add Zeppelin as a new service from the EMR console. To add Zeppelin, install it on any master node ECS instance. You can also manually install and maintain other components on the ECS instances. For information about the services that you can add to different types of clusters, see Add services.

Does EMR support Oozie and its alternatives?

The Oozie component is not included in EMR DataLake clusters that run EMR V5.8.0 or later, or EMR V3.42.0 or later. If you require a workflow scheduling service, you can use EMR Workflow. For more information, see What is EMR Workflow?.

Reason for three master nodes in HA clusters

Newer EMR high-availability clusters use three master nodes for greater reliability than a two-master-node setup. Two-master node configurations are no longer supported, and the master node group cannot be scaled in. For these clusters, EMR distributes the master nodes across different physical hosts to reduce the risk of failure.

Enabling data disk encryption and its effects

After you enable data disk encryption, data is encrypted both in transit and at rest. This feature helps meet security and compliance requirements. Data disk encryption is transparent to applications at the OS layer of ECS instances and does not affect job execution.

How to clean up a failed cluster

Cluster creation failures are usually caused by incorrect RDS configurations that lead to a deployment failure, or by insufficient inventory of some ECS instances.

If some ECS instances were created but the cluster status is Startup Failed, release the instances in the ECS console. After all instances are released, the EMR cluster is automatically released.

If the EMR deployment fails and the cluster status is Unexpectedly Terminated, no resources were created and no charges are incurred. You can click Delete in the Actions column for the cluster to remove it.

Adding services to an existing cluster

Yes. You can add services to a cluster after it is created. For more information, see Add services.

Service restart after configuration changes

Server-side configuration changes for services like Spark, Hive, and HDFS take effect only after you restart the services. Changes to client-side configurations take effect after you click Deploy Client Configuration, without requiring a service restart. For more information about how to modify or add configuration items, see Manage configuration items.

What is a rolling restart?

The rolling restart mechanism restarts ECS instances one by one. The next instance restarts only after the current one and all its services are fully restored. Each node restart takes about five minutes.

Associating a public IP to an existing cluster

You can request an Elastic IP address (EIP) and associate it with an ECS instance in a Virtual Private Cloud (VPC) that does not have a public IP address. This allows the ECS instance to be accessed over the internet. For more information, see Associate an EIP.

When to enable a deployment set

A deployment set is an ECS feature that controls the distribution strategy of ECS instances. We recommend that you enable the deployment set feature for core node groups that use instance types with local disks to improve data security. A deployment set prevents multiple ECS instances from being deployed on the same physical host. This avoids a single point of failure and helps prevent the loss of local HDFS data on EMR if a physical host fails.

Due to the limitations of ECS deployment sets, a maximum of 20 ECS instances can be added to a deployment set. For more information, see Enable a deployment set.

Specifying a deployment set during cluster scale-out

By default, deployment sets are enabled for instance types with local disks and disabled for other instance types. You can adjust this setting as needed. For instructions on how to enable a deployment set, see Enable a deployment set.

Specifying disk size during cluster scale-out

When you scale out a cluster, the node group's settings determine the disk size for new nodes. This size cannot be changed during the scale-out process. If needed, you can adjust the disk size of the node group. For instructions on how to expand a disk, see Expand a disk.

Can I expand or shrink disks?

You can only expand data disks. You cannot shrink data disks or resize system disks.

On the Nodes tab of the target cluster, click Expand Disk for the target node group to expand its data disks. For specific instructions, see Expand a disk.

Cluster scale-out and scale-in

Yes, but the scaling rules vary by node type:

• Scale-out: You can scale out only core and task node groups. The configuration of the new nodes is the same as that of the existing nodes by default. Before you scale out, make sure all related orders are paid. An unpaid order will cause the scale-out operation to fail. For specific instructions, see Scale out a cluster.

• Scale-in: The master node group does not support scale-in. The rules for other node groups vary by type:

  • For pay-as-you-go or preemptible instance task node groups and pay-as-you-go Gateway node groups, see Scale in a cluster.

  • For pay-as-you-go core node groups, subscription task node groups, and subscription core node groups, see Manually scale in a node group.

Changing ECS instance configuration in a node group

When changing the configuration of ECS instances in a node group, you can either upgrade or downgrade the ECS instance specification based on your needs.

• To upgrade the ECS instance specification, directly upgrade the ECS instance specification within the node group. For more information, see Upgrade node configurations.

• To downgrade the ECS instance specification:

  Important

  If you need to downgrade the configuration, proceed with caution based on the node group type and service dependencies.

  • If ZooKeeper is deployed on the core/worker node group, you cannot directly change the configuration of the core/worker node group.

    The ZooKeeper service is highly dependent on the node group. Changing the configuration may cause service failures or data inconsistencies that affect cluster stability.

  • If ZooKeeper is not deployed on the core/worker node group, you can change the configuration of the core/worker node group.

    You can add a new node group and adjust its configuration to complete the change. For more information, see Manage node groups and Manually scale in a node group.

Selecting multiple instance types for a node group

The rules for selecting instance types for a node group are as follows:

• Subscription: You can select only a single instance type.

• Pay-as-you-go or preemptible instance:

  If the node group is a task node group, you can select up to 10 alternative instance types of the same specification (same vCPU and memory). For other node group types, you cannot select multiple instance types.

Error: "AddNumber is not valid" during scale-out

• Symptom: You receive the error message The specified parameter AddNumber is not valid. add instances number :xxx larger than deploymentSet availableAmount: xxx deploymentSetId: ds-uf6gwfou0a13kekupt14xxxx when scaling out a cluster.

• Cause: This error indicates that the deployment set feature is enabled for your cluster, and the number of nodes in the node group has reached the deployment set limit. For more information about deployment sets, see Enable a deployment set.

• Solution: Contact ECS Support to request an increase in the deployment set quota for your account.

How do I stop collecting service logs?

If you do not want EMR to collect your data, you can disable the collection of service operational logs.

Procedure:

1. Disable service operational log collection.

   • During cluster creation: In the software configuration step, click Collect Service Operational Logs.

   • After cluster creation: On the Basic Information page of the target cluster, in the Software Information section, click Collection Status of Service Operational Logs.

2. Verify that collection is disabled.

   Check whether namenode-log exists in /usr/local/ilogtail/user_log_config.json. If it does not exist, service log collection is disabled.

   Note

   After you disable service log collection, it takes about two to three minutes for the configuration to be synchronized. Please wait patiently.

What information do service operational logs collect?

Service operational logs include only logs from the running service components of the cluster. You can enable or disable the collection of all service logs with a single click. Note that if you disable log collection, the cluster's health check feature and technical support will be limited.

Which cluster types support EMR Doctor?

Only DataLake and Hadoop cluster types support the health check feature. After a cluster is created, you can use this feature on the Monitoring and Diagnostics > Health Check tab for the target cluster in the EMR console.

If your Hadoop cluster does not have this feature, you must enable EMR Doctor. For more information, see Enable EMR Doctor (for Hadoop clusters).

Impact of EMR Doctor installation or upgrade

Installing or upgrading EMR Doctor does not restart any services or affect your existing jobs. After the installation, EMR Doctor automatically configures the necessary parameters in the existing cluster, so you do not need to perform manual configurations.

During the installation or upgrade, EMR Doctor deploys configurations for YARN, Spark, Tez, and Hive services. If you modified and saved some configurations but have not deployed them, make sure that the deployment process does not affect the services.

What data does EMR Doctor collect?

EMR Doctor does not collect your actual data, nor does it scan your files or file content.

EMR Doctor collects only necessary event data, such as job start and end times, metrics, and counters.

Is EMR Doctor free of charge?

Yes. EMR Doctor is currently free of charge.

How does data collection affect job execution?

EMR Doctor's storage metadata collection dynamically adjusts the resources used for collection based on user resources and does not consume excessive resources.

EMR Doctor's job collection uses Java probe technology and does not start a separate Java process for monitoring. Collection is performed asynchronously and does not block the main job process. If collection overhead becomes too high, EMR Doctor automatically discards data. You can also adjust parameters such as the collection frequency.

When are collection reports available?

After EMR Doctor is installed or upgraded, the daily report feature analyzes data based on the jobs that you run and whether storage metadata is collected. Therefore, the cluster must have an active workload.

• Compute jobs: After the compute jobs on the cluster are collected, the latest report is available the next day. The report provides a cluster assessment and recommendations based on an analysis of the previous day's job execution status.

• Storage analysis: EMR Doctor does not enable storage analysis by default. You can enable it manually. After you enable it, collection usually runs at about 10:00 AM. After collection is complete, analysis runs and a report is generated in the early morning of the next day. If you enable collection in the afternoon, you must wait until the third day to see the results.

Specific values for recommended configurations

EMR Doctor provides directional recommendations, such as reducing memory configuration or modifying GC parameters, but does not provide specific parameter values. This is because EMR Doctor uses point-in-time sampling for collection to minimize the impact on your program. You must test and validate all recommended configurations for your specific workloads.

Error: "Insufficient ECS inventory" during scale-out

• Symptom: The cluster fails to scale out, with a failure reason of "Insufficient ECS inventory_OutofStock" or "Insufficient ECS inventory_OperationDenied.NoStock".

• Cause: The ECS instance type for the node group that you want to scale out has insufficient inventory to fulfill your request.

• Solution: Wait until the required ECS instance type is in stock and then try to scale out again, or scale out by creating a new node group and selecting a different ECS instance type. For more information, see Create a node group.

Error: "Insufficient ECS inventory" during cluster creation

• Symptom: Creating a cluster or adding a node group fails, with a failure reason of "Insufficient ECS inventory_OutofStock" or "Insufficient ECS inventory_OperationDenied.NoStock".

• Cause: The ECS instance type that you selected for the cluster or node group has insufficient inventory.

• Solution: When you create the cluster, select another ECS instance type that has sufficient inventory and meets your business requirements.

How do I delete unneeded services?

You cannot delete existing services from a cluster. Once a service is started, you cannot delete it from the console or by using an API.

How to log on to a cluster node

After an EMR cluster is created, you can log on to the master node by using the password set during cluster creation. For information about how to log on to other nodes, see Log on to other nodes of a cluster.

How to view an instance's vSwitch

In EMR on ECS, vSwitch information is associated with node groups and cannot be viewed directly on the Basic Information page. Go to the Nodes page and click the name of the node group to which the instance belongs to view its associated vSwitch information.

Resolving packet loss on a large-scale cluster

• Symptom: Frequent network packet loss occurs in the cluster, and system logs may show error messages such as neighbour: arp_cache: neighbor table overflow!. This indicates that the Address Resolution Protocol (ARP) cache table is full and can no longer manage IP-to-MAC address mappings, leading to network performance issues.

• Cause: In large-scale distributed systems, especially when a single cluster exceeds 1,000 servers and runs a version earlier than EMR-5.18.0 or EMR-3.52.0 (exclusive), you may encounter network instability and packet loss. You can optimize ARP cache management by adjusting system parameters.

  The ARP cache stores mappings between IP addresses and MAC addresses. The key parameters are as follows:

  • net.ipv4.neigh.default.gc_thresh1: The minimum number of entries to keep in the ARP cache. Garbage collection is not performed if the number of entries is below this value. The default value is 128.

  • net.ipv4.neigh.default.gc_thresh2: The soft limit for the number of entries in the ARP cache. Garbage collection is performed within 5 seconds if the number of entries exceeds this value. The default value is 512.

  • net.ipv4.neigh.default.gc_thresh3: The hard limit for the number of entries in the ARP cache. The default value is 1024.

  Note

  The default values are too small for clusters with more than 1,000 nodes and can cause network packet loss and instability. Therefore, the parameters must be adjusted.

• Solution:

  1. Edit the /etc/sysctl.conf file and add the following content to increase the ARP cache capacity limit and optimize the maximum connection tracking value.

     net.ipv4.neigh.default.gc_thresh1 = 512
     net.ipv4.neigh.default.gc_thresh2 = 2048
     net.ipv4.neigh.default.gc_thresh3 = 10240
     net.nf_conntrack_max = 524288

  2. Run the sudo sysctl -p command to apply the new settings.

     Note

     If you encounter the error message sysctl: cannot stat /proc/sys/net/nf_conntrack_max: No such file or directory when you run the sysctl -p command, first run the sudo modprobe nf_conntrack command to load the corresponding module. Then, run the sysctl -p command again to update the configuration.

Handling a SystemMaintenance.Redeploy event

If you receive a system event of the Instance redeployment due to system maintenance (SystemMaintenance.Redeploy) type for a local disk instance, it indicates that Alibaba Cloud has detected potential software or hardware failure risks on the underlying host of the ECS instance. This risk requires the ECS instance to be redeployed. Do not click Redeploy in the ECS console directly to avoid data loss.

Solution:

1. Check the event details to identify the affected node.

2. In the node group that contains the faulty node, scale out to add a new node. For more information, see Scale out a cluster.

3. Scale in the faulty node.

   • To scale in a core node group or a subscription task node group, see Manually scale in a node group.

     Note

     When you release a subscription ECS instance, ECS calculates and displays the refund amount. If you have questions, submit a ticket and select Elastic Compute Service for Product.

   • To scale in a pay-as-you-go task node group, see Scale in a cluster.

Automatically adding cluster ID tags to cloud disks

To automatically tag the cloud disks of your EMR cluster's ECS instances with the cluster ID, enable tag inheritance in the Tag console.

Procedure:

1. Log on to the Tag console.

2. In the left-side navigation pane, choose Tags > Tag inheritance.

3. Read the instructions for enabling the feature and select the checkbox to create a service-linked role.

   When you enable tag inheritance, the system automatically creates a service-linked role named AliyunServiceRoleForTag to perform operations related to tag inheritance. For more information, see Service-linked role for Tag.

4. Click Enable and set rules.

5. Configure tag inheritance rules.

   For resources that support tag inheritance, specify the tag keys to inherit. You can choose to inherit all tag keys or only specific tag keys.

   

6. Click OK.

For more information about tag settings, see Tag inheritance.

Error: IdempotentParameterMismatch

• Symptom: You may encounter the following error message when you perform operations such as releasing a cluster or upgrading a configuration.

• Cause: The same client token was used in multiple requests.

  The request uses the same client token as a previous, but non-identical request. Do not reuse a client token with different requests, unless the requests are identical.

• Solution: Check whether your operation is already in progress. If so, do not submit it again. Otherwise, refresh the console page. The EMR console automatically generates a new client token.

Error: QuotaExceeded.PrivateIpAddress

• Symptom: You may encounter the following error message when you create or scale out a cluster.

  [QuotaExceeded.PrivateIpAddress] The specified VSwitch "vsw-xxxx" does not have enough IP addresses.

• Cause: The selected vSwitch does not have enough available private IP addresses to fulfill the cluster creation or scale-out request.

• Solution: Create a new node group and select a vSwitch that has a sufficient number of available IP addresses. Then, retry the cluster creation or scale-out operation.

Error: LostProxy

• Symptom: The "taihao-proxy disconnect" error occurs when you create a cluster, scale out a cluster, or update a service configuration.

• Cause: The EMR management agent (proxy) on a cluster node has lost its connection.

• Solution:

  1. Check the cluster status and fix node issues.

     • If multiple nodes are disconnected, check the CPU and memory metrics.

       • If CPU or memory utilization is high, the cluster is overloaded. Upgrade the configuration or scale out the cluster to reduce the pressure.

       • If CPU and memory utilization are low, check the security group configuration to ensure that network communication is normal.

     • If only a few nodes are disconnected, check the load on those nodes to determine whether CPU or memory utilization has reached 100%. If the load is too high, check for abnormal processes that are consuming resources. If you find any, terminate them and check whether the node status returns to normal. If no abnormal processes are found, consider the following solutions:

       • For a master node, investigate the processes that have high CPU consumption. You can upgrade the specification of the master node or add MASTER-EXTEND nodes to distribute the load.

       • For a non-master node, if a single ECS instance is overloaded or unresponsive, you can decommission the problematic node or add a new node.

         Log on to the node and run the following command to restart the service.

         service taihao-proxy restart

  2. After you complete the checks and operations, retry creating the cluster, scaling it out, or updating the service configuration.

Error: "Insufficient account balance"

• Symptom: You encounter the following error message when you create, scale out, or upgrade a cluster.

  InvalidAccountStatus.NotEnoughBalance Message: Your account does not have enough balance to order pay-as-you-go products. 

• Cause: Your account has an insufficient balance.

• Solution: Check your account balance and make sure it is sufficient to cover the cost of the required resources. After your balance is sufficient, retry the operation.

Error: QuotaExceed.DiskCapacity

• Symptom: You may encounter the following error message when you scale out a cluster or expand a disk.

  [QuotaExceed.DiskCapacity] The used capacity of disk type has exceeded the quota in the zone,  quota check fail.

• Cause: The disk quota for the instance has reached its limit.

• Solution: The used capacity of the specified disk type has exceeded the quota in the availability zone. Go to Quota Center to query and apply for an increase in your disk capacity quota.

Error: QuotaExceed.ElasticQuota

• Symptom: You may encounter the following error message when you create or scale out a cluster.

  QuotaExceed.ElasticQuota Message: The number of the specified ECS instances has exceeded the quota of the specified instance type. 

• Cause: The ECS instance quota has been reached.

• Solution: Select a different instance type or reduce the number of instances and try again. You can also request a quota increase in the ECS console or Quota Center.

Handling bootstrap action failures

Check the execution log of the failed bootstrap action in the operation history:

• If the log contains a clear error message, fix the bootstrap script based on the error message and retry the operation.

• If the log contains the keyword exitCode but no clear error, add more detailed logging to the bootstrap script for better debugging, and then retry the operation.

• If the task times out or there is no output in the log, check the following:

  • Ensure that the user has read permissions on the OSS bucket where the bootstrap script is located.

  • Check the ECS network configuration to ensure that it can access the OSS internal endpoint, and then retry the operation.

SparkContext initialization error from EMR Gateway