E-MapReduce:FAQ

What does a stateful cluster restart constitute?

A stateful cluster restart constitutes a ResourceManager restart and a NodeManager restart. ResourceManager maintains the basic information about and status of applications. NodeManager maintains the information about and status of containers that are running. ResourceManager and NodeManager constantly synchronize their own status to external storage systems such as ZooKeeper, LevelDB, and Hadoop Distributed File System (HDFS). The status of ResourceManager and NodeManager can be automatically reloaded and recovered after they are restarted. This ensures that the status of applications and containers can be automatically recovered after the cluster is upgraded or restarted. In most cases, the upgrade or restart of a cluster is imperceptible to applications and containers.

How do I enable ResourceManager high availability (HA)?

How do I configure a hot update?

1. Configure key parameters.

   You can check or configure parameters related to a hot update on the Configure tab of the YARN service page of a cluster in the EMR console. The following table describes the parameters.

   Parameter	Description	Recommended value
   yarn.scheduler.configuration.store.class	The type of the backing store. If you set this parameter to fs, a file system is used as the backing store.	fs
   yarn.scheduler.configuration.max.version	The maximum number of configuration files that can be stored in the file system. Excess configuration files are automatically deleted if the number of configuration files exceeds the value of this parameter.	100
   yarn.scheduler.configuration.fs.path	The path in which the capacity-scheduler.xml file is stored. If you do not configure this parameter, a storage path is automatically created. If no prefix is specified, the relative path of the default file system is used as the storage path.	/yarn/<Cluster name>/scheduler/conf Important Replace <Cluster name> with a specific cluster name. Multiple clusters for which the YARN service is deployed may use the same distributed storage.

2. View configurations of the capacity-scheduler.xml file.

   • Method 1 (RESTful API): Access a URL in the following format: http://<rm-address>/ws/v1/cluster/scheduler-conf.

   • Method 2 (HDFS): Access the configuration path ${yarn.scheduler.configuration.fs.path}/capacity-scheduler.xml.<timestamp> to view configurations of the capacity-scheduler.xml file. <timestamp> indicates the time at which the capacity-scheduler.xml file is generated. The capacity-scheduler.xml file that has the largest timestamp value is the latest configuration file.

3. Update the configurations.

   For example, you can modify the parameter yarn.scheduler.capacity.maximum-am-resource-percent and delete the parameter yarn.scheduler.capacity.xxx. To delete a parameter, you need to only remove the value field of the parameter.

   curl -X PUT -H "Content-type: application/json" 'http://<rm-address>/ws/v1/cluster/scheduler-conf' -d '
   {
     "global-updates": [
       {
         "entry": [{
           "key":"yarn.scheduler.capacity.maximum-am-resource-percent",
           "value":"0.2"
         },{
           "key":"yarn.scheduler.capacity.xxx"
         }]
       }
     ]
   }'

How do I handle the uneven distribution of resources among applications in a queue?

View queue resource usage

To view queue resource usage, check Used Capacity on the YARN UI. This value shows the percentage of a queue's resources currently in use, calculated as the higher of memory and vCore usage.

1. Access the YARN UI. For details, see Access the web UIs of open source components.

2. On the All Applications page, click the ID of the target job.

3. Click the queue in the Queue row.

   The Application Queues section displays the queue resource usage.

   The queue details include parameters such as Queue State, Used Capacity, Configured Capacity, Configured Max Capacity, Effective Capacity, Absolute Used Capacity, and Used Resources. The Used Capacity parameter indicates current resource usage, for example, <memory:896, vCores:1> (10.9%).

YARN .out log file accumulation

• Cause: Some Hadoop dependency libraries use Java Logging APIs to generate logs, bypassing log4j rotation settings. The standard error (stderr) from these daemons is redirected to .out files. Without automatic cleanup, these files accumulate and can fill up the data disk.

• Solution: Use the head and tail commands with log timestamps to check whether these Java API logs are consuming excessive disk space. These logs are typically at the INFO level and do not affect component functionality. To prevent disk exhaustion, you can disable this logging.

  For example, to optimize the Timeline Server by disabling logging from the Jersey dependency library, follow these steps:

  1. Run the following command to monitor the .out log files related to hadoop-timelineserver- in the YARN log path. The log path is /var/log/emr/yarn/ for a DataLake cluster and /mnt/disk1/log/hadoop-yarn for a Hadoop cluster.

     tail /var/log/emr/yarn/*-hadoop-timelineserver-*.out

     The log output shows records from the com.sun.jersey component.

     xxx                     com.sun.jersey.server.wadl.generators.AbstractWadlGeneratorGrammarGenerator attachTypes
     INFO: Couldn't find grammar element for class javax.ws.rs.core.Response
     xxx                     com.sun.jersey.server.wadl.generators.AbstractWadlGeneratorGrammarGenerator attachTypes
     INFO: Couldn't find grammar element for class javax.ws.rs.core.Response
     xxx                     com.sun.jersey.server.wadl.generators.AbstractWadlGeneratorGrammarGenerator attachTypes
     INFO: Couldn't find grammar element for class javax.ws.rs.core.Response
                             com.sun.jersey.server.wadl.generators.AbstractWadlGeneratorGrammarGenerator attachTypes
     INFO: Couldn't find grammar element for class javax.ws.rs.core.Response
     xxx                     com.sun.jersey.server.wadl.generators.AbstractWadlGeneratorGrammarGenerator attachTypes
     INFO: Couldn't find grammar element for class javax.ws.rs.core.Response
     xxx                     com.sun.jersey.server.wadl.generators.AbstractWadlGeneratorGrammarGenerator attachTypes
     INFO: Couldn't find grammar element for class javax.ws.rs.core.Response

  2. To disable INFO-level logs from the Jersey library, create a configuration file. On the EMR node that hosts the Timeline Server, run the following command as the root user to create the file.

     sudo su root -c "echo 'com.sun.jersey.level = OFF' > $HADOOP_CONF_DIR/off-logging.properties"

  3. In the EMR console, go to the Configure tab for the YARN service. Find the YARN_TIMELINESERVER_OPTS parameter (or yarn_timelineserver_opts for a Hadoop cluster) and append -Djava.util.logging.config.file=off-logging.properties to its value.

  4. Save the configuration and restart the Timeline Server for the change to take effect. Monitor the service. If the Timeline Server starts successfully and its .out log no longer shows messages from com.sun.jersey, you have successfully disabled the logging.

How do I check whether the ResourceManager service is normal?

How do I obtain the status of an application?

Application troubleshooting

1. Check the application state on the application details page or by using the application REST API.

   • Application state not found. Possible causes include:

     • The client process terminated before submitting the application to YARN. This may indicate an issue with a client-side component, such as BRS or flowagent. Check the client submission logs.

     • The client cannot connect to the YARN ResourceManager. Verify that the ResourceManager address is correct and that there are no network issues. A network problem can result in the following error on the client: com.aliyun.emr.flow.agent.common.exceptions.EmrFlowException: ###[E40001,RESOURCE_MANAGER]: Failed to access to resource manager, cause: The stream is closed.

   • NEW_SAVING: In this state, the application information is being written to the ZooKeeper state store. If the application remains in this state, possible causes include:

     • An issue with ZooKeeper. Check if the ZooKeeper service is running properly.

     • A problem with reading data from or writing data to ZooKeeper. For the solution, see What do I do if the ResourceManager is in the Standby state and cannot automatically switch to the Active state?

   • SUBMITTED: This state is rare. A possible cause is that the Capacity Scheduler is blocked due to lock contention from too many node update requests. This typically occurs in large-scale clusters and requires process optimization. For a related case, see YARN-9618.

   • ACCEPTED: Check the Diagnostics. Take the appropriate action based on the message provided.

     • Error message: "Queue's AM resource limit exceeded."

       • Possible cause: The sum of used ApplicationMaster (AM) resources and the requested AM resources exceeds the queue's limit for AM resources. The condition checked in the UI is: ${Used Application Master Resources} + ${AM Resource Request} < ${Max Application Master Resources}.

       • Solution: Increase the AM resource limit for the queue. For example, set the yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent parameter to 0.5.

     • Error message: "User's AM resource limit exceeded."

       • Possible cause: The sum of the AM resources used by the user and the requested AM resources exceeds the per-user AM resource limit for the queue.

       • Solution: Increase the user-limit ratio. You can modify the values of the yarn.scheduler.capacity.<queue-path>.user-limit-factor and yarn.scheduler.capacity.<queue-path>.minimum-user-limit-percent parameters.

     • Error message: "AM container is launched, waiting for AM container to Register with RM."

       • Possible cause: The ApplicationMaster has started, but its internal initialization is not complete, for example, due to a ZooKeeper connection timeout.

       • Solution: Examine the ApplicationMaster logs.

     • Error message: "Application is Activated, waiting for resources to be assigned for AM."

       Proceed to Step 3 to investigate why the ApplicationMaster resource request is not being met.

   • RUNNING: Proceed to Step 2 to check if the container resource requests have been met.

   • FAILED: Check the diagnostics. Take the appropriate action based on the message provided.

     • Error message: "Maximum system application limit reached,cannot accept submission of application"

       • Possible cause: The total number of running applications in the cluster has exceeded the configured limit (parameter: yarn.scheduler.capacity.maximum-applications, default value: 10000).

       • Solution: Check JMX metrics to verify that each queue has the expected number of running applications. Identify and address any applications causing excessive repeated submissions. If all applications are legitimate, consider increasing this configuration value based on the cluster's workload.

     • Error message: "Application XXX submitted by user YYY to unknown queue: ZZZ"

       • Possible cause: The application was submitted to a queue that does not exist.

       • Solution: Submit the application to an existing leaf queue.

     • Error message: "Application XXX submitted by user YYY to non-leaf queue: ZZZ"

       • Possible cause: The application was submitted to a parent queue.

       • Solution: Submit the application to an existing leaf queue.

     • Error message: "Queue XXX is STOPPED. Cannot accept submission of application: YYY"

       • Possible cause: The application was submitted to a STOPPED or DRAINING queue. A queue in these states is offline or being decommissioned.

       • Solution: Submit the application to a queue in the RUNNING state.

     • Error message: "Queue XXX already has YYY applications, cannot accept submission of application: ZZZ"

       • Possible cause: The number of applications in the queue has reached its limit.

       • Solution:

         1. Investigate whether any problematic applications are being submitted repeatedly to the queue.

         2. Adjust the configuration: yarn.scheduler.capacity.<queue-path>.maximum-applications.

     • Error message: "Queue XXX already has YYY applications from user ZZZ cannot accept submission of application: AAA"

       • Possible cause: The number of applications from the user has reached its limit.

       • Solution:

         1. Check if the user is repeatedly submitting a problematic application.

         2. Adjust the following configuration parameters: yarn.scheduler.capacity.<queue-path>.maximum-applications, yarn.scheduler.capacity.<queue-path>.minimum-user-limit-percent, and yarn.scheduler.capacity.<queue-path>.user-limit-factor.

2. Check if YARN resource allocation is incomplete.

   1. On the applications list page, click an application ID to navigate to its details page.

   2. In the list at the bottom of the page, click the AM-ID to open the App Attempt page.

   3. Check the Total Outstanding Resource Requests list for pending resources. Alternatively, query for them with the PendingRequests REST API.

      • No pending resources: This indicates that YARN has completed the allocation. Exit this check and investigate the ApplicationMaster. When no pending resources exist, the Total Outstanding Resource Requests section shows 0 for all metrics (for example, memory:0, vCores:0), and the resource request table below is empty. This table includes columns such as Priority, ResourceName, Capability, and NumContainers.

      • Pending resources exist: This indicates that YARN resource allocation is not complete. Continue to the next step.

3. Check resource limits.

   Check cluster or queue resources. View resource information such as Effective Max Resource and Used Resources.

   1. Check if the resources of the cluster, the queue, or its parent queue are exhausted.

   2. Check if any resource dimension in the leaf queue is approaching or has reached its limit.

   3. When cluster resource utilization is high (for example, above 85%), application allocation speed may decrease. This can happen because most nodes have no available resources. When a node cannot satisfy a resource request, the scheduler may reserve the request. A large number of reservations can slow down the allocation process. This issue can also be caused by an imbalance between memory and CPU resources. For example, some nodes may have available memory but no available CPU, while others have available CPU but no available memory.

4. Check if containers fail to start even after their resources are allocated. The App Attempt page in the YARN web UI shows the number of allocated containers. Observe this number for a short period to see if it changes. If a container fails to start, check the corresponding NodeManager or container logs to investigate the failure.

5. Dynamically change the log level. Go to the Log Level page in the YARN web UI (http://RM_IP:8088/logLevel). In the Class Name field, enter the target class name (for example, a class in the org.apache.hadoop.yarn.server.resourcemanager.scheduler package, such as org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity). In the Level drop-down list, select DEBUG, and then click Set Log Level. This dynamically sets the log level for the specified class to DEBUG.

   Important

   Enable DEBUG logging only while you are reproducing the issue. A duration of less than a minute is typically sufficient, as logs are generated very quickly. After you capture the necessary logs, set the log level back to INFO.

Parameters for maximum available resources

YARN configuration changes not taking effect

• Possible causes

  • For configurations that do not support hot updates, the associated component was not restarted.

  • For configurations that support hot updates, the required action was not performed.

• Solution: After modifying a configuration, perform the correct follow-up actions.

  Configuration file	Type	Actions
  capacity-scheduler.xml fair-scheduler.xml	Scheduler configuration	Run the refresh_queues operation on ResourceManager.
  yarn-env.sh yarn-site.xml mapred-env.sh mapred-site.xml	YARN component runtime configuration	Restart the associated component. For example: If you modify a parameter such as YARN_RESOURCEMANAGER_HEAPSIZE in the yarn-env.sh file or yarn.resourcemanager.nodes.exclude-path in the yarn-site.xml file, you must restart ResourceManager. If you modify a parameter such as YARN_NODEMANAGER_HEAPSIZE in the yarn-env.sh file or yarn.nodemanager.log-dirs in the yarn-site.xml file, you must restart NodeManager. If you modify a parameter such as MAPRED_HISTORYSERVER_OPTS in the mapred-env.sh file or mapreduce.jobhistory.http.policys in the mapred-site.xml file, you must restart MRHistoryServer.

What do I do if the following error message is reported for an exception that occurs on an application client: Exception while invoking getClusterNodes of class ApplicationClientProtocolPBClientImpl over rm2 after 1 fail over attempts. Trying to fail over immediately?

• Problem description: The active ResourceManager cannot be accessed. The ResourceManager logs contain the following exception information: WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from 10.33.**.**:53144 got version 6 expected version 9.

• Cause: An early version of Hadoop is used. The version of remote procedure call (RPC) used by the application client is incompatible with the Hadoop version.

• Solution: Use Hadoop of a version that is compatible with the application client of the RPC version.

What do I do if ResourceManager cannot automatically switch from the Standby state to the Active state?

Resolving ResourceManager OOM errors

Why does localization fail when a node starts to run a job, and why are job logs unable to be collected or deleted?

• Problem description: The NodeManager logs contain the following exception information: java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.

• Cause: The HDFS configurations are incorrect.

• Solution:

  1. The exception information is encapsulated and is not the root cause of the issue. To identify the root cause, you must check the debug-level logs.

     • In the CLI for a Hadoop client, you can run a command such as hadoop fs -ls / to access Hadoop. Then, run the following command to enable debugging:

       export HADOOP_LOGLEVEL=DEBUG

     • In a runtime environment with the Log4j configuration, add log4j.logger.org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider=DEBUG to the end of the Log4j configuration.

  2. The example in the following figure shows that the root cause of the issue is that a user modified the configurations of NameServices. The user changed emr-cluster to hadoop-emr-cluster. However, the new node used the original configurations of NameServices after a scale-out.

  3. On the Configure tab of the HDFS service page of a cluster in the EMR console, check whether the parameters are correctly configured.

How do I handle a resource localization exception?

• Problem description:

  • The AM container that is used to process jobs fails to start, and the following error message is reported:

    Application application_1412960082388_788293 failed 2 times due to AM Container for appattempt_1412960082388_788293_000002 exited with exitCode: -1000 due to: EPERM: Operation not permitted. The exception information is the diagnostic information about the failed job.

  • An error is reported when an application resource package is decompressed after it is downloaded for resource localization. You can check the following NodeManager logs:

    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Failed to download rsrc { { hdfs://hadoopnnvip.cm6:9000/user/heyuan.lhy/apv/small_apv_20141128.tar.gz, 1417144849604, ARCHIVE, null },pending,[(container_1412960082388_788293_01_000001)],14170282104675332,DOWNLOADING}
    EPERM: Operation not permitted
            at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
            at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:226)
            at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:629)
            at org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:186)
            at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:235)
            at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:949)
            at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:945)
            at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
            at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:945)
            at org.apache.hadoop.yarn.util.FSDownload.changePermissions(FSDownload.java:398)
            at org.apache.hadoop.yarn.util.FSDownload.changePermissions(FSDownload.java:412)
            at org.apache.hadoop.yarn.util.FSDownload.changePermissions(FSDownload.java:412)
            at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:352)
            at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:57)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:744)

• Cause: The application resource package contains soft links, which result in the resource localization exception.

• Solution: Delete the soft links that are contained in the application resource package.

What do I do if the error message "No space left on device" is displayed and a container fails to start or run?

Domain names fail to be resolved in NodeManager or during the running of jobs. What do I do?

• Problem description: The following error message is displayed: java.net.UnknownHostException: Invalid host name: local host is: (unknown).

• Possible causes and solutions:

  • Check whether the Domain Name System (DNS) server is correctly configured.

    Run the following command to check the configuration of the DNS server:

    cat /etc/resolv.conf

  • Check whether the required rules are configured for Port 53 of the firewall.

    If the required rules are configured, disable the firewall.

  • Check whether the Name Service Cache Daemon (NSCD) service is enabled for the DNS server.

    Run the following command to check the status of the NSCD service:

    systemctl status nscd

    If the NSCD service is enabled for the DNS server, run the following command to disable the NSCD service:

    systemctl stop nscd

Handling NodeManager OOM errors

NodeManager startup failure: Missing cgroup directory

• Error message: ResourceHandlerException: Unexpected: Cannot create yarn cgroup Subsystem:cpu Mount point:/proc/mounts User:hadoop Path:/sys/fs/cgroup/cpu/hadoop-yarn

• Cause: An unexpected ECS instance restart, often caused by a kernel defect (a known issue in version 4.19.91-21.2.al7.x86_64), destroys the CPU cgroup's in-memory data and invalidates the cgroup.

• Solution: Modify the bootstrap script for existing and scale-out node groups to create the cgroup directory and configure the rc.local file to automatically recreate this directory on each instance startup.

  # enable cgroups
  mkdir -p /sys/fs/cgroup/cpu/hadoop-yarn
  chown -R hadoop:hadoop /sys/fs/cgroup/cpu/hadoop-yarn
  # enable cgroups after reboot
  echo "mkdir -p /sys/fs/cgroup/cpu/hadoop-yarn" >> /etc/rc.d/rc.local
  echo "chown -R hadoop:hadoop /sys/fs/cgroup/cpu/hadoop-yarn" >> /etc/rc.d/rc.local
  chmod +x /etc/rc.d/rc.local

Ineffective NodeManager resource configuration

• Description: After modifying the yarn.nodemanager.resource.cpu-vcores and yarn.nodemanager.resource.memory-mb parameters, saving the configuration, and restarting NodeManager, the resource counts do not update.

• Cause: These parameters must be applied at the node group level because instances in a node group can have different CPU and memory specifications.

• Solution: In the E-MapReduce (EMR) console, apply the resource settings to the NodeManager's node group. For more information, see Manage configuration items. On the Configure tab of the YARN service, switch the filter to Node Group Configuration. Select the target node group, such as emr-core (CORE), and modify the parameters on the yarn-site.xml sub-tab.

Troubleshoot unhealthy nodes

• Causes

  • The disk health checker marks a node as unhealthy if the ratio of healthy directories to total directories falls below the threshold defined by the yarn.nodemanager.disk-health-checker.min-healthy-disks parameter in yarn-site.xml (default: 0.25). For example, on a NodeManager node with four disks, the node becomes unhealthy only if the directories on all four disks are unhealthy. Otherwise, the NodeManager status report only reports that local-dirs are bad or log-dirs are bad. For more information, see What to do if the "local-dirs are bad" or "log-dirs are bad" message is reported?.

  • Issue reported by the NodeManager health check script: This check is disabled by default. You can enable this check by setting the yarn.nodemanager.health-checker.script.path parameter in yarn-site.xml to the path of a health check script.

• Solutions

  • For disk issues, see What to do if the "local-dirs are bad" or "log-dirs are bad" message is reported?.

  • For issues reported by the health check script, troubleshoot the problem based on your custom script.

Node disk issues: 'local-dirs are bad' or 'log-dirs are bad'

• Cause: The disk health checker periodically verifies that local-dirs (the cache directory for task dependencies, intermediate data, and other files) and log-dirs (the directory for task execution logs) meet several conditions. If a directory fails any condition, the system marks it as bad.

  • Readable

  • Writable

  • Executable

  • Disk usage is below the configured threshold. This is controlled by the yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage parameter in the yarn-site.xml file, which defaults to 90%.

    Available disk space is greater than the minimum free space requirement. This is controlled by the yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb parameter in the yarn-site.xml file, which defaults to 0.

• Solution

  • Insufficient disk space typically causes this issue. If any of the following scenarios apply, consider scaling out the disks:

    • The NodeManager node is high-spec and runs many tasks.

    • The disk capacity is relatively small.

    • Tasks depend on large data or files.

    • Tasks generate a large amount of intermediate data.

    • Task logs are large and consume significant disk space.

  • Check the yarn.nodemanager.localizer.cache.target-size-mb parameter in the yarn-site.xml file. If this value is too high relative to the disk capacity, the cache for historical tasks can consume excessive disk space. The system automatically clears this cache only after it exceeds the configured size.

  • If a damaged disk causes the issue, see Replace a damaged local disk in the cluster.

What do I do if the error message "User [dr.who] is not authorized to view the logs for application ***" is displayed?

• Problem description: Information in the following figure is displayed when the log page is opened.

• Cause: The access control list (ACL) rules are checked when you access the NodeManager Log page. If ACL rules are enabled and a remote user wants to view the logs of an application, the remote user must meet one of the following conditions:

  • The remote user is the admin user.

  • The remote user is the owner of the application.

  • The remote user meets the ACL rules that are customized for the application.

• Solution: Check whether the remote user meets one of the preceding conditions.

What do I do if the error message "HTTP ERROR 401 Authentication required" or "HTTP ERROR 403 Unauthenticated users are not authorized to access this page" is displayed?

• Problem description: The error HTTP ERROR 401 or HTTP ERROR 403 is reported when you access a web UI or call a RESTful API. The details of HTTP ERROR 401 is shown in the following figure.

  

• Cause: YARN uses the simple authentication method and does not allow anonymous access. For more information, see Authentication for Hadoop HTTP web-consoles in the official documentation of Apache Hadoop.

• Solutions:

  • Method 1: Configure a URL parameter to specify a remote user, such as user.name=***.

  • Method 2: In the Configuration Filter section on the Configure tab of the HDFS service page of a cluster in the EMR console, search for the hadoop.http.authentication.simple.anonymous.allowed parameter and change the value of the parameter to true to allow anonymous access. For more information about the parameter, see Authentication for Hadoop HTTP web-consoles in the official documentation of Apache Hadoop. Then, restart the HDFS service. For more information, see Restart a service.

Why is the display value of TotalVcore incorrect?

In the cluster or metrics RESTful API section in the upper-right corner of the web UI of YARN, the display value of TotalVcore is incorrect. This is a computing logic issue for TotalVcore in Apache Hadoop versions earlier than 2.9.2. For more information, see Total #VCores in cluster metrics is wrong when CapacityScheduler reserved some containers in the official documentation of Apache Hadoop.

The issue is fixed in EMR V3.37.x, EMR V5.3.x, and their later minor versions.

What do I do if the information about an application that is displayed on the web UI of TEZ is incomplete?

Spark Thrift Server task on the YARN web UI

If you select Spark when creating a cluster, the cluster automatically starts a default Spark Thrift Server service. This service uses one YARN driver's resources. By default, Spark Thrift Server tasks request resources from YARN through this driver.

OSS URL support for yarn.timeline-service.leveldb-timeline-store.path

The yarn.timeline-service.leveldb-timeline-store.path parameter does not support OSS bucket URLs.

When you create a Hadoop cluster, the yarn.timeline-service.leveldb-timeline-store.path parameter defaults to the value of the hadoop.tmp.dir parameter. Do not modify the hadoop.tmp.dir parameter for HDFS, as this affects the yarn.timeline-service.leveldb-timeline-store.path parameter.

Timeline Server connection timeouts or high resource usage

When running a large number of Tez jobs, you may experience connection timeouts when writing events to the YARN Timeline Server. This occurs because the Timeline Server process consumes excessive CPU resources, causing the node's CPU usage to reach its limit. This affects job execution and non-core services such as report generation. To resolve this issue, modify the following configuration items.

In the EMR console, modify the configurations for the Tez and YARN services. For instructions, see Manage configuration items.

1. On the tez-site.xml tab of the Configure page for the Tez service, add the tez.yarn.ats.event.flush.timeout.millis parameter and set its value to 60000. This parameter sets the timeout for Tez tasks writing events to the YARN Timeline Server.

2. On the yarn-site.xml tab of the Configure page for the YARN service, add or modify the following configuration items. Then, on the Status page of the YARN service, restart TimelineServer.

   Parameter	Value	Description
   yarn.timeline-service.store-class	org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore	The event storage class for the YARN Timeline Server.
   yarn.timeline-service.rolling-period	daily	The event rolling period for the YARN Timeline Server.
   yarn.timeline-service.leveldb-timeline-store.read-cache-size	4194304	The read cache size for the YARN Timeline Server LevelDB store.
   yarn.timeline-service.leveldb-timeline-store.write-buffer-size	4194304	The write buffer size for the YARN Timeline Server LevelDB store.
   yarn.timeline-service.leveldb-timeline-store.max-open-files	500	The maximum number of open files for the YARN Timeline Server LevelDB store.