E-MapReduce:FAQ

What does a stateful cluster restart constitute?

A stateful cluster restart constitutes a ResourceManager restart and a NodeManager restart. ResourceManager maintains the basic information about and status of applications. NodeManager maintains the information about and status of containers that are running. ResourceManager and NodeManager constantly synchronize their own status to external storage systems such as ZooKeeper, LevelDB, and Hadoop Distributed File System (HDFS). The status of ResourceManager and NodeManager can be automatically reloaded and recovered after they are restarted. This ensures that the status of applications and containers can be automatically recovered after the cluster is upgraded or restarted. In most cases, the upgrade or restart of a cluster is imperceptible to applications and containers.

How do I enable ResourceManager high availability (HA)?

How do I configure a hot update?

1. Configure key parameters.

   You can check or configure parameters related to a hot update on the Configure tab of the YARN service page of a cluster in the EMR console. The following table describes the parameters.

   Parameter	Description	Recommended value
   yarn.scheduler.configuration.store.class	The type of the backing store. If you set this parameter to fs, a file system is used as the backing store.	fs
   yarn.scheduler.configuration.max.version	The maximum number of configuration files that can be stored in the file system. Excess configuration files are automatically deleted if the number of configuration files exceeds the value of this parameter.	100
   yarn.scheduler.configuration.fs.path	The path in which the capacity-scheduler.xml file is stored. If you do not configure this parameter, a storage path is automatically created. If no prefix is specified, the relative path of the default file system is used as the storage path.	/yarn/<Cluster name>/scheduler/conf Important Replace <Cluster name> with a specific cluster name. Multiple clusters for which the YARN service is deployed may use the same distributed storage.

2. View configurations of the capacity-scheduler.xml file.

   • Method 1 (RESTful API): Access a URL in the following format: http://<rm-address>/ws/v1/cluster/scheduler-conf.

   • Method 2 (HDFS): Access the configuration path ${yarn.scheduler.configuration.fs.path}/capacity-scheduler.xml.<timestamp> to view configurations of the capacity-scheduler.xml file. <timestamp> indicates the time at which the capacity-scheduler.xml file is generated. The capacity-scheduler.xml file that has the largest timestamp value is the latest configuration file.

3. Update the configurations.

   For example, you can modify the parameter yarn.scheduler.capacity.maximum-am-resource-percent and delete the parameter yarn.scheduler.capacity.xxx. To delete a parameter, you need to only remove the value field of the parameter.

   curl -X PUT -H "Content-type: application/json" 'http://<rm-address>/ws/v1/cluster/scheduler-conf' -d '
   {
     "global-updates": [
       {
         "entry": [{
           "key":"yarn.scheduler.capacity.maximum-am-resource-percent",
           "value":"0.2"
         },{
           "key":"yarn.scheduler.capacity.xxx"
         }]
       }
     ]
   }'

How do I handle the uneven distribution of resources among applications in a queue?

How do I view the resource usage of a queue?

You can view the value of the Used Capacity parameter on the web UI of YARN to view the resource usage of a queue. The Used Capacity parameter specifies the percentage of resources used by a queue to all resources that are allocated to the queue. The percentage of memory resources used by the queue and the percentage of the number of vCores used by the queue are separately calculated. The greater value is used as the value of the Used Capacity parameter. To view the resource usage of a queue, perform the following operations:

1. Access the web UI of YARN. For more information, see Access the web UIs of open source components.

2. On the All Applications page, click the ID of a specific job.

3. Click the desired queue in the Queue row.

   In the Application Queues section, view the resource usage of the queue.

   

A large number of .out log files are generated by YARN service components and cannot be automatically cleared. Why?

• Cause: Specific dependency libraries of Hadoop components call Java logging APIs to generate log records, and do not support Log4j-based log rotation. The standard error (stderr) output of the daemon components is redirected to .out log files. Due to the fact that no automatic cleanup mechanism is provided, prolonged log accumulation may cause the data disk storage to be fully occupied.

• Solution: Use the head and tail commands in combination with the timestamp information generated in logs to check whether the logs that are generated by Java logging APIs occupy a large amount of storage space. In most cases, the dependency libraries generate INFO-level logs, which do not affect the use of components. To prevent the storage capacity of data disks from being occupied, you can modify configurations to disable the generation of INFO-level logs.

  For example, perform the following steps to disable log generation of Jersey dependency libraries of Timeline Server:

  1. Run the following command to monitor .out log files that are relevant to timelineserver- in the log storage path of YARN. In a DataLake cluster, the path is /var/log/emr/yarn/. In a Hadoop cluster, the path is /mnt/disk1/log/hadoop-yarn.

     tail /var/log/emr/yarn/*-hadoop-timelineserver-*.out

     The output contains records generated by the com.sun.jersey package.

     

  2. On the node on which Timeline Server is run, run the following command to create and configure a configuration file as the root user to disable log generation of Timeline Server. This prevents INFO-level logs generated by Jersey dependency libraries from being exported to .out log files.

     sudo su root -c "echo 'com.sun.jersey.level = OFF' > $HADOOP_CONF_DIR/off-logging.properties"

  3. On the Configure tab of the YARN service page in the EMR console, find the YARN_TIMELINESERVER_OPTS parameter (corresponding to the yarn_timelineserver_opts parameter in a Hadoop cluster), and add -Djava.util.logging.config.file=off-logging.properties to the value of the parameter to specify the location of the created configuration file.

     

  4. Save the configuration and restart Timeline Server for the configuration to take effect. If Timeline Server can start as expected, and .out log files no longer contain log information that is relevant to com.sun.jersey, log generation of Jersey dependency libraries is disabled.

How do I check whether the ResourceManager service is normal?

How do I obtain the status of an application?

How do I troubleshoot issues for an application?

1. Check the status of the application. You can go to the details page of the application or call the RESTful API for the application to view the status of the application. The application may be in one of the following states:

   • The state of the application is unknown. Possible causes:

     • The application fails and exits before the application is submitted to YARN. In this case, view the application submission logs to check whether issues occur on client components, such as BRS and FlowAgent.

     • The application cannot connect to ResourceManager. In this case, check whether the endpoint of ResourceManager is correct and whether the network connection is normal. If the network connection is abnormal, the following error may be reported in the client: com.aliyun.emr.flow.agent.common.exceptions.EmrFlowException: ###[E40001,RESOURCE_MANAGER]: Failed to access to resource manager, cause: The stream is closed.

   • NEW_SAVING: The information about the application is being written to the Zookeeper state store. Possible reasons that cause the application to be stuck in the NEW_SAVING state:

     • Errors occur on ZooKeeper. Check whether the ZooKeeper service is normal.

     • Errors occur on the read or write operations of ZooKeeper. For more information about solutions, see What do I do if ResourceManager cannot automatically switch from the Standby state to the Active state?

   • SUBMITTED: In most cases, an application is not in this state. If lock acquisition caused by excess node update requests occurs in Capacity Scheduler and Capacity Scheduler is blocked, the application may be in the SUBMITTED state. In most cases, the lock acquisition in Capacity Scheduler and the blocking of Capacity Scheduler occur in a large-scale cluster. You must optimize the request processing procedure to resolve the issues. For more information, see YARN-9618.

   • ACCEPTED: If the application is in this state, check the diagnostic information. You can select a solution based on the error message that appears:

     • Error message: Queue's AM resource limit exceeded.

       • Possible cause: The sum of used ApplicationMaster (AM) resources and requested AM resources exceeds the upper limit of AM resources in a queue. AM resource usage must observe the following inequation: ${Used Application Master Resources} + ${AM Resource Request} < ${Max Application Master Resources}.

       • Solution: Increase the upper limit of AM resources in a queue. For example, set the yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent parameter to 0.5.

     • Error message: User's AM resource limit exceeded.

       • Possible cause: The sum of used AM resources and requested AM resources exceeds the upper limit of AM resources in a queue for a specific user.

       • Solution: Increase the upper limit of AM resources in a queue for a specific user. For example, modify the yarn.scheduler.capacity.<queue-path>.user-limit-factor and yarn.scheduler.capacity.<queue-path>.minimum-user-limit-percent parameters.

     • Error message: AM container is launched, waiting for AM container to Register with RM.

       • Possible cause: The AM is started but the AM initialization is not complete. For example, ZooKeeper connection times out.

       • Solution: Further troubleshoot the issue based on the AM logs.

     • Error message: Application is Activated, waiting for resources to be assigned for AM.

       Perform Step 3 to analyze the cause of inadequate AM resources.

   • RUNNING: If the application is in this state, perform Step 2 to check whether the container resource request is complete.

   • FAILED: If the application is in this state, check the diagnostic information. You can select a solution based on the error message that appears:

     • Error message: Maximum system application limit reached, cannot accept submission of application.

       • Possible cause: The number of running applications in the cluster exceeds the upper limit specified by the yarn.scheduler.capacity.maximum-applications parameter. The default value of this parameter is 10000.

       • Solution: View the Java Management Extensions (JMX) metrics to check whether the number of running applications in each queue is as expected. Troubleshoot the applications that are repeatedly submitted. If all applications run properly, you can set the parameter to a greater value based on the cluster usage.

     • Error message: Application XXX submitted by user YYY to unknown queue: ZZZ.

       • Possible cause: The application is submitted to a queue that does not exist.

       • Solution: Submit the application to an existing leaf queue.

     • Error message: Application XXX submitted by user YYY to non-leaf queue: ZZZ.

       • Possible cause: The application is submitted to a parent queue.

       • Solution: Submit the application to an existing leaf queue.

     • Error message: Queue XXX is STOPPED. Cannot accept submission of application: YYY.

       • Possible cause: The application is submitted to a queue that is in the STOPPED or DRAINING state.

       • Solution: Submit the application to a queue in the RUNNING state.

     • Error message: Queue XXX already has YYY applications, cannot accept submission of application: ZZZ.

       • Possible cause: The number of applications in the queue reaches the upper limit.

       • Solution:

         1. Check whether a large number of applications are repeatedly submitted to the queue.

         2. Modify the yarn.scheduler.capacity.<queue-path>.maximum-applications parameter.

     • Error message: Queue XXX already has YYY applications from user ZZZ cannot accept submission of application: AAA.

       • Possible cause: The number of applications in a queue for a user reaches the upper limit.

       • Solution:

         1. Check whether a large number of applications are repeatedly submitted to the queue for the user.

         2. Modify the yarn.scheduler.capacity.<queue-path>.maximum-applications, yarn.scheduler.capacity.<queue-path>.minimum-user-limit-percent, and yarn.scheduler.capacity.<queue-path>.user-limit-factor parameters.

2. Verify that YARN resource allocation is not complete.

   1. On the page that displays the list of applications, click the ID of the application to go to the details page of the application.

   2. Click the ID of an application attempt on the details page of the application to go to the details page of the application attempt.

   3. Check whether resources that are in the Pending state exist in the Total Outstanding Resource Requests list. You can also query resources that are in the Pending state by calling the RESTful API that is used to request pending requests.

      • If resources that are in the Pending state do not exist, the YARN resource allocation is complete. Exit the troubleshooting procedure and check the AM resource allocation.

      • If resources that are in the Pending state exist, the YARN resource allocation is not complete. Proceed to the next step.

3. Check limits on resources.

   Check resources in a cluster or in a queue. For example, check the values of the Effective Max Resource and Used Resources parameters.

   1. Check whether the following resources are fully consumed: resources in a cluster, resources in a queue, or resources in the parent queue.

   2. Check whether the resources of a dimension in a leaf queue are close to or reach the upper limit.

   3. If the resource usage of a cluster is close to 100%, such as more than 85%, the speed at which resources are allocated to applications may decrease. One reason is that most of the machines do not have resources. If a machine does not have resources to allocate, the machine is reserved. If the number of reserved machines reaches a certain number, the speed at which resources are allocated may decrease. Another possible reason is that memory resources are out of proportion to CPU resources. For example, some machines have idle memory resources but no idle CPU resources. However, some machines have idle CPU resources but no idle memory resources.

4. Check whether resources are successfully allocated to a container but the container fails to start. On the App Attempt page of the web UI of YARN, you can view the number of containers to which resources are allocated and the changes on the number of containers over a short period of time. If a container fails to start, troubleshoot the failure based on NodeManager logs or the logs of the container.

5. Dynamically adjust the log level. On the Log Level page of the web UI of YARN at http://RM_IP:8088/logLevel, change the log level for the org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity parameter to DEBUG. The parameter indicates the package of Capacity Scheduler. The following figure provides an example.

   Important

   You can adjust the log level when you reproduce the issue and change the log level back to INFO after tens of seconds. This is because many logs can be generated in a short period of time.

What parameters determine the maximum number of available resources for a single task or container?

What do I do if the modification of YARN configurations does not take effect?

• Possible causes:

  • For other configurations, the component associated with the configurations is not restarted.

  • For hot update configurations, the specified operation required for the configurations to take effect is not performed.

• Solution: Make sure that the subsequent operations required for the configurations to take effect are correctly performed.

  Configuration file	Category	Subsequent operation
  capacity-scheduler.xml fair-scheduler.xml	Scheduler configurations	Perform the refresh_queues operation on ResourceManager. The configurations are hot update configurations.
  yarn-env.sh yarn-site.xml mapred-env.sh mapred-site.xml	Configurations of YARN components	Restart the component that is associated with the configurations. Examples: Restart ResourceManager after you modify the YARN_RESOURCEMANAGER_HEAPSIZE parameter in the yarn-env.sh file or the yarn.resourcemanager.nodes.exclude-path parameter in the yarn-site.xml file. Restart NodeManager after you modify the YARN_NODEMANAGER_HEAPSIZE parameter in the yarn-env.sh file or the yarn.nodemanager.log-dirs parameter in the yarn-site.xml file. Restart MRHistoryServer after you modify the MAPRED_HISTORYSERVER_OPTS parameter in the mapred-env.sh file or the mapreduce.jobhistory.http.policys parameter in the mapred-site.xml file.

What do I do if the following error message is reported for an exception that occurs on an application client: Exception while invoking getClusterNodes of class ApplicationClientProtocolPBClientImpl over rm2 after 1 fail over attempts. Trying to fail over immediately?

• Problem description: The active ResourceManager cannot be accessed. The ResourceManager logs contain the following exception information: WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from 10.33.**.**:53144 got version 6 expected version 9.

• Cause: An early version of Hadoop is used. The version of remote procedure call (RPC) used by the application client is incompatible with the Hadoop version.

• Solution: Use Hadoop of a version that is compatible with the application client of the RPC version.

What do I do if ResourceManager cannot automatically switch from the Standby state to the Active state?

What do I do if an OOM issue occurs in ResourceManager?

Why does localization fail when a node starts to run a job, and why are job logs unable to be collected or deleted?

• Problem description: The NodeManager logs contain the following exception information: java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.

• Cause: The HDFS configurations are incorrect.

• Solution:

  1. The exception information is encapsulated and is not the root cause of the issue. To identify the root cause, you must check the debug-level logs.

     • In the CLI for a Hadoop client, you can run a command such as hadoop fs -ls / to access Hadoop. Then, run the following command to enable debugging:

       export HADOOP_LOGLEVEL=DEBUG

     • In a runtime environment with the Log4j configuration, add log4j.logger.org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider=DEBUG to the end of the Log4j configuration.

  2. The example in the following figure shows that the root cause of the issue is that a user modified the configurations of NameServices. The user changed emr-cluster to hadoop-emr-cluster. However, the new node used the original configurations of NameServices after a scale-out.

  3. On the Configure tab of the HDFS service page of a cluster in the EMR console, check whether the parameters are correctly configured.

How do I handle a resource localization exception?

• Problem description:

  • The AM container that is used to process jobs fails to start, and the following error message is reported:

    Application application_1412960082388_788293 failed 2 times due to AM Container for appattempt_1412960082388_788293_000002 exited with exitCode: -1000 due to: EPERM: Operation not permitted. The exception information is the diagnostic information about the failed job.

  • An error is reported when an application resource package is decompressed after it is downloaded for resource localization. You can check the following NodeManager logs:

    INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Failed to download rsrc { { hdfs://hadoopnnvip.cm6:9000/user/heyuan.lhy/apv/small_apv_20141128.tar.gz, 1417144849604, ARCHIVE, null },pending,[(container_1412960082388_788293_01_000001)],14170282104675332,DOWNLOADING}
    EPERM: Operation not permitted
            at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
            at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:226)
            at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:629)
            at org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:186)
            at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:235)
            at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:949)
            at org.apache.hadoop.fs.FileContext$10.next(FileContext.java:945)
            at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
            at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:945)
            at org.apache.hadoop.yarn.util.FSDownload.changePermissions(FSDownload.java:398)
            at org.apache.hadoop.yarn.util.FSDownload.changePermissions(FSDownload.java:412)
            at org.apache.hadoop.yarn.util.FSDownload.changePermissions(FSDownload.java:412)
            at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:352)
            at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:57)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:744)

• Cause: The application resource package contains soft links, which result in the resource localization exception.

• Solution: Delete the soft links that are contained in the application resource package.

What do I do if the error message "No space left on device" is displayed and a container fails to start or run?

Domain names fail to be resolved in NodeManager or during the running of jobs. What do I do?

• Problem description: The following error message is displayed: java.net.UnknownHostException: Invalid host name: local host is: (unknown).

• Possible causes and solutions:

  • Check whether the Domain Name System (DNS) server is correctly configured.

    Run the following command to check the configuration of the DNS server:

    cat /etc/resolv.conf

  • Check whether the required rules are configured for Port 53 of the firewall.

    If the required rules are configured, disable the firewall.

  • Check whether the Name Service Cache Daemon (NSCD) service is enabled for the DNS server.

    Run the following command to check the status of the NSCD service:

    systemctl status nscd

    If the NSCD service is enabled for the DNS server, run the following command to disable the NSCD service:

    systemctl stop nscd

What do I do if an OOM issue occurs in NodeManager?

After an ECS instance is restarted, NodeManager fails to be started because the cgroups directory is missing. What do I do?

• Error message: ResourceHandlerException: Unexpected: Cannot create yarn cgroup Subsystem:cpu Mount point:/proc/mounts User:hadoop Path:/sys/fs/cgroup/cpu/hadoop-yarn

• Cause: The abnormal restart of the ECS instance may be caused by a kernel defect of the instance. This is a known issue in the version 4.19.91-21.2.al7.x86_64. After the ECS instance is restarted, the cgroups become invalid. This is because the data of cgroups in the memory is deleted after the restart.

• Solution: Modify the bootstrap scripts of node groups. Create a directory of cgroups in the cluster, and run the rc.local script to create the directory if the ECS instance is started. The following code provides an example:

  # enable cgroups
  mkdir -p /sys/fs/cgroup/cpu/hadoop-yarn
  chown -R hadoop:hadoop /sys/fs/cgroup/cpu/hadoop-yarn
  # enable cgroups after reboot
  echo "mkdir -p /sys/fs/cgroup/cpu/hadoop-yarn" >> /etc/rc.d/rc.local
  echo "chown -R hadoop:hadoop /sys/fs/cgroup/cpu/hadoop-yarn" >> /etc/rc.d/rc.local
  chmod +x /etc/rc.d/rc.local

What do I do if the resource configurations of NodeManager do not take effect after I save the configurations and restart NodeManager?

• Description: The yarn.nodemanager.resource.cpu-vcores and yarn.nodemanager.resource.memory-mb parameters are modified and saved. However, the configurations do not take effect after NodeManager is restarted.

• Cause: The CPU cores and memory size of instances in a node group may vary. You need to modify the yarn.nodemanager.resource.cpu-vcores and yarn.nodemanager.resource.memory-mb parameters for a node group for the settings to take effect.

• Solution: Log on to the E-MapReduce (EMR) console. On the Configure tab of the YARN service page, select Node Group Configuration from the drop-down list right to the search box, and select the node group to which NodeManager belongs. Then, modify the yarn.nodemanager.resource.cpu-vcores and yarn.nodemanager.resource.memory-mb parameters. For more information, see Manage configuration items.

What do I do if a node is marked as unhealthy?

• Causes:

  • Exceptions are detected during the disk health check: If the ratio of the number of normal directories to the total number of directories on a node is lower than the specified ratio, the node is marked as unhealthy. The ratio is specified by the yarn.nodemanager.disk-health-checker.min-healthy-disks parameter in the yarn-site.xml file. The default value of this parameter is 0.25. If multiple disks, such as four disks, are deployed on a node on which NodeManager resides, the node is marked as unhealthy only if directories are abnormal on all the four disks. Otherwise, the message "local-dirs are bad" or "log-dirs are bad" is recorded in the report of NodeManager status. For more information, see What do I do if the message "local-dirs are bad" or "log-dirs are bad" is reported?

  • Exceptions are detected by the health check script of NodeManager: By default, the script is disabled. You need to configure the yarn.nodemanager.health-checker.script.path parameter in the yarn-site.xml file to enable the health check script.

• Solutions:

  • If the issue is caused by a disk exception, see the solutions in the What do I do if the message "local-dirs are bad" or "log-dirs are bad" is reported? section.

  • If the issue is detected by the health check script, resolve the issue based on the custom script.

What do I do if the message "local-dirs are bad" or "log-dirs are bad" appears?

• Cause: An exception is detected in the disk health check. By default, the feature of disk health check is enabled. This feature periodically checks whether local-dirs and log-dirs meet certain conditions. local-dirs is the cache directory of tasks and stores the files and intermediate data that are required for running tasks. log-dirs is the log directory of tasks and stores all the logs of tasks. If one of the following conditions is not met, local-dirs or log-dirs is marked as bad:

  • Readable.

  • Writable.

  • Executable.

  • The disk usage is lower than the threshold specified by the yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage parameter in the yarn-site.xml file. The default value of this parameter is 90%.

    The remaining available disk space is larger than the minimum available disk space that is specified by the yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb parameter in the yarn-site.xml file. The default value of this parameter is 0.

• Solutions:

  • In most cases, the issue is caused by insufficient disk space. If one of the following conditions is met, we recommend that you scale out disks:

    • The specifications of the node on which NodeManager resides are high, and a large number of tasks are running on the node.

    • The disk space is small.

    • The size of data or files required for running tasks is large.

    • A large amount of intermediate data is generated when tasks are running.

    • The size of task logs is large, and task logs occupy a high proportion of disk space.

  • Check the yarn.nodemanager.localizer.cache.target-size-mb parameter in the yarn-site.xml file. If the ratio of the cache size to the disk size is too large, most disk space is occupied by cached task data. This is because the cache can be automatically cleared only if the specified threshold is exceeded.

  • Repair the damaged disks. For more information about how to repair a damaged disk, see Replace a damaged local disk for an EMR cluster.

What do I do if the error message "User [dr.who] is not authorized to view the logs for application ***" is displayed?

• Problem description: Information in the following figure is displayed when the log page is opened.

• Cause: The access control list (ACL) rules are checked when you access the NodeManager Log page. If ACL rules are enabled and a remote user wants to view the logs of an application, the remote user must meet one of the following conditions:

  • The remote user is the admin user.

  • The remote user is the owner of the application.

  • The remote user meets the ACL rules that are customized for the application.

• Solution: Check whether the remote user meets one of the preceding conditions.

What do I do if the error message "HTTP ERROR 401 Authentication required" or "HTTP ERROR 403 Unauthenticated users are not authorized to access this page" is displayed?

• Problem description: The error HTTP ERROR 401 or HTTP ERROR 403 is reported when you access a web UI or call a RESTful API. The details of HTTP ERROR 401 is shown in the following figure.

  

• Cause: YARN uses the simple authentication method and does not allow anonymous access. For more information, see Authentication for Hadoop HTTP web-consoles in the official documentation of Apache Hadoop.

• Solutions:

  • Method 1: Configure a URL parameter to specify a remote user, such as user.name=***.

  • Method 2: In the Configuration Filter section on the Configure tab of the HDFS service page of a cluster in the EMR console, search for the hadoop.http.authentication.simple.anonymous.allowed parameter and change the value of the parameter to true to allow anonymous access. For more information about the parameter, see Authentication for Hadoop HTTP web-consoles in the official documentation of Apache Hadoop. Then, restart the HDFS service. For more information, see Restart a service.

Why is the display value of TotalVcore incorrect?

In the cluster or metrics RESTful API section in the upper-right corner of the web UI of YARN, the display value of TotalVcore is incorrect. This is a computing logic issue for TotalVcore in Apache Hadoop versions earlier than 2.9.2. For more information, see Total #VCores in cluster metrics is wrong when CapacityScheduler reserved some containers in the official documentation of Apache Hadoop.

The issue is fixed in EMR V3.37.x, EMR V5.3.x, and their later minor versions.

What do I do if the information about an application that is displayed on the web UI of TEZ is incomplete?

Why is a Spark Thrift JDBC/ODBC Server task running on the web UI of YARN?

If you select Spark when you create a cluster, a default Spark Thrift Server service is automatically started. The service occupies the resources of one YARN driver. By default, tasks that run in Spark Thrift Server apply for required resources from YARN by using this driver.

Can the yarn.timeline-service.leveldb-timeline-store.path parameter be set to an OSS bucket URL?

No, the yarn.timeline-service.leveldb-timeline-store.path parameter cannot be set to an OSS bucket URL.

If the cluster that you create is a Hadoop cluster, the default value of the yarn.timeline-service.leveldb-timeline-store.path parameter is the same as the value of the hadoop.tmp.dir parameter. Do not modify the hadoop.tmp.dir parameter, which affects the settings of the yarn.timeline-service.leveldb-timeline-store.path parameter.

What do I do if the connection to Timeline Server times out or the CPU load or memory usage is extremely high?

In the case of a large number of Tez jobs, the connection to Timeline Server of YARN may time out when data is written to Timeline Server. This is because the Timeline Server process occupies a large amount of CPU resources and the CPU load of the node on which Timeline Server runs reaches the upper limit. As a result, job development and other non-core services, such as report generation, are affected. In this case, you can temporarily stop the Timeline Server process to reduce the CPU load of the node. You can add or modify the following configuration items to resolve the issue.

Add configurations to or modify configurations of the Tez and YARN services in the EMR console. For more information, see Manage configuration items.

1. Go to the tez-site.xml tab on the Configure tab of the Tez service page in the EMR console. Add a configuration item whose name is tez.yarn.ats.event.flush.timeout.millis and value is 60000. This configuration item specifies the timeout period that is allowed when a Tez job is run to write events to Timeline Server of YARN.

2. Go to the yarn-site.xml tab on the Configure tab of the YARN service page in the EMR console and add or modify the configuration items that are described in the following table. After the addition or modification operation is complete, you must restart Timeline Server on the Status tab of the YARN service page.

   Configuration item	Value	Description
   yarn.timeline-service.store-class	org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore	Specifies the event storage class of YARN Timeline Server.
   yarn.timeline-service.rolling-period	daily	Specifies the event rolling period of YARN Timeline Server.
   yarn.timeline-service.leveldb-timeline-store.read-cache-size	4194304	Specifies the size of the read cache of the YARN Timeline Server LevelDB store.
   yarn.timeline-service.leveldb-timeline-store.write-buffer-size	4194304	Specifies the size of the write buffer of the YARN Timeline Server LevelDB store.
   yarn.timeline-service.leveldb-timeline-store.max-open-files	500	Specifies the maximum number of files that can be opened in the YARN Timeline Server LevelDB store.