YARN-4946 is a bug in which the YARN ResourceManager (RM) removes a completed application from its history only after log aggregation for that application enters a completed state. If the log aggregation state is not synchronized to the RM state store before a restart, the RM loads the application as incomplete after the restart and cannot delete it. Affected applications accumulate in the RM over time.
When the number of accumulated applications reaches the value of ${yarn.resourcemanager.state-store.max-completed-applications} or ${yarn.resourcemanager.max-completed-applications} (default: 10,000 for both), job scheduling in the RM is disrupted. YARN-9571 reverts the YARN-4946 change. For details, see YARN-4946 and YARN-9571.
Check if your cluster is affected
Affected service: YARN service in a high-availability (HA) E-MapReduce (EMR) Hadoop cluster that includes ZooKeeper.
Severity: Critical. An EMR cluster with this defect may fail to recover after a restart following extended operation.
Affected EMR versions: V4.6.0, V4.7.0, V4.8.0, V4.9.0, V5.1.0, V5.2.0, and V5.2.1.
All affected versions use Hadoop 3.2.1.
Check the RM log for the following error:
Application should be expired, max number of completed apps kept in memory met: maxCompletedAppsInMemory = 10000, but not removing app XXX from memory as log aggregation have not finished yet.If this error appears repeatedly in the RM log, your cluster is affected. The defect causes the RM to become unavailable, or to remain unavailable for an extended period, after a restart.
Prerequisites
Before you begin, make sure you have:
A high-availability EMR Hadoop cluster running one of the affected versions
SSH access to the master nodes
Fix the defect
Replace the defective RM JAR file on each RM node and restart the RMs in the correct order: standby RM first, then active RM.
This solution applies only to the EMR versions listed above. Restarting the YARN service after replacing the JAR file may cause running jobs to fail. Perform the restart during off-peak hours.
Perform all steps on the standby RM first, then repeat on the active RM.
After restarting one RM, wait for it to return to a normal running state before restarting the other. If high availability is not enabled for your Hadoop cluster, no action is required.
Step 1: Download the patched JAR file
Click hadoop-yarn-server-resourcemanager-3.2.1.jar to download the patched YARN RM JAR file.
Step 2: Replace the JAR file on the standby RM node
Log in to the standby RM master node and copy the downloaded JAR file to the Hadoop installation directory. This example uses
/usr/lib/hadoop-current/share/hadoop/yarn/.Back up the original JAR file and copy the new one:
mv $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.2.1.jar /tmp/ cp hadoop-yarn-server-resourcemanager-3.2.1.jar $HADOOP_HOME/share/hadoop/yarn/$HADOOP_HOMEis the Hadoop installation directory —/usr/lib/hadoop-currentin this example.
Step 3: Restart the standby RM and verify
Restart the standby YARN RM from the EMR console. After the restart, verify that the defect is fixed:
The error message
but not removing app XXX from memory as log aggregation have not finished yet.no longer appears in the RM log.Jobs can be submitted successfully.
If both conditions are met, the defect is fixed on the standby node.
Step 4: Replace the JAR file on the active RM node and restart
Search for the active RM in the EMR console and repeat Step 2 on the active RM master node, then restart the active RM from the EMR console.
Roll back
If you encounter a problem after replacing the JAR file, restore the original JAR file from the backup and restart the RM on the affected node:
cp /tmp/hadoop-yarn-server-resourcemanager-3.2.1.jar $HADOOP_HOME/share/hadoop/yarn/