YARN
YARN is a distributed resource management system and a core component of Hadoop. It manages cluster resources, schedules jobs, and monitors tasks.
Terms
|
Name |
Description |
|
ResourceManager |
Manages and schedules cluster resources and allocates resources to jobs running on YARN. In a non-HA cluster, ResourceManager is deployed on the EMR master node. In an HA cluster, it is deployed on multiple master nodes for high availability. |
|
NodeManager |
Manages resources, monitors status, and runs jobs on individual nodes. Deployed on the core or task nodes of an EMR cluster. |
|
MapReduce History Server (MRHistoryServer) |
Parses MapReduce job metrics and displays job execution status. Also periodically deletes expired aggregation logs. |
|
TimelineServer |
Collects job metrics and displays job execution status. Note
TimelineServer only monitors the resource usage of individual jobs. It does not affect the development, execution, or submission of data jobs. |
|
WebAppProxyServer |
Redirects job links to reduce web-based attacks. |
|
ApplicationMaster |
Manages the lifecycle of an application. ApplicationMaster negotiates resources from ResourceManager and works with NodeManager to execute and monitor tasks. |
Benefits
YARN in EMR provides the following benefits:
-
HA clusters automatically deploy YARN in high availability mode.
-
Simplified operations and maintenance (O&M).
The console supports node scale-out, NodeManager shutdown, and rolling restarts.
-
Supports monitoring and alerts.
Monitor various metrics and configure intelligent alerts.
-
Auto Scaling supports graceful shutdown.
Nodes wait for running tasks to finish before going offline, preventing immediate NodeManager shutdown that would cause task failures and reruns.