Starting from EMR 3.49.0 and 5.15.0, EMR dynamically sets default memory values for YARN components at cluster creation time based on the selected instance types and deployed services. You can adjust these values in the EMR console after the cluster is created.
If YARN components receive less memory than expected after cluster initialization, check two things:
The number of services deployed in the cluster. EMR distributes available memory across all deployed services, so more services means less memory per component.
Whether the Elastic Compute Service (ECS) instance specifications in the node group meet the memory requirements of all deployed services.
Heap memory size for YARN components
On the Configure tab of the YARN service page in the EMR console, set the following parameters in yarn-env.sh or mapred-env.sh. All parameters take effect at the cluster level and require a component restart to apply.
Component | Parameter | Config file | When to adjust | Constraints |
ResourceManager |
| yarn-env.sh | High volume of small jobs causes ResourceManager slowness | Minimum value: 1024. Restart ResourceManager to apply. |
NodeManager |
| yarn-env.sh | Full garbage collection (GC) occurs because Shuffle Service occupies excessive NodeManager memory | Restart NodeManager to apply. |
WebAppProxyServer |
| yarn-env.sh | Default value is insufficient | Restart WebAppProxyServer to apply. |
TimelineServer |
| yarn-env.sh | Default value is insufficient | Restart TimelineServer to apply. |
TimelineServer |
| yarn-env.sh | Direct memory pressure on TimelineServer | Minimum value: 512m. Restart TimelineServer to apply. |
MRHistoryServer |
| mapred-env.sh | Default value is insufficient | Restart MRHistoryServer to apply. |
Cluster resource configuration for YARN
On the Configure tab of the YARN service page in the EMR console, set the following yarn-site.xml parameters.
Scheduler allocation limits
These parameters apply at the cluster level and control the per-container resource ceiling and floor. Changes require a ResourceManager restart.
Parameter | Description | Default | When to adjust |
| Maximum memory a single container can request from the scheduler | — | Increase for workloads that submit large single-container jobs. An excessively high value can cause resource fragmentation. |
| Minimum memory a single container can request from the scheduler | — | Rarely needs adjustment. |
| Maximum vCPUs a single container can request from the scheduler | 32 | Increase for workloads that submit large single-container jobs. An excessively high value can cause resource fragmentation. |
| Minimum vCPUs a single container can request from the scheduler | 1 | Rarely needs adjustment. |
NodeManager resource limits
These parameters apply at the node group level. When configuring them, select Node group level configuration in the EMR console—otherwise the change does not take effect. EMR supports mixed instance types across node groups, with all ECS instances within a node group sharing the same instance type. Configuring NodeManager resources at the node group level prevents smaller nodes from being overloaded and larger nodes from being underutilized.
Parameter | Description | Default | When to adjust |
| Total physical memory available to NodeManager for containers | Set at cluster creation based on instance type | Configure based on your cluster deployment. Restart NodeManager to apply. |
| Total vCPUs available to NodeManager for containers | Number of vCPUs for the instance type. For high-memory instance types, the default is twice the vCPU count of a regular-memory instance type. | Adjust based on workload characteristics. Restart NodeManager to apply. |
Relationship between scheduler and NodeManager memory
yarn.scheduler.maximum-allocation-mb must be greater than yarn.nodemanager.resource.memory-mb to ensure jobs are scheduled correctly.
Note the following when managing these two parameters:
EMR lets you configure both parameters when creating a cluster or when scaling out by adding a node group for the first time.
When you upgrade a node group's instance specifications or manually change
yarn.nodemanager.resource.memory-mb,yarn.scheduler.maximum-allocation-mbis not updated automatically. Update it manually as needed.The first time you configure
yarn.scheduler.maximum-allocation-mbfor a new node group, ResourceManager is not automatically restarted. Manually restart ResourceManager to apply the change.
Restarting ResourceManager may cause running jobs to fail. Perform the restart during off-peak hours.