You may need to upgrade the specification of the CPU and memory of the master node when using an EMR cluster.
- Only Subscription EMR clusters support configuration upgrades.
- Local disk-based instances do not support configuration upgrades. You can only improve cluster performance by increasing the number of nodes.
- Non-local disk-based instances support configuration upgrades and do not support specification downgrades.
- On the Cluster Management page, click View Details in the Actions column for the cluster that you want to upgrade node specification for.
- On the Cluster Overview page, click Resource Allocation. From the drop-down list, select Upgrade.
- Modify the node configuration as needed.
- Click OK and wait until the order is generated.
- Pay the order.
- Go to the Cluster Overview page, refresh the page, and confirm that the node configuration has been upgraded. For example, CPU: 4 cores and Memory: 16 GB. A message appears in the instance group section as shown in the following figure.
- Click The specification upgrades are complete. Restart the servers for the upgrades to take effect. A dialog box appears as shown in the following figure.
- Click OK.
- When you restart a cluster, the ECS instances of the cluster are restarted. During restart, the big data services deployed on ECS instances are not available. Make sure that the cluster restart does not impact your business.
- Rolling Start
- If you select Rolling Start, ECS instances restart one by one after the previous ECS instance completes restarting and all big data services deployed on this instance are restored. Restarting an instance takes about five minutes.
- If you clear Rolling Start, all ECS instances restart at the same time.
- Only Restart Updated Nodes
- An updated node (a core node or the master node) refers to a node of which the configuration has been updated.
- If you select Only Restart Updated Nodes, nodes that have not been upgraded do not restart. For example, if only the configurations of nodes in the core group are updated, then only the ECS instances in the core group are restarted. The ECS instances in the master group are not restarted.
- If you clear Only Restart Updated Nodes, all nodes are restarted. That is to say, all instances of the cluster are restarted.
- In the progress of restarting, the Instance Group Restarting prompt appears next to the instance group name as shown in the following figure.
- When the prompt disappears, the configurations are complete. You can log on to the cluster for verification.
- If you only update the CPU configurations, you can skip this step. If you update the memory configurations, you need to modify the cluster service configurations for YARN to use new resources. If you update both, you need to modify both configurations. Procedure:
- On the Clusters and Services page, click YARN.
- Modify the CPU configurations
Go to the Service page, click Configure and locate the yarn.nodemanager.resource.cpu-vcores configuration item. For computing-intensive scenarios, we recommend that you set the ratio to 1:1. For mixed scenarios, we recommend that you set the ratio to less than 1:2. For example, a 32 vCPU-based compute node that is intended for computing-intensive scenarios requires large amounts of configurations for regular expressions. You can set as follows.
yarn.nodemanager.resource.cpu-vcores=32. If the compute node is intended for mixed scenarios, you can set the value of yarn.nodemanager.resource.cpu-vcores to a number between 32 and 64.
- Modify the memory configurations
Go to the Service page, click the Configure tab and locate the yarn.nodemanager.resource.memory-mb configuration item. Set the value of the configuration item to 0.8 times the machine memory. The unit is megabytes. For example, if the memory size is 32 GB, the setting is as follows.
- Click Save in the upper-right corner to save the configurations.
- Click Actions and select Configure All Components from the drop-down list.
- Click History. Wait until Successful appears in the Status column.
- Click Actions and select RESTART All Components from the drop-down list.
- Click History. When Successful appears in the Status column, you can uses new resources in YARN.