In actual use, the CPU or memory of cluster nodes, especially master nodes, may be insufficient. Currently, E-MapReduce does not support direct upgrade configuration. We recommend that you upgrade nodes configuration in the following way.
- Confirm the cluster to be upgraded.
- Go to the EMR console, click Manage in the dashboard to enter the Cluster Management page.
- Click Upgrade Configuration on the right corner.
- Modify the upgrade configuration of nodes to be upgraded.
- Click OK. Wait for a while, the order is generating.
- Pay for the order.
Return to the Cluster Management page, refresh the page to make sure that the node configuration has become the target specification, for example, CPU is 4 core and memory is 16G.
- Go to the ECS console, find the upgraded instances and restart them one by one.
- Modify the cluster configuration so that Yarn can use the new resources.
- Modify the yarn-site.xml file.
- Change the value of yarn.nodemanager.resource.memory-mb to machine memory * 0.8 , the unit is MB. Change the value of yarn.scheduler.maximum-allocation-mb to machine memory * 0.8, the unit is MB. For example, in my new configuration, the memory is 32 GB:
If your cluster does not support page modification, you must log on to the node, and modify the corresponding configuration values in the
/etc/emr/hadoop-conf/yarn-site.xmlfile for each node.
- Restart the Yarn service. Generally, you only must restart the worker node. However, after the restart, the Node Manger port is changed. Therefore, we recommend that you restart the Resource Manager.
- pen a ticket to the E-MapReduce team for providing information about the new node configuration. The E-MapReduce team will synchronize the configuration to guarantee normal cluster operating.