If the vCPUs or memory of the master node group or a core node group in an E-MapReduce (EMR) cluster is insufficient, you can upgrade the node configurations of the node group. This topic describes how to upgrade node configurations.

Prerequisites

An EMR cluster is created. For more information, see Create a cluster.

Limits

  • You can upgrade only the node configurations of subscription clusters.
    Note If you use a high-availability cluster, only one of the master nodes is restarted. During the restart process, the other master node still runs to ensure normal services.
  • Elastic Compute Service (ECS) instances with local disks, such as instances of the d1 and i2 instance families, do not support configuration upgrade. You can only add nodes when vCPUs or memory is insufficient.
  • ECS instances that do not use local disks support configuration upgrade. However, they do not support configuration downgrade.

Procedure

  1. Go to the Upgrade dialog box.
    1. Log on to the Alibaba Cloud EMR console by using your Alibaba Cloud account.
    2. In the top navigation bar, select the region where your cluster resides and select a resource group based on your business requirements.
    3. Click the Cluster Management tab.
    4. On the Cluster Management page, find the cluster for which you want to upgrade node configurations and click Details in the Actions column.
    5. On the Cluster Overview page, choose Change Configuration > Upgrade.
  2. Upgrade node configurations.
    1. In the Upgrade dialog box, modify the configurations based on your business requirements.
    2. Read the terms of service and select E-MapReduce Service Terms.
    3. Click OK.
      An order is generated after a short period of time.
    4. Complete the payment.
    After you complete the payment, a message that indicates the upgrade is successful and you must restart the node group is displayed in the Instance Info section of the Cluster Overview page. Upgrade configuration
  3. In the Instance Info section of the Cluster Overview page, click the message The configuration is upgraded. Restart the node group for the new configuration to take effect.
    Notice When you restart a cluster, the ECS instances of the cluster are restarted. During the restart, the big data services deployed on the ECS instances are unavailable. Therefore, before you restart the cluster, make sure that the service unavailability does not affect your business.
    Restart
    Parameter Description
    Rolling Restart
    • Rolling Restart is selected by default. If you select Rolling Restart, the system restarts an ECS instance only after the previous ECS instance is restarted and all big data services deployed on it are restored.

      It takes about 5 minutes to restart an ECS instance.

    • If you clear Rolling Restart, the system restarts all ECS instances at the same time.
    Restart Scaled Nodes Only
    • Restart Scaled Nodes Only is selected by default. If you select Restart Scaled Nodes Only, the system restarts only the nodes whose disk space is expanded or configurations are upgraded.

      For example, if you have upgraded only the configurations of the master node group, the system restarts the nodes in the master node group and do not restart those in core node groups.

    • If you clear Restart Scaled Nodes Only, the system restarts all the nodes of the cluster.
  4. Click OK.
    In the Instance Info section of the Cluster Overview page, the message Restarting is displayed for the node group whose node configurations are upgraded. RestartAfter the message disappears, the restart is complete and the configurations have taken effect. You can log on to the cluster to verify the configurations.
    Note
    • If you have upgraded only vCPU configurations, no more operations are required.
    • If you have upgraded only memory configurations or both vCPU and memory configurations, we recommend that you perform steps in Modify configurations for YARN. This way, the YARN service can use the added resources.

Modify configurations for YARN

  1. In the left-side navigation pane, choose Cluster Service > YARN.
  2. Modify vCPU configurations.
    1. On the YARN service page, click the Configure tab.
    2. In the Configuration Filter section, search for the yarn.nodemanager.resource.cpu-vcores parameter. Then, change its value based on your business requirements.
      In compute-intensive scenarios, we recommend that you set this parameter to a value that is equal to the number of vCPUs of each ECS instance. In scenarios that involve both compute-intensive and I/O intensive tasks, we recommend that you set this parameter to a value that ranges from the number of vCPUs of each ECS instance to twice the number. For example, if you use 32-vCPU ECS instances in compute-intensive scenarios, set this parameter to 32. If you use 32-vCPU ECS instances in scenarios that involve both compute-intensive and I/O intensive tasks, set this parameter to a value that ranges from 32 to 64.
  3. Modify memory configurations.
    1. On the YARN service page, click the Configure tab.
    2. In the Configuration Filter section, search for the yarn.nodemanager.resource.memory-mb parameter. Then, change its value to the memory size of each ECS instance multiplied by 0.8. Unit: MiB.
      For example, if the memory size of each ECS instance is 32 GiB, set this parameter to 26214.
  4. Save the configurations.
    1. Click Save in the upper-right corner of the Service Configuration section.
    2. In the Confirm Changes dialog box, specify Description and click OK.
  5. Issue the configurations.
    1. On the YARN service page, choose Actions > Configure All Components.
    2. In the Cluster Activities dialog box, specify Description and click OK.
    3. In the Confirm message, click OK.
    You can click History in the upper-right corner to view the status of the Configure YARN task. After the task enters the Successful state, perform the following step to restart the YARN service.
  6. Restart the YARN service.
    1. On the YARN service page, choose Actions > Restart All Components.
    2. In the Cluster Activities dialog box, specify Description and click OK.
    3. In the Confirm message, click OK.
    You can click History in the upper-right corner to view the status of the Restart YARN task. After the task enters the Successful state, the YARN service is restarted.