edit-icon download-icon

E-MapReduce Cluster Operation Guide

Last Updated: Jul 24, 2018

This document provides instructions on how to operate E-MapReduce clusters.

Note: In E-MapReduce cluster version that is 3.2 or later, you can do most operations through WebUI.

General environmental variables

Enter the env command on the cluster. The latest version of environmental variable configuration on the cluster is displayed. (The following configuration is only for your reference.)

  1. JAVA_HOME=/usr/lib/jvm/java
  2. HADOOP_HOME=/usr/lib/hadoop-current
  3. HADOOP_CLASSPATH=/usr/lib/hbase-current/lib/*:/usr/lib/tez-current/*:/usr/lib/tez-current/lib/*:/etc/emr/tez-conf:/usr/lib/hbase-current/lib/*:/usr/lib/tez-current/*:/usr/lib/tez-current/lib/*:/etc/emr/tez-conf:/opt/apps/extra-jars/*:/opt/apps/extra-jars/*
  4. HADOOP_CONF_DIR=/etc/emr/hadoop-conf
  5. SPARK_HOME=/usr/lib/spark-current
  6. SPARK_CONF_DIR=/etc/emr/spark-conf
  7. HBASE_HOME=/usr/lib/hbase-current
  8. HBASE_CONF_DIR=/etc/emr/hbase-conf
  9. HIVE_HOME=/usr/lib/hive-current
  10. HIVE_CONF_DIR=/etc/emr/hive-conf
  11. PIG_HOME=/usr/lib/pig-current
  12. PIG_CONF_DIR=/etc/emr/pig-conf
  13. TEZ_HOME=/usr/lib/tez-current
  14. TEZ_CONF_DIR=/etc/emr/tez-conf
  15. ZEPPELIN_HOME=/usr/lib/zeppelin-current
  16. ZEPPELIN_CONF_DIR=/etc/emr/zeppelin-conf
  17. HUE_HOME=/usr/lib/hue-current
  18. HUE_CONF_DIR=/etc/emr/hue-conf
  19. PRESTO_HOME=/usr/lib/presto-current
  20. PRESTO_CONF_DIR=/etc/emr/presot-conf

Start or Stop a Service Process through WebUI

You can start, stop or restart the service process running on the specified ECS instance on the Web interface of E-MapReduce. The operation procedure for each service are similar. Here, we take HDFS as an example to introduce how to start, stop, and restart the DataNode process on a emr-worker-1 instance. Please take the following steps:

  1. On Cluster Management page, click Manage after the cluster you want to operate in the Operation column.
  2. Click HDFS in the service list to enter the HDFS Management page.
  3. Click the Component Topology tab to see the list of service processes running on all instances in the cluster.
  4. Click Start in the Operation column for the DataNode process running on the emr-worker-1 intance, enter record information in the prompt box, and then click OK.

    After about 10 seconds, refresh the page, you can see that the status changes from STOPPED to STARTED in Components Status column.HDFS

  5. With the process started, click Restart in the Operation column, enter record information in the prompt box, and then click OK.

    After about 40 seconds, the process can restart successfully.

  6. Click Stop in the Operation column, enter record information in the prompt box, and then click OK.

    After about 10 seconds, refresh the page, you can see that the status changes from STARTED to STOPPED in Components Status column.

Batch Operations through WebUI

In addition to operating the service process running on the specified ECS instance, you can also perform batch operations. We take HDFS as an example to introduce how to restart DataNode processes on all instances.

  1. On Cluster Management page, click Manage after the cluster you want to operate in the Operation column.
  2. Click Operation in the HDFS row in the service list.
  3. Select RESTART DataNode in the menu, enter record information in the prompt box, and then click OK.

    You can click HDFS and enter the Component Topology tab to view the status of each process.

    Note: After the Rolling Start is performed, the normal start can no longer be performed, otherwise errors may occur.DataNode_restart

Start or Stop a Service Process through CLI

YARN

Operation account: hadoop

  • ResourceManager (Master node)
    1. //Start
    2. /usr/lib/hadoop-current/sbin/yarn-daemon.sh start resourcemanager
    3. //Stop
    4. /usr/lib/hadoop-current/sbin/yarn-daemon.sh stop resourcemanager
  • NodeManager (Core node)
    1. //Start
    2. /usr/lib/hadoop-current/sbin/yarn-daemon.sh start nodemanager
    3. //Stop
    4. /usr/lib/hadoop-current/sbin/yarn-daemon.sh stop nodemanager
  • JobHistoryServer (Master node)
    1. //Start
    2. /usr/lib/hadoop-current/sbin/mr-jobhistory-daemon.sh start historyserver
    3. //Stop
    4. /usr/lib/hadoop-current/sbin/mr-jobhistory-daemon.sh stop historyserver
  • WebProxyServer (Master node)
    1. //Start
    2. /usr/lib/hadoop-current/sbin/yarn-daemon.sh start proxyserver
    3. //Stop
    4. /usr/lib/hadoop-current/sbin/yarn-daemon.sh stop proxyserver

HDFS

Operation account: hdfs

  • NameNode (Master node)
    1. //Start
    2. /usr/lib/hadoop-current/sbin/hadoop-daemon.sh start namenode
    3. //Stop
    4. /usr/lib/hadoop-current/sbin/hadoop-daemon.sh stop namenode
  • DataNode (Core node)

    1. //Start
    2. /usr/lib/hadoop-current/sbin/hadoop-daemon.sh start datanode
    3. //Stop
    4. /usr/lib/hadoop-current/sbin/hadoop-daemon.sh stop datanode

    Hive

    Operation account: hadoop

  • MetaStore (Master node)

    1. //Start (The memory here can be extended based on requirements.)
    2. HADOOP_HEAPSIZE=512 /usr/lib/hive-current/bin/hive --service metastore >/var/log/hive/metastore.log 2>&1 &
  • HiveServer2 (Master node)
    1. //Start
    2. HADOOP_HEAPSIZE=512 /usr/lib/hive-current/bin/hive --service hiveserver2 >/var/log/hive/hiveserver2.log 2>&1 &

HBase

Operation account: hdfs

NOTE: Select HBase component first before using the following method to start processes; otherwise, errors will occur.

  • HMaster (Master node)
    1. //Start
    2. /usr/lib/hbase-current/bin/hbase-daemon.sh start master
    3. //Restart
    4. /usr/lib/hbase-current/bin/hbase-daemon.sh restart master
    5. //Stop
    6. /usr/lib/hbase-current/bin/hbase-daemon.sh stop master
  • HRegionServer (Core node)
    1. //Start
    2. /usr/lib/hbase-curren/bin/hbase-daemon.sh start regionserver
    3. //Restart
    4. /usr/lib/hbase-curren/bin/hbase-daemon.sh restart regionserver
    5. //Stop
    6. /usr/lib/hbase-curren/bin/hbase-daemon.sh stop regionserver
  • ThriftServer (Master node)
    1. //Start
    2. /usr/lib/hbase-current/bin/hbase-daemon.sh start thrift -p 9099 >/var/log/hive/thriftserver.log 2>&1 &
    3. //Stop
    4. /usr/lib/hbase-current/bin/hbase-daemon.sh stop thrift

Hue

Operation account: hadoop

  1. //Start
  2. su -l root -c "${HUE_HOME}/build/env/bin/supervisor >/dev/null 2>&1 &"
  3. //Stop
  4. ps axu | grep hue //Find all hue processes.
  5. kill -9 huepid //Kill all the hue processes found.

Zeppelin

Operation account: hadoop

  1. //Start (The memory here can be extended based on requirements.)
  2. su -l root -c "ZEPPELIN_MEM=\"-Xmx512m -Xms512m\" ${ZEPPELIN_HOME}/bin/zeppelin-daemon.sh start"
  3. //Stop
  4. su -l root -c "${ZEPPELIN_HOME}/bin/zeppelin-daemon.sh stop"

Presto

Operation account: hdfs

  • PrestoServer (Master node)
    1. //Start
    2. /usr/lib/presto-current//bin/launcher --config=/usr/lib/presto-current/etc/worker-config.properties start
    3. //Stop
    4. /usr/lib/presto-current//bin/launcher --config=/usr/lib/presto-current/etc/worker-config.properties stop
    *(Core node)
    1. //Start
    2. /usr/lib/presto-current//bin/launcher --config=/usr/lib/presto-current/etc/coordinator-config.properties start
    3. //Stop
    4. /usr/lib/presto-current//bin/launcher --config=/usr/lib/presto-current/etc/coordinator-config.properties stop

Batch Operations through CLI

You may write a script command to operate worker (core) nodes in batches.In E-MapReduce clusters, the connection from the master node to all worker nodes is established over SSH using the hadoop and hdfs accounts.

For example, if you need to stop nodemanager for all the 10 worker nodes in batches, you can do it as follows:

  1. for i in `seq 1 10`;do ssh emr-worker-$i /usr/lib/hadoop-current/sbin/yarn-daemon.sh stop nodemanager;done
Thank you! We've received your feedback.