E-MapReduce Cluster Operation Guide - Developer Guide| Alibaba Cloud Documentation Center

This document provides instructions on how to operate E-MapReduce clusters.

Note: In E-MapReduce cluster version that is 3.2 or later, you can do most operations through WebUI.

General environmental variables

Enter the env command on the cluster. The latest version of environmental variable configuration on the cluster is displayed. (The following configuration is only for your reference.)

JAVA_HOME=/usr/lib/jvm/java
HADOOP_HOME=/usr/lib/hadoop-current
HADOOP_CLASSPATH=/usr/lib/hbase-current/lib/*:/usr/lib/tez-current/*:/usr/lib/tez-current/lib/*:/etc/emr/tez-conf:/usr/lib/hbase-current/lib/*:/usr/lib/tez-current/*:/usr/lib/tez-current/lib/*:/etc/emr/tez-conf:/opt/apps/extra-jars/*:/opt/apps/extra-jars/*
HADOOP_CONF_DIR=/etc/emr/hadoop-conf
SPARK_HOME=/usr/lib/spark-current
SPARK_CONF_DIR=/etc/emr/spark-conf
HBASE_HOME=/usr/lib/hbase-current
HBASE_CONF_DIR=/etc/emr/hbase-conf
HIVE_HOME=/usr/lib/hive-current
HIVE_CONF_DIR=/etc/emr/hive-conf
PIG_HOME=/usr/lib/pig-current
PIG_CONF_DIR=/etc/emr/pig-conf
TEZ_HOME=/usr/lib/tez-current
TEZ_CONF_DIR=/etc/emr/tez-conf
ZEPPELIN_HOME=/usr/lib/zeppelin-current
ZEPPELIN_CONF_DIR=/etc/emr/zeppelin-conf
HUE_HOME=/usr/lib/hue-current
HUE_CONF_DIR=/etc/emr/hue-conf
PRESTO_HOME=/usr/lib/presto-current
PRESTO_CONF_DIR=/etc/emr/presot-conf

Start or Stop a Service Process through WebUI

You can start, stop or restart the service process running on the specified ECS instance on the Web interface of E-MapReduce. The operation procedure for each service are similar. Here, we take HDFS as an example to introduce how to start, stop, and restart the DataNode process on a emr-worker-1 instance. Please take the following steps:

On Cluster Management page, click Manage after the cluster you want to operate in the Operation column.
Click HDFS in the service list to enter the HDFS Management page.
Click the Component Topology tab to see the list of service processes running on all instances in the cluster.
Click Start in the Operation column for the DataNode process running on the emr-worker-1 intance, enter record information in the prompt box, and then click OK.

After about 10 seconds, refresh the page, you can see that the status changes from STOPPED to STARTED in Components Status column.
With the process started, click Restart in the Operation column, enter record information in the prompt box, and then click OK.

After about 40 seconds, the process can restart successfully.
Click Stop in the Operation column, enter record information in the prompt box, and then click OK.

After about 10 seconds, refresh the page, you can see that the status changes from STARTED to STOPPED in Components Status column.

Batch Operations through WebUI

In addition to operating the service process running on the specified ECS instance, you can also perform batch operations. We take HDFS as an example to introduce how to restart DataNode processes on all instances.

On Cluster Management page, click Manage after the cluster you want to operate in the Operation column.
Click Operation in the HDFS row in the service list.
Select RESTART DataNode in the menu, enter record information in the prompt box, and then click OK.

You can click HDFS and enter the Component Topology tab to view the status of each process.

Note: After the Rolling Start is performed, the normal start can no longer be performed, otherwise errors may occur.

Start or Stop a Service Process through CLI

YARN

Operation account: hadoop

ResourceManager (Master node)

//Start
/usr/lib/hadoop-current/sbin/yarn-daemon.sh start resourcemanager
//Stop
/usr/lib/hadoop-current/sbin/yarn-daemon.sh stop resourcemanager

NodeManager (Core node)

//Start
/usr/lib/hadoop-current/sbin/yarn-daemon.sh start nodemanager
//Stop
/usr/lib/hadoop-current/sbin/yarn-daemon.sh stop nodemanager

JobHistoryServer (Master node)

//Start
/usr/lib/hadoop-current/sbin/mr-jobhistory-daemon.sh start historyserver
//Stop
/usr/lib/hadoop-current/sbin/mr-jobhistory-daemon.sh stop historyserver

WebProxyServer (Master node)

//Start
/usr/lib/hadoop-current/sbin/yarn-daemon.sh start proxyserver
//Stop
/usr/lib/hadoop-current/sbin/yarn-daemon.sh stop proxyserver

HDFS

Operation account: hdfs

NameNode (Master node)

//Start
/usr/lib/hadoop-current/sbin/hadoop-daemon.sh start namenode
//Stop
/usr/lib/hadoop-current/sbin/hadoop-daemon.sh stop namenode

DataNode (Core node)

//Start
/usr/lib/hadoop-current/sbin/hadoop-daemon.sh start datanode
//Stop
/usr/lib/hadoop-current/sbin/hadoop-daemon.sh stop datanode

Hive

Operation account: hadoop

MetaStore (Master node)

//Start (The memory here can be extended based on requirements.)
HADOOP_HEAPSIZE=512 /usr/lib/hive-current/bin/hive --service metastore >/var/log/hive/metastore.log 2>&1 &

HiveServer2 (Master node)

//Start
HADOOP_HEAPSIZE=512 /usr/lib/hive-current/bin/hive --service hiveserver2 >/var/log/hive/hiveserver2.log 2>&1 &

HBase

Operation account: hdfs

NOTE: Select HBase component first before using the following method to start processes; otherwise, errors will occur.

HMaster (Master node)

//Start
/usr/lib/hbase-current/bin/hbase-daemon.sh start master
//Restart
/usr/lib/hbase-current/bin/hbase-daemon.sh restart master
//Stop
/usr/lib/hbase-current/bin/hbase-daemon.sh stop master

HRegionServer (Core node)

//Start
/usr/lib/hbase-curren/bin/hbase-daemon.sh start regionserver
//Restart
/usr/lib/hbase-curren/bin/hbase-daemon.sh restart regionserver
//Stop
/usr/lib/hbase-curren/bin/hbase-daemon.sh stop regionserver

ThriftServer (Master node)

//Start
/usr/lib/hbase-current/bin/hbase-daemon.sh start thrift -p 9099 >/var/log/hive/thriftserver.log 2>&1 &
//Stop
/usr/lib/hbase-current/bin/hbase-daemon.sh stop thrift

Hue

Operation account: hadoop

//Start
su -l root -c "${HUE_HOME}/build/env/bin/supervisor >/dev/null 2>&1 &"
//Stop
ps axu | grep hue     //Find all hue processes.
kill -9 huepid        //Kill all the hue processes found.

Zeppelin

Operation account: hadoop

//Start (The memory here can be extended based on requirements.)
su -l root -c "ZEPPELIN_MEM=\"-Xmx512m -Xms512m\" ${ZEPPELIN_HOME}/bin/zeppelin-daemon.sh start"
//Stop
su -l root -c "${ZEPPELIN_HOME}/bin/zeppelin-daemon.sh stop"

Presto

Operation account: hdfs

PrestoServer (Master node)

//Start
/usr/lib/presto-current//bin/launcher --config=/usr/lib/presto-current/etc/worker-config.properties start
//Stop
/usr/lib/presto-current//bin/launcher --config=/usr/lib/presto-current/etc/worker-config.properties stop

*(Core node)

//Start
/usr/lib/presto-current//bin/launcher --config=/usr/lib/presto-current/etc/coordinator-config.properties start
//Stop
/usr/lib/presto-current//bin/launcher --config=/usr/lib/presto-current/etc/coordinator-config.properties stop

Batch Operations through CLI

You may write a script command to operate worker (core) nodes in batches.In E-MapReduce clusters, the connection from the master node to all worker nodes is established over SSH using the hadoop and hdfs accounts.

For example, if you need to stop nodemanager for all the 10 worker nodes in batches, you can do it as follows:

  for i in `seq 1 10`;do ssh emr-worker-$i /usr/lib/hadoop-current/sbin/yarn-daemon.sh stop nodemanager;done