Q&A about E-MapReduce

• Create a job

  An EMR job is essentially a set of configurations for running a job. An EMR job cannot be run directly. Instead, you need to specify the job JAR file, data input and output paths, and some run parameters in the configurations for running an EMR job. Provide a name for the set of configurations to complete the creation of a job. When you need to debug the job, an execution plan is required.

• Create an execution plan

  An execution plan associates a job with a cluster. You can use an execution plan to run a sequence of jobs manually or schedule an execution plan to run the jobs periodically. You can choose a cluster for jobs to run using an execution plan. The cluster can be an on-demand cluster or an existing cluster. An on-demand cluster is released automatically after the execution of all jobs is completed. You can view the running status of each execution of an execution plan on the corresponding running log page.

1. On the master node, switch to the hadoop user using the su command

   su hadoop

2. Then you can connect to a core node using SSH without entering a password.

   ssh emr-worker-1

3. Gain root privileges using the sudo command.

   sudo vi /etc/hosts

1. Go to the execution plan page, click Running log for the execution plan, which contains the logs you want to view.
2. On the running log page, find the execution record that you want. Click the corresponding cluster name in the Execute cluster column to view the cluster ID on the Details page.
3. Locate OSS://mybucket/emr/spark/cluserID under the OSS://mybucket/emr/spark directory.
4. Log files of jobs are stored in the corresponding folders that are created based on the job execution order ID under the OSS://mybucket/emr/spark/clusterID/jobs directory.

• For a cluster

  You can view the running time of each cluster on the cluster list. Formula: running time = completion time of cluster release - start time of cluster creation. Calculation of the running time starts when a cluster has been created and ends when the cluster has been released.

• For an execution plan
  You can view the running time of each execution plan on the running log page. The running time calculation policy is based on the cluster that the execution plan runs on.

  • If you choose a create-as-needed cluster for your execution plan, then each run of the execution plan involves creating a cluster, submitting jobs, and releasing the cluster. Formula: running time = consumption time for cluster creation + consumption time for running all jobs + consumption time for cluster release.
  • If you choose an existing cluster for your execution plan, then cluster creation and cluster release will not be involved in the run cycle. Therefore, running time = consumption time for running all jobs.
• For a job

  Specifically, we are referring to jobs that are included in an execution plan. For each running log of an execution plan, you can click View job list in the Operation column to see all jobs that are included in the corresponding execution plan. The running time calculation formula for each job is: running time = job execution completion time - start time of job execution. Job execution refers to jobs being scheduled to run on a Spark or Hadoop cluster.

CREATE EXTERNAL TABLE storage_log(content STRING) PARTITIONED BY (ds STRING)
    ROW FORMAT DELIMITED
    FIELDS TERMINATED BY '\t'
    STORED AS TEXTFILE
    LOCATION 'oss://log-124531712/biz-logs/airtake/pro/storage'; 
    hive> select * from storage_log;
    OK
    Time taken: 0.3 seconds
    No data has been retrieved using the external table.

alter table storage_log add partition(ds=123);                                                                                                                                             OK
    Time taken: 0.137 seconds
    hive> select * from storage_log;
    OK
    abcd    123
    efgh    123

<property>  
    <name>mapred.child.java.opts</name>  
    <value>-Xmx1024m -Djava.library.path=/usr/local/share/</value>  
  </property>  
  <property>  
    <name>mapreduce.admin.user.env</name>  
    <value>LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:/usr/local/lib</value>  
  </property>

• (Recommended) EMR provides MetaService, which allows you to access OSS data without an AccessKey so that you can specify the data source using the oss://bucket/object/path path format.
• (Not recommended) You can set the AccessKeyId, AccessKeySecret, and endpoint parameters on the Configuration object for a MapReduce job (for a Spark job, set these parameters on the SparkConf object). Or you can include the AccessKeyId, the AccessKeySecret, and the endpoint in a URI directly. For more information, see the Development preparation section.

• This error may occur when you use earlier versions of mysql-connector-java. Use the latest version of mysql-connector-java.
• In the job parameters, use —driver-class-path ossref://bucket/…/mysql-connector-java-[version].jar to load mysql-connector-java package. This issue may also occur if you directly package mysql-connector-javainto the Job JAR file.

• If you choose an instance with two vCPUs and 4 GB of memory as the master node, then the master node is prone to running out of memory. We recommend that you increase the memory of the master node.
• If you choose an instance with two vCPUs and 4 GB of memory as a slave (worker) node, set the parameters as follows when running a MapReduce or Hive job. For a MapReduce job, set the yarn.app.mapreduce.am.resource.mb parameter to 1024. For a Hive job, set the yarn.app.mapreduce.am.resource.mb parameter to 1024. This step is to prevent a job from being suspended due to an OOM error.

• High-availability cluster (service discovery mode)

  
  ! connect jdbc:hive2://emr-header-1:2181,emr-header-2:2181,emr-header-3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_HOST@EMR.${clusterId). COM

• High-availability cluster (directly connecting to a node)

  Connect to the emr-header-1 node.

  ! connect jdbc:hive2://emr-header-1:10000/;principal=hive/emr-header-1@EMR.${clusterId}. COM

  Connect to the emr-header-2 node.

  ! connect jdbc:hive2://emr-header-2:10000/;principal=hive/emr-header-2@EMR.${clusterId}. COM

• Non-HA cluster

  ! connect jdbc:hive2://emr-header-1:10000/;principal=hive/emr-header-1@EMR.${clusterId}. COM