Frequently asked questions about Alibaba Cloud Elastic MapReduce (EMR)

A: In EMR, two steps are required to run a job:

• Create a job

  In EMR, creating a “job” creates a "job running configuration" which cannot be run directly. If you have created a "job" in EMR, you have created a "configuration about how to run the job." The configuration contains the Job JAR to be run for the job, the input and output addresses of data and certain running parameters. After you create such a configuration and provide a name for it, a job is defined. However, when you want to debug the running job, the execution plan is required.

• Create an execution plan

  The execution plan associates the job with the cluster. Through the execution plan, we can combine multiple jobs into a job sequence and prepare a running cluster for the job. The execution can also automatically create a temporary cluster or associate the job with an existing cluster. The execution plan also helps to configure a periodical execution plan for the job sequence and automatically releases the cluster after the task is completed. You can also view the execution status and log entries on the execution record list.

A: Use the following steps:

1. Switch to the hadoop account on the master node.

   su hadoop

2. Log on to the corresponding core node with password-free SSH authentication.

   ssh emr-worker-1

3. Get root privileges through the sudo command.

   sudo vi /etc/hosts

A: users can also find all the log files directly from the OSS and download them. However, since OSS does not directly view the logs, it can be much more difficult to use it. If you have enabled logging and specified a log location for OSS, how can you find the job log entries? For example, this path of OSS://mybucket/emr/spark.

1. 1. Go to the execution plan page, find the corresponding execution plan and click View Records to enter the execution log page.
2. 2. Find the specific execution log entry on the execution log page, such as the last execution log entry. Click the corresponding Execution Cluster to view the ID of the execution cluster.
3. 3. Then search for the cluster ID directory OSS://mybucket/emr/spark/under OSS://mybucket/emr/spark directory.
4. 4. There will be multiple directories under OSS://mybucket/emr/spark/cluster ID/jobs based on the execution ID of the job, and each directory stores the running log file of the job.

A: Three following timing policies are available:

• The timing policy of the cluster

  In the cluster list, you can see the running time of every cluster. The formula for calculating the running time is: Running time = Time when the cluster is released - Time when the cluster is established. Once a cluster has been created, the timing starts until the end of the life cycle of the cluster.

• The timing policy of the execution plan
  In the running log list of the execution plan, you can see the running time of every execution plan, and the timing policy can be summarized into two situations:

  • 1. If the execution plan is executed on demand, the running process of every execution log involves cluster creation, job submission, job running, and cluster release. The formula for calculating the on-demand execution plan is: Running time = The time when the cluster is created + The total time used for running all the jobs in the execution plan + The time when the cluster is released.
  • 2. If the execution plan is associated with an existing cluster, the entire execution cycle does not involve the creation and release of a cluster. The running time is the total time used for running all the jobs in the execution plan.
• The timing policy of the running job

  The specified job refers to the job assigned to the execution plan. Click the View Job List to the right of the running log of every execution plan to view the job. The formula for calculating the runtime of every job is: Running time = The actual time when the job stops running - The actual time when the job starts running. The specified time when the job starts or stops refers to the time when the job is scheduled to run or stopped running by the Spark or Hadoop cluster.

A: For example:

CREATE EXTERNAL TABLE storage_log(content STRING) PARTITIONED BY (ds STRING)
    ROW FORMAT DELIMITED
    FIELDS TERMINATED BY '\t'
    STORED AS TEXTFILE
    LOCATION 'oss://log-124531712/biz-logs/airtake/pro/storage'; 
    hive> select * from storage_log;
    OK
    Time taken: 0.3 seconds
    The external table that you have created contains no data.

Hive does not automatically bind itself with the partition directory of the specified directory. You need to bind them manually, for example:

alter table storage_log add partition(ds=123);                                                                                                                                             OK
    Time taken: 0.137 seconds
    hive> select * from storage_log;
    OK
    abcd    123
    efgh    123

A: You can use multiple methods, including the following example: Modify the mapred-site.xml file, for example:

<property>  
    <name>mapred.child.java.opts</name>  
    <value>-Xmx1024m -Djava.library.path=/usr/local/share/</value>  
  </property>  
  <property>  
    <name>mapreduce.admin.user.env</name>  
    <value>LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:/usr/local/lib</value>  
  </property>

This URI is used for specifying input/output data sources in the job, and is similar to hdfs://. Follow this procedure when you perform operations on OSS data:

• (Recommended) EMR provides MetaService, which allows you to access OSS data without AssessKey, and directly write to the OSS path: // bucket/Object/path.
• (Not recommended) You can configure AccessKeyId, AccessKeySecret, and endpoint to Configuration (SparkConf in Spark jobs, Configuration in MR jobs), or you can directly specify AccessKeyId, AccessKeySecret, and endpoint in the URL. For more information, see the Development preparation section.

A:

• 1. Earlier versions of mysql-connector-java may encounter such issues. Update mysql-connector-java to the latest version.
• 2. In the job parameters, use—driver-class-path ossref://bucket/…/mysql-connector-java-[version].jar to load mysql-connector-java package. This issue may also occur if you directly package mysql-connector-javainto the Job Jar.

A:

• If you choose two-core 4 GB machines for the master node, the memory of the master node is heavily loaded. This may cause insufficient memory issues. We recommend that you expand the memory capacity of the master node.
• If you choose two-core 4 GB machines for the slave nodes, adjust the parameters when you run MR or Hive jobs. For MR jobs, add the -D yarn.app.mapreduce.am.resource.mb=1024 parameter. For Hive jobs, add the set yarn.app.mapreduce.am.resource.mb=1024 parameter. This can prevent jobs to be suspended.

A:

• HA cluster (discovery mode)

  
  ! connect jdbc:hive2://emr-header-1:2181,emr-header-2:2181,emr-header-3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_HOST@EMR.${clusterId). COM

• HA cluster (directly connected to a machine)

  Connect to emr-header-1.

  ! connect jdbc:hive2://emr-header-1:10000/;principal=hive/emr-header-1@EMR.${clusterId}. COM

  Connect to emr-header-2.

  ! connect jdbc:hive2://emr-header-2:10000/;principal=hive/emr-header-2@EMR.${clusterId}. COM

• Non-HA cluster

  ! connect jdbc:hive2://emr-header-1:10000/;principal=hive/emr-header-1@EMR.${clusterId}. COM