edit-icon download-icon

Hive job configuration

Last Updated: May 04, 2018

When users are applying for a cluster in E-MapReduce, they are provided with a Hive environment by default. Users can directly create and operate their tables and data by using Hive. The procedure is as follows.

  1. Prepare the Hive script in advance, for example:

    1. USE DEFAULT;
    2. set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
    3. set mapreduce.job.maps=12;
    4. set mapreduce.job.reduces=6;
    5. set hive.stats.autogather=false;
    6. DROP TABLE uservisits;
    7. CREATE EXTERNAL TABLE IF NOT EXISTS uservisits (sourceIP STRING,destURL STRING,visitDate STRING,adRevenue DOUBLE,user
    8. Agent STRING,countryCode STRING,languageCode STRING,searchWord STRING,duration INT ) ROW FORMAT DELIMITED FIELDS TERMI
    9. NATED BY ',' STORED AS SEQUENCEFILE LOCATION '/HiBench/Aggregation/Input/uservisits';
    10. DROP TABLE uservisits_aggre;
    11. CREATE EXTERNAL TABLE IF NOT EXISTS uservisits_aggre ( sourceIP STRING, sumAdRevenue DOUBLE) STORED AS SEQUENCEFILE LO
    12. CATION '/HiBench/Aggregation/Output/uservisits_aggre';
    13. INSERT OVERWRITE TABLE uservisits_aggre SELECT sourceIP, SUM(adRevenue) FROM uservisits GROUP BY sourceIP;
  2. Save this script into a script file, such as uservisits_aggre_hdfs.hive, and then upload it to an OSS directory (for example, oss://path/to/uservisits_aggre_hdfs.hive).

  3. Log on to Alibaba Cloud E-MapReduce Console Job List.

  4. Click Create a job in the upper right corner to enter the job creation page.

  5. Enter the job name.

  6. Select the Hive job type to create a Hive job. This type of job is submitted in the background by using the following process:

    1. hive [user provided parameters]
  7. Enter the Parameters in the option box with parameters subsequent to Hive commands. For example, if it is necessary to use a Hive script uploaded to OSS, the following must be entered:

    1. -f ossref://path/to/uservisits_aggre_hdfs.hive

    You can also click Select OSS path to view and select from OSS, the system will automatically complete the path of Hive script on OSS. Switch the Hive script prefix to ossref (click Switch resource type) to guarantee this file is properly downloaded by E-MapReduce.

  8. Select the policy for failed operations.

  9. Click OK to complete the Hive job configuration.

Thank you! We've received your feedback.