Hive job configuration

Last Updated: May 05, 2017

When users are applying for a cluster in E-MapReduce, they are provided with a Hive environment by default. Users can directly create and perform their table and data by using Hive. Operation steps are as follows:

  1. Prepare the Hive script in advance, for example:

    1. USE DEFAULT;
    2. set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
    3. set mapreduce.job.maps=12;
    4. set mapreduce.job.reduces=6;
    5. set hive.stats.autogather=false;
    6. DROP TABLE uservisits;
    7. CREATE EXTERNAL TABLE IF NOT EXISTS uservisits (sourceIP STRING,destURL STRING,visitDate STRING,adRevenue DOUBLE,user
    8. Agent STRING,countryCode STRING,languageCode STRING,searchWord STRING,duration INT ) ROW FORMAT DELIMITED FIELDS TERMI
    9. NATED BY ',' STORED AS SEQUENCEFILE LOCATION '/HiBench/Aggregation/Input/uservisits';
    10. DROP TABLE uservisits_aggre;
    11. CREATE EXTERNAL TABLE IF NOT EXISTS uservisits_aggre ( sourceIP STRING, sumAdRevenue DOUBLE) STORED AS SEQUENCEFILE LO
    12. CATION '/HiBench/Aggregation/Output/uservisits_aggre';
    13. INSERT OVERWRITE TABLE uservisits_aggre SELECT sourceIP, SUM(adRevenue) FROM uservisits GROUP BY sourceIP;
  2. Save this script into a script file, such as “uservisits_aggre_hdfs.hive”, and then upload it to an OSS directory (for example: oss://path/to/uservisits_aggre_hdfs.hive).

  3. Log on to Alibaba Cloud E-MapReduce Console Job List.

  4. Click Create a job in the top right corner to enter the job creation page.

  5. Input the job name.

  6. Select the Hive job type to create a Hive job. This type of job is submitted in the background via the following process:

    1. hive [user provided parameters]
  7. Fill in the Parameters option box with parameters subsequent to Hive commands. For example, if it is necessary to use a Hive script uploaded to OSS, the following should be filled in:

    1. -f ossref://path/to/uservisits_aggre_hdfs.hive

    You can also click Select OSS path to view and select from OSS, the system will automatically complete the absolute path of Hive script on OSS. Switch the Hive script prefix to “ossref” (click Switch resource type) to guarantee E-MapReduce downloading this file correctly.

  8. Select the policy for failed operations.

  9. Click OK to complete the Hive job definition.

Thank you! We've received your feedback.