Object Storage Service:Use Hive on an EMR cluster to process data stored in OSS-HDFS

Procedure

1. Log on to the E-MapReduce console. In the left-side navigation pane, click EMR on ECS and create an EMR cluster.

   When you create the EMR cluster, make sure that Product Version is EMR-3.46.2 or later, or EMR-5.12.2 or later, and Root Storage Directory of Cluster is set to an OSS-HDFS-enabled bucket. Use the defaults for other parameters. For details, see Create a cluster.

2. Run the following command on the terminal to connect to Hive.

   For other connection methods, see Hive connection methods.

   hive

3. Run the following command on the terminal to connect to Hive.

   For other connection methods, see Hive connection methods.

   hive

4. Add partitions to an existing table.

   You can add partitions to an existing table to store the data of the table in smaller units. You can specify query conditions based on partitions. This way, only partitions that meet the specified conditions are scanned and the query performance is improved.

   • Command syntax

     ALTER TABLE <table_name> ADD [IF NOT EXISTS] PARTITION <pt_spec> [PARTITION <pt_spec> PARTITION <pt_spec>...] LOCATION 'location';

     The following table describes the parameters in the preceding command.

     Parameter	Required	Description
     table_name	Yes	The name of the table to which you want to add partitions.
     IF NOT EXISTS	No	The parameter that causes the error to be skipped if the table contains a partition with the same name. If the IF NOT EXISTS parameter is not specified and a partition whose name is the same as the partition that you want to add already exists, an operation failure occurs and an error is returned.
     pt_spec	Yes	The partitions that you want to add. The value of this parameter is in the (partition_col1 = partition_col_value1, partition_col2 = partition_col_value2, ...) format. In this format, partition_col is the names of partition key columns, and partition_col_value is their values. The names of partition key columns are case-insensitive and their values are case-sensitive.
     location	Yes	The OSS path that is used to store data in a partition.

   • Examples

     The following sample code provides an example on how to add a partition to a table named sale_detail to store the sale records in the China (Hangzhou) region in December 2021 and specify the OSS path that is used to store data in the partition:

     ALTER TABLE sale_detail ADD IF NOT EXISTS PARTITION (sale_date='202112', region='hangzhou') LOCATION 'oss://examplebucket.cn-hangzhou.oss-dls.aliyuncs.com/path/2021/';