E-MapReduce:Balance the load of HiveServer2

Use ZooKeeper

ZooKeeper is installed by default in high-availability clusters. The beeline client uses ZooKeeper service discovery to automatically select and connect to an available HiveServer2 node.

1. Log on to your cluster in SSH mode. For more information, see Log on to a cluster.

2. Run the following command to connect to HiveServer2 through ZooKeeper:

   Parameter	Description
   master-1-1:2181,master-1-2:2181,master-1-3:2181	ZooKeeper node addresses. Replace with the addresses of your ZooKeeper nodes.
   serviceDiscoveryMode=zooKeeper	Enables ZooKeeper-based service discovery so the client automatically selects an available HiveServer2 node.
   zooKeeperNamespace=hiveserver2	The ZooKeeper path where HiveServer2 registers itself. The default value is hiveserver2. Change this only if you have modified the default registration path in your HiveServer2 configuration.

   beeline -u 'jdbc:hive2://master-1-1:2181,master-1-2:2181,master-1-3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2'

   The connection string uses the following parameters:

Use SLB

1. Create an SLB instance. For more information, see Create and manage a CLB instance.

2. Add the Elastic Compute Service (ECS) instances on which HiveServer2 is deployed to the default server group and configure the backend server weights based on your requirements. For more information, see Add and manage backend servers in the default server group.

3. Set the listener protocol to TCP. Set Backend Protocol/Port to TCP:10000. Configure the frontend listening port and scheduling algorithm based on your requirements. For more information, see Add a TCP listener.

4. Log on to your cluster in SSH mode. For more information, see Log on to a cluster.

5. Run the following command to connect to HiveServer2 through the SLB instance:

   Placeholder	Description
   <slb_ip_or_host>	The IP address of the SLB instance, or a hostname that resolves to that IP address.
   <slb_port>	The frontend listening port of the SLB instance.

   beeline -u 'jdbc:hive2://<slb_ip_or_host>:<slb_port>'

   Replace the placeholders with your actual values:

Use ZooKeeper

1. Run the kinit command to get a Ticket Granting Ticket (TGT). For more information, see Use Kerberos authentication to access a Hive client.

2. Log on to your cluster in SSH mode. For more information, see Log on to a cluster.

3. Run the following command to connect to HiveServer2 through ZooKeeper:

   beeline -u 'jdbc:hive2://master-1-1:2181,master-1-2:2181,master-1-3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2'

   Replace master-1-1:2181,master-1-2:2181,master-1-3:2181 with the addresses of your ZooKeeper nodes.

Use SLB

Create and configure the SLB instance by following the steps in Use SLB under Common EMR clusters (Kerberos disabled). For Kerberos clusters, you must also create a Kerberos principal for the SLB IP address and update the Hive configuration.

The following example uses 121.40.. as the SLB IP address and 10000 as the frontend listening port. Replace these with your actual values.

1. Run the kinit command to get a TGT. For more information, see Use Kerberos authentication to access a Hive client.

2. Create a Hive principal for the SLB IP address and export it to a keytab file.

   1. Log on to the master-1-1 node. For more information, see Log on to a cluster.

   2. Run the following command to open the Kerberos administration tool:

      kadmin.local

      Note

      If your cluster uses open source MIT Kerberos, connect to the Key Distribution Center (KDC) using kadmin.local or kadmin.

   3. Run the following commands to create a principal and export it to /tmp/slb.keytab:

      addprinc -randkey hive/121.40.**.**
      xst -k /tmp/slb.keytab hive/121.40.**.**
      exit

      Important

      Each time you export a principal, previously generated keytab files for that principal become invalid. Delete the existing /tmp/slb.keytab file before re-exporting to make sure the file contains only the latest principal.

3. Transfer the slb.keytab file to all nodes where HiveServer2 is deployed, then merge its content into the Hive keytab on each node:

   ktutil
   rkt /tmp/slb.keytab
   wkt /etc/taihao-apps/hive-conf/keytab/hive.keytab
   quit

4. Verify the keytab content:

   klist -kt /etc/taihao-apps/hive-conf/keytab/hive.keytab

   

5. Update the Hive configuration. On the Configure tab of the Hive service page in the EMR console, search for hive.server2.authentication.kerberos.principal and set its value to the principal created in step 2. In this example, the value is hive/121.40..@EMR.**.COM. Save the configuration and apply the changes.

6. Restart HiveServer2.

   1. On the Status tab of the Hive service page in the EMR console, find HiveServer2 and click Restart in the Actions column.

   2. In the dialog box that appears, enter the Execution Reason and click OK.

   3. In the Confirm message, click OK.

7. Run the following command on the master-1-1 node to connect to HiveServer2:

   beeline -u 'jdbc:hive2://121.40.**.**:10000/default;principal=hive/121.40.**.**@EMR.**.COM'