E-MapReduce:Connect to a Kerberos-enabled Hive by using Java

Prerequisites

You have created a cluster and enabled Kerberos Authentication in the Advanced Settings section of the Software Configuration page. For more information, see Create a cluster.

Step 1: Get the Kerberos configuration

1. Log on to the cluster's master node by using SSH. For more information, see Log on to a cluster.

2. Run the following command to get the Kerberos configuration file krb5.conf. This file is typically located at /etc/krb5.conf on the master-1-1 node.

   cat /etc/krb5.conf

   Use the default_realm value from this file in your Java code. The following example shows the configuration:

   [logging]
     default = FILE:/mnt/disk1/log/kerberos/krb5libs.log
     kdc = FILE:/mnt/disk1/log/kerberos/krb5kdc.log
     admin_server = FILE:/mnt/disk1/log/kerberos/kadmind.log
   
   [libdefaults]
     default_realm = EMR.C-EXAMPLE.COM
     dns_lookup_realm = false
     dns_lookup_kdc = false
     ticket_lifetime = 24h
     renew_lifetime = 7d
     forwardable = true
     rdns = false
     dns_canonicalize_hostname = true
     pkinit_anchors = FILE:/etc/pki/tls/certs/ca-bundle.crt
     kdc_timeout = 30s
     max_retries = 3
   
   [realms]
     EMR.C-EXAMPLE.COM = {
       kdc = master-1-1.c-ce2fcb9c9c0b****.cn-hangzhou.emr.aliyuncs.com:88
       admin_server = master-1-1.c-ce2fcb9c9c0b****.cn-hangzhou.emr.aliyuncs.com:749
     }

Step 2: Get the keytab file and principal

1. Copy the Hive keytab file to your local development environment.

   scp root@<public IP address>:/etc/taihao-apps/hive-conf/keytab/hive.keytab /tmp/hive.keytab

   Replace <public IP address> with the public IP address of the master node. For more information, see Obtain the public IP address and the name of a node.

2. Verify the validity of the keytab file on your local machine and obtain the principal.

   klist -kt /tmp/hive.keytab

   The following example shows the output from a successful command:

   Keytab name: FILE:/tmp/hive.keytab
   KVNO Timestamp           Principal
   ---- ------------------- ------------------------------------------------------
      2 02/25/2025 10:40:41 hive/master-1-1.c-EXAMPLE.cn-hangzhou.emr.aliyuncs.com@EMR.C-EXAMPLE.COM
      2 02/25/2025 10:40:41 hive/master-1-1.c-EXAMPLE.cn-hangzhou.emr.aliyuncs.com@EMR.C-EXAMPLE.COM

   You will use this principal (for example, hive/master-1-1.c-EXAMPLE.cn-hangzhou.emr.aliyuncs.com@EMR.C-EXAMPLE.COM) in your Java code.

Step 3: Configure network access control policies

Configure security group rules to allow your local development environment to access the EMR cluster.

1. Obtain the public IP address of your local development environment.

   You can find your public IP address by using an IP address lookup service.

2. Go to the security group page.

   1. Log on to the EMR console.

   2. In the top navigation bar, select the region and resource group.

   3. On the Clusters page, click the cluster ID of the target cluster.

   4. On the Basic Information page, click the link next to Cluster Security Group.

3. On the Rules page, click Add Rule.

   Set Protocol to All Traffic and Source to the public IP address of your local environment. Use the default values for the other parameters. For more information, see Add security group rules.

Step 4: Write the Java code

<dependencies>
    <dependency>
        <groupId>org.apache.hive</groupId>
        <artifactId>hive-jdbc</artifactId>
        <version>3.1.3</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>3.2.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-auth</artifactId>
        <version>3.2.1</version>
    </dependency>
</dependencies>

package com.aliyun.emr.example;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.security.UserGroupInformation;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;

public class Main {
    private static final String DRIVER_CLASS = "org.apache.hive.jdbc.HiveDriver";
    
    public static void main(String[] args) throws Exception {
        // Set the realm and Key Distribution Center (KDC) address for Kerberos authentication.
        System.setProperty("java.security.krb5.realm", "EMR.EXAMPLE.COM");
        System.setProperty("java.security.krb5.kdc", "$IPORHOST");
        
        Configuration conf = new Configuration();
        conf.set("hadoop.security.authentication", "kerberos");
        UserGroupInformation.setConfiguration(conf);
        // Log on to Kerberos by using the keytab file.
        UserGroupInformation.loginUserFromKeytab(
            "hive/master-1-1.c-EXAMPLE.cn-hangzhou.emr.aliyuncs.com@EMR.C-EXAMPLE.COM",
            "/tmp/hive.keytab"
        );
        
        Class.forName(DRIVER_CLASS);
        
        // Define the Hive principal for JDBC connection authentication.
        String hivePrincipal = "hive/master-1-1.c-EXAMPLE.cn-hangzhou.emr.aliyuncs.com@EMR.C-EXAMPLE.COM";
        // Construct the Hive JDBC URL, which includes the connection address and principal information.
        String hiveUrl = "jdbc:hive2://$IPORHOST:10000/;principal=" + hivePrincipal;
        Connection connection = DriverManager.getConnection(hiveUrl);
        Statement statement = connection.createStatement();
        
        ResultSet resultSet = statement.executeQuery("SHOW DATABASES");
        while (resultSet.next()) {
            System.out.println(resultSet.getString(1));
        }
        
        resultSet.close();
        statement.close();
        connection.close();
    }
}

System.setProperty("sun.security.krb5.debug", "true");

Troubleshooting