How to use Spark in AnalyticDB for MySQL Lakehouse Edition to access Hive data - AnalyticDB

Spark in AnalyticDB for MySQL allows you to access Hive data via the Thrift and JDBC protocols. You can also enable Kerberos authentication for a Hive data source to enhance data security. This ensures that only Kerberos-authenticated clients can access the cluster and submit jobs. This topic describes how to access a Hive data source using the Hive service of an E-MapReduce cluster as an example.

Prerequisites

An AnalyticDB for MySQL Enterprise Edition, Basic Edition, or Data Lakehouse Edition cluster is created.
A database account is created for the AnalyticDB for MySQL cluster.
- If you use an Alibaba Cloud account, you need to only create a privileged account.
- If you use a Resource Access Management (RAM) user, you must create a privileged account and a standard account and associate the standard account with the RAM user.
An Object Storage Service (OSS) bucket is created in the same region as the AnalyticDB for MySQL cluster.
The AnalyticDB for MySQL cluster and the E-MapReduce cluster must be in the same region. For more information, see Create a cluster.
The E-MapReduce cluster must meet the following requirements:
- The cluster type is EMR on ECS.
- The use case is Data Lake. The cluster must have the Hadoop-Common, HDFS, YARN, and Hive services. The metadata must be stored in a Self-managed RDS or Built-in MySQL.
Important
If you access a Kerberos-encrypted Hive data source, ensure Kerberos Authentication is enabled.

Preparations

Download the JAR package required for Spark within AnalyticDB for MySQL to access Hive. For the download link, see mysql-connector-java.jar.
(Conditionally required) If your Hive version is earlier than 2.3, which can cause incompatibility issues, log on to the cluster's master node, navigate to the /opt/apps/HIVE/hive-current/lib path, and download all JAR packages. For logon instructions, see Log on to a cluster.
(Conditionally required) If Kerberos authentication is enabled for the Hive cluster, download the required configuration files.
1. Log on to the E-MapReduce cluster. For instructions, see Log on to a master node.
2. Download the krb5.conf file. For instructions, see Basic Kerberos operations.
3. Download the hadoop.keytab, core-site.xml, and hdfs-site.xml files. Follow these steps:
  1. Run the following command to find Hadoop-related environment variables and configuration information.
```
env | grep hadoop
```
    The following is a sample output:
```
HADOOP_HOME=/opt/apps/HADOOP-COMMON/hadoop-common-current/
HADOOP_CONF_DIR=/etc/taihao-apps/hadoop-conf
PATH=/opt/apps/JINDOSDK/jindosdk-current/bin:/opt/apps/HADOOP-COMMON/hadoop-common-current/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/apps/HIVE/hive-current/bin:/opt/apps/SPARK-EXTENSION/spark-extension-current/bin:/opt/apps/SPARK3/spark-current/bin:/root/bin
```
  2. Go to the directory specified in the HADOOP_CONF_DIR environment variable and download the hadoop.keytab, core-site.xml, and hdfs-site.xml files.
4. Run the following command to view the principal.
```
listprincs
```
  In the output, the string that starts with hadoop/master is the required principal.
Upload the JAR packages and Kerberos authentication configuration files obtained in Steps 1–3 to OSS. For instructions, see Simple upload.

Accessing a Hive data source from a Spark JAR

Write a sample Spark application to access Hive, and then compile and package it into the JAR dependency for the Spark job. The generated JAR package is named hive_test.jar. Upload the generated JAR package to OSS. For instructions, see simple upload. The following code provides an example:

package com.aliyun.spark

import org.apache.spark.sql.SparkSession

object SparkHive {
 def main(args: Array[String]): Unit = {
 val sparkSession = SparkSession
 .builder()
 .appName("Spark HIVE TEST")
 .enableHiveSupport()
 .getOrCreate()

 val welcome = "hello, adb-spark"

 // The name of the Hive table.
 val tableName = args(0)

 import sparkSession.implicits._
 // Saves a DataFrame with one row and one column (welcome_col) to the Hive table specified by the tableName argument.
 val df = Seq(welcome).toDF("welcome_col")
 df.write.format("hive").mode("overwrite").saveAsTable(tableName)

 // Read data from the Hive table specified by the tableName variable.
 val dfFromHive = sparkSession.sql(
 s"""
 |select * from $tableName
 |""".stripMargin)
 dfFromHive.show(10)
 }
}

Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. Find the cluster that you want to manage and click the cluster ID.
In the left-side navigation pane, click Job Development > Spark JAR Development.

At the top of the editor, select a Job Resource Group, set the job type to Batch, and write a Spark job.

Access a Hive data source

You can connect to a metadatabase by using different protocols.

Thrift protocol

{
  "args": [
    "hello_adb"
  ],
  "jars": [
    # JARs required by AnalyticDB for MySQL Spark to access Hive.
    "oss://<your-bucket-name>/mysql-connector-java.jar",
    # JARs required by the server-side Hive.
    "oss://<your-bucket-name>/hive_lib/*" 
  ],
  "file": "oss://<your-bucket-name>/hive_test.jar" ,
  "name": "spark-on-hive",
  "className": "com.aliyun.spark.SparkHive",
  "conf": {
    "spark.adb.eni.enabled": "true",
    "spark.adb.eni.adbHostAlias.enabled": "true",
    "spark.adb.eni.vswitchId": "vsw-bp1mbnyrjtf3ih1****", 
    "spark.adb.eni.securityGroupId": "sg-bp180fryne3qle****",
    "spark.driver.resourceSpec": "medium",
    "spark.executor.instances": 1,
    "spark.executor.resourceSpec": "medium",
    "spark.hadoop.hive.metastore.uris":"thrift://master-1-1.c-9c9b32****.cn-hangzhou.emr.aliyuncs.com:9083",
    "spark.adb.eni.extraHosts":"172.24.xx.xx master-1.c-9c9b322d****.cn-hangzhou.emr.aliyuncs.com", 
    # Configure the following parameters only if the E-MapReduce cluster is a high-availability cluster.
    "spark.hadoop.dfs.nameservices":"<your-dfs.nameservices-name>",
    "spark.hadoop.dfs.client.failover.proxy.provider.<your-dfs.nameservices-name>":"org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
    "spark.hadoop.dfs.ha.namenodes.<your-dfs.nameservices-name>":"<your-namenode-name>",
    "spark.hadoop.dfs.namenode.rpc-address.<your-dfs.nameservices-name>.<your-NameNode-name>":"master-1-1.c-9c9b322****.cn-hangzhou.emr.aliyuncs.com:9000",
    # Parameters for a custom Hive version. Required if the server-side Hive version is earlier than 2.3 to prevent version incompatibility.
    "spark.sql.hive.metastore.jars": "path",
    "spark.sql.hive.metastore.version": "<your-hive-version>",
    "spark.sql.hive.metastore.jars.path": "/tmp/*/*.jar"
  }
}

Parameter descriptions

Note

In the following table, the Required column indicates whether a parameter is mandatory for this scenario.

Parameter	Required	Description
`args`	Yes	The arguments for your JAR package. Separate multiple arguments with a comma (,).
`name`	Yes	The name of the Spark job.
`className`	Yes	The entry class of the Java or Scala application. You do not need to specify an entry class for a Python application.
`jars`	Yes	The OSS path of the JAR packages that the Spark job depends on.
`file`	Yes	The OSS path of the sample program `hive_test.jar`.
`spark.adb.eni.enabled`	Yes	Specifies whether to enable access over an elastic network interface (ENI). Set this parameter to `true` to enable ENI-based access.
`spark.adb.eni.adbHostAlias.enabled`	Yes	Specifies whether to enable Spark to resolve domain names in Hive table locations. You must set this parameter to `true`.
`spark.adb.eni.vswitchId`	Yes	The ID of the vSwitch in the VPC where the E-MapReduce cluster resides. Log on to the VPC console. On the Resource management tab of the target VPC, find the vSwitch ID.
`spark.adb.eni.securityGroupId`	Yes	The ID of the security group. Find the security group ID on the Basic Information page of the target E-MapReduce cluster.
`spark.adb.eni.extraHosts`	No	Specifies extra host-to-IP-address mappings for Spark to correctly resolve domain names for Hive table locations. To obtain the domain name: In the `<Hive_CONF_DIR>/core-site.xml` file of the self-managed cluster, find the value of the fs.defaultFS parameter. For example, if the value of fs.defaultFS is `hdfs://master-1.c-9c9b322d**.cn-hangzhou.emr.aliyuncs.com:9000`, the domain name is `master-1.c-9c9b322d.cn-hangzhou.emr.aliyuncs.com`. To obtain the IP address: Log on to the master node of the self-managed cluster and find the IP address-to-domain name mapping in the `/etc/hosts` file. Note** This parameter is required for a self-managed Hive cluster if its domain name cannot be resolved. It is optional for an E-MapReduce cluster that uses DNS for domain resolution. Use a space to separate an IP address and its domain name. Use a comma to separate multiple pairs. Example: `"ip0 master0, ip1 master1"`.
`spark.hadoop.hive.metastore.uris`	Yes	The URI for accessing the Hive metastore. To find the URI, log on to the EMR on ECS console. On the Services page, go to the Hive configurations and find the value of the hive.metastore.uris parameter.
`spark.hadoop.dfs.nameservices`	No	The name of the HDFS service. To find the name, log on to the EMR on ECS console. On the Services page, go to the HDFS configurations and find the value of the dfs.nameservices parameter. Note This parameter is required if the E-MapReduce cluster is a high-availability cluster.
`spark.hadoop.dfs.client.failover.proxy.provider.<your-dfs.nameservices-name>`	No	The provider that the client uses to connect to the NameNode. To find the provider, log on to the EMR on ECS console. On the Services page, go to the HDFS configurations and find the value of the dfs.client.failover.proxy.provider.<your-dfs.nameservices-name> parameter. The default value is org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider. Note This parameter is required if the E-MapReduce cluster is a high-availability cluster.
`spark.hadoop.dfs.ha.namenodes.<your-dfs.nameservices-name>`	No	The name of the NameNode. To find the name, log on to the EMR on ECS console. On the Services page, go to the HDFS configurations and find the value of the `dfs.ha.namenodes.<your-dfs.nameservices-name>` parameter in the `hdfs-site.xml` file. Note This parameter is required if the E-MapReduce cluster is a high-availability cluster.
`spark.hadoop.dfs.namenode.rpc-address.<your-dfs.nameservices-name>.<your-namenode-name>`	No	The remote procedure call (RPC) address of the NameNode. To find the address, log on to the EMR on ECS console. On the Services page, go to the HDFS configurations and find the value of the dfs.namenode.rpc-address.<your-dfs.nameservices-name>.<your-namenode-name> parameter. Note This parameter is required if the E-MapReduce cluster is a high-availability cluster.
`spark.sql.hive.metastore.jars`	No	The value must be `path`. Note This parameter is required if the server-side Hive version is earlier than 2.3 and a version incompatibility occurs.
`spark.sql.hive.metastore.version`	No	The version number of the server-side Hive. Note This parameter is required if the server-side Hive version is earlier than 2.3 and a version incompatibility occurs.
`spark.sql.hive.metastore.jars.path`	No	The value must be `/tmp//.jar`. Note This parameter is required if the server-side Hive version is earlier than 2.3 and a version incompatibility occurs.
Other `conf` parameters	Yes	The parameters are generally consistent with those in Apache Spark. The parameter format is `key:value`, and multiple parameters are separated by commas (,). For more information about `conf` parameters, see Conf Parameters.

JDBC protocol

{
  "args": [
    "hello_adb"
  ],
  "name": "spark-on-hive",
  "className": "com.aliyun.spark.SparkHive", 
  "jars": [
    # JARs required by AnalyticDB for MySQL Spark to access Hive.
    "oss://<your-bucket-name>/mysql-connector-java.jar",
    # JARs required by the server-side Hive.
    "oss://<your-bucket-name>/hive_lib/*" 
  ],
  "file": "oss://<your-bucket-name>/hive_test.jar" ,
  "conf": {
    "spark.adb.eni.enabled": "true",
    "spark.adb.eni.vswitchId": "vsw-bp1mbnyrjtf3ih1****", 
    "spark.adb.eni.securityGroupId": "sg-bp180fryne3qle****",
    "spark.adb.eni.adbHostAlias.enabled": "true",
    "spark.adb.eni.extraHosts":"172.24.xx.xx master-1.c-9c9b322d****.cn-hangzhou.emr.aliyuncs.com", 
    "spark.driver.resourceSpec": "medium",
    "spark.executor.instances": 1,
    "spark.executor.resourceSpec": "medium",
    "spark.sql.catalogImplementation": "hive",
    "spark.hadoop.javax.jdo.option.ConnectionDriverName": "com.mysql.jdbc.Driver",
    "spark.hadoop.javax.jdo.option.ConnectionUserName": "<your-hive-username>",
    "spark.hadoop.javax.jdo.option.ConnectionPassword": "<your-hive-password>",
    "spark.hadoop.javax.jdo.option.ConnectionURL": "jdbc:mysql://rm-bp1h5d11r8qtm****.mysql.rds.aliyuncs.com/<your-database-name>", 
    # Parameters for a custom Hive version. Required if the server-side Hive version is earlier than 2.3 to prevent version incompatibility.
    "spark.sql.hive.metastore.jars": "path",
    "spark.sql.hive.metastore.version": "<your-hive-version>",
    "spark.sql.hive.metastore.jars.path": "/tmp/*/*.jar"
  }
}

Parameter descriptions

Note

In the following table, the Required column indicates whether a parameter is mandatory for this scenario.

Parameter	Required	Description
`args`	Yes	The arguments for your JAR package. Separate multiple arguments with a comma (,).
`name`	Yes	The name of the Spark job.
`className`	Yes	The entry class of the Java or Scala application. You do not need to specify an entry class for a Python application.
`jars`	Yes	The OSS path of the JAR packages that the Spark job depends on.
`file`	Yes	The OSS path of the sample program `hive_test.jar`.
`spark.adb.eni.enabled`	Yes	Specifies whether to enable access over an ENI. Set this parameter to `true` to enable ENI-based access.
`spark.adb.eni.vswitchId`	Yes	The ID of the vSwitch in the VPC where the E-MapReduce cluster resides. Log on to the VPC console. On the Resource management tab of the target VPC, find the vSwitch ID.
`spark.adb.eni.securityGroupId`	Yes	The ID of the security group. You can find the security group ID on the Basic Information page of the target E-MapReduce cluster.
`spark.adb.eni.adbHostAlias.enabled`	Yes	Specifies whether to enable Spark to resolve domain names in Hive table locations. You must set this parameter to `true`.
`spark.adb.eni.extraHosts`	No	Specifies extra host-to-IP-address mappings for Spark to correctly resolve domain names for Hive table locations. To obtain the domain name: In the `<Hive_CONF_DIR>/core-site.xml` file of the self-managed cluster, find the value of the fs.defaultFS parameter. For example, if the value of fs.defaultFS is `hdfs://master-1.c-9c9b322d**.cn-hangzhou.emr.aliyuncs.com:9000`, the domain name is `master-1.c-9c9b322d.cn-hangzhou.emr.aliyuncs.com`. To obtain the IP address: Log on to the master node of the self-managed cluster and find the IP address-to-domain name mapping in the `/etc/hosts` file. Note** This parameter is required for a self-managed Hive cluster if its domain name cannot be resolved. It is optional for an E-MapReduce cluster that uses DNS for domain resolution. Use a space to separate an IP address and its domain name. Use a comma to separate multiple pairs. Example: `"ip0 master0, ip1 master1"`.
`spark.hadoop.javax.jdo.option.ConnectionDriverName`	Yes	The name of the connection driver. To find the name, log on to the EMR on ECS console. On the Services page, go to the Hive configurations and find the value of the javax.jdo.option.ConnectionDriverName parameter in the hivemetastore-site.xml file.
`spark.hadoop.javax.jdo.option.ConnectionUserName`	Yes	The username for the database account of your ApsaraDB RDS instance or the cluster's built-in MySQL database. To find the username, log on to the EMR on ECS console. On the Services page, go to the Hive configurations and find the value of the javax.jdo.option.ConnectionUserName parameter in the hivemetastore-site.xml file.
`spark.hadoop.javax.jdo.option.ConnectionPassword`	Yes	The password for the database account. To find the password, log on to the EMR on ECS console. On the Services page, go to the Hive configurations and find the value of the javax.jdo.option.ConnectionPassword parameter in the hivemetastore-site.xml file.
`spark.hadoop.javax.jdo.option.ConnectionURL`	Yes	The connection URL of the database, including the database name. The format is `jdbc:mysql://rm-xxxxxx.mysql.rds.aliyuncs.com/<database name>`. To find the URL, log on to the EMR on ECS console. On the Services page, go to the Hive configurations and find the value of the javax.jdo.option.ConnectionURL parameter in the hivemetastore-site.xml file.
`spark.sql.hive.metastore.jars`	No	The value must be `path`. Note This parameter is required if the server-side Hive version is earlier than 2.3 and a version incompatibility occurs.
`spark.sql.hive.metastore.version`	No	The version number of the server-side Hive. Note This parameter is required if the server-side Hive version is earlier than 2.3 and a version incompatibility occurs.
`spark.sql.hive.metastore.jars.path`	No	The value must be `/tmp//.jar`. Note This parameter is required if the server-side Hive version is earlier than 2.3 and a version incompatibility occurs.
Other `conf` parameters	Yes	The configurations are basically consistent with those in Apache Spark, where parameters are in the `key:value` format and multiple parameters are separated by commas (,). For more `conf` parameters, see Configuration parameters.

Access a Kerberos-encrypted Hive data source

{
  "args": ["hello_adb"],
  "name": "spark-on-hive",
  "className": "com.aliyun.spark.SparkHive", 
  "jars": [
    # Dependencies for Spark to access Hive.
    "oss://testBucketname/mysql-connector-java.jar",
    # JAR packages that the server-side Hive depends on. Specify this parameter if a Hive version incompatibility occurs.
    "oss://testBucketname/hive_lib/*" ],
  "file": "oss://testBucketname/hive_test.jar" ,
  "conf": {
    "spark.adb.eni.enabled": "true",
    "spark.adb.eni.vswitchId": "vsw-bp1mbnyrjtf3ih1****", 
    "spark.adb.eni.securityGroupId": "sg-bp180fryne3qle****",
    "spark.adb.eni.adbHostAlias.enabled": "true",
    "spark.adb.eni.extraHosts":"172.24.xx.xx master-1.c-9c9b322d****.cn-hangzhou.emr.aliyuncs.com", 
    "spark.driver.resourceSpec": "medium",
    "spark.executor.instances": 1,
    "spark.executor.resourceSpec": "medium",
    "spark.sql.catalogImplementation": "hive",
    "spark.hadoop.javax.jdo.option.ConnectionDriverName": "com.mysql.jdbc.Driver",
    "spark.hadoop.javax.jdo.option.ConnectionUserName": "hive_username",
    "spark.hadoop.javax.jdo.option.ConnectionPassword": "hive_password",
    "spark.hadoop.javax.jdo.option.ConnectionURL": "jdbc:mysql://master-1-1.c-49f95900****.cn-beijing.emr.aliyuncs.com/hivemeta?createDatabaseIfNotExist=true&characterEncoding=UTF-8", 
    # Required for custom Hive versions earlier than 2.3 if incompatibility occurs.
    "spark.sql.hive.metastore.jars": "path",
    "spark.sql.hive.metastore.version": "",
    "spark.sql.hive.metastore.jars.path": "/tmp/*/*.jar"
    # Kerberos-related configurations.
    "spark.kubernetes.driverEnv.ADB_SPARK_DOWNLOAD_FILES": "oss://testBucketname/hadoop/hadoop.keytab, oss://testBucketname/hadoop/core-site.xml, oss://testBucketname/hadoop/hdfs-site.xml",
    "spark.executorEnv.ADB_SPARK_DOWNLOAD_FILES": "oss://testBucketname/hadoop/krb5.conf, oss://testBucketname/hadoop/hadoop.keytab, oss://testBucketname/hadoop/core-site.xml, oss://testBucketname/hadoop/hdfs-site.xml",
    "spark.kubernetes.driverEnv.HADOOP_CONF_DIR": "/tmp/testBucketname/hadoop",
    "spark.executorEnv.HADOOP_CONF_DIR": "/tmp/testBucketname/hadoop",
    "spark.kerberos.keytab": "local:///tmp/testBucketname/hadoop/hadoop.keytab",
    "spark.executor.extraJavaOptions":"-Djava.security.krb5.conf=/tmp/testBucketname/hadoop/krb5.conf",
    "spark.kubernetes.kerberos.krb5.path": "oss://testBucketname/hadoop/krb5.conf",
    "spark.kerberos.principal":"hadoop/master-1-1.c-49f95900****.cn-beijing.emr.aliyuncs.com@EMR.C-49F95900****.COM"
  }
}

Parameters

Parameter	Required	Description
args	Yes	Based on your business requirements, specify the parameters to use with the JAR package. Separate multiple parameters with commas (,).
className	Yes	The name of the Java program entry class.
jars	Yes	The OSS paths to the dependent JAR packages for the Spark job.
file	Yes	The OSS path to the sample program `hive_test.jar`.
spark.adb.eni.enabled	Yes	Specifies whether to enable Elastic Network Interface (ENI). Set this parameter to true to access a Kerberos-encrypted Hive data source.
spark.adb.eni.vswitchId	Yes	The ID of the vSwitch in the VPC where the E-MapReduce cluster resides. Log on to the VPC console. On the Resource management tab of the target VPC, find the vSwitch ID.
spark.adb.eni.securityGroupId	Yes	The ID of the security group. Find the security group ID on the Basic Information page of the target E-MapReduce cluster.
spark.adb.eni.adbHostAlias.enabled	Yes	Specifies whether to enable Spark to resolve domain names in Hive table locations. You must set this parameter to `true`.
spark.sql.catalogImplementation	Yes	Set this parameter to Hive to use the Hive catalog.
spark.hadoop.javax.jdo.option.ConnectionDriverName	Yes	The name of the connection driver. To find the name, log on to the EMR on ECS console. On the Services page, go to the Hive configurations and find the value of the javax.jdo.option.ConnectionDriverName parameter in the hivemetastore-site.xml file.
spark.hadoop.javax.jdo.option.ConnectionUserName	Yes	The username for the database account of your ApsaraDB RDS instance or the cluster's built-in MySQL database. To find the username, log on to the EMR on ECS console. On the Services page, go to the Hive configurations and find the value of the javax.jdo.option.ConnectionUserName parameter in the hivemetastore-site.xml file.
spark.hadoop.javax.jdo.option.ConnectionPassword	Yes	The password for the database account. To find the password, log on to the EMR on ECS console. On the Services page, go to the Hive configurations and find the value of the javax.jdo.option.ConnectionPassword parameter in the hivemetastore-site.xml file.
spark.hadoop.javax.jdo.option.ConnectionURL	Yes	The connection URL of the database, including the database name. To find the URL, log on to the EMR on ECS console. On the Services page, go to the Hive configurations and find the value of the javax.jdo.option.ConnectionURL parameter in the hivemetastore-site.xml file.
spark.kubernetes.driverEnv.ADB_SPARK_DOWNLOAD_FILES	Yes	The OSS paths to the hadoop.keytab, core-site.xml, and hdfs-site.xml files for the Spark driver.
spark.executorEnv.ADB_SPARK_DOWNLOAD_FILES	Yes	The OSS paths to the krb5.conf, hadoop.keytab, core-site.xml, and hdfs-site.xml files for the Spark executor.
spark.kubernetes.driverEnv.HADOOP_CONF_DIR	Yes	A Spark driver parameter that specifies the local directory containing the hadoop.keytab, core-site.xml, and hdfs-site.xml files. The path must be in the `/tmp/<OSS-directory-of-the-files>` format. For example, if the OSS directory that contains the hadoop.keytab, core-site.xml, and hdfs-site.xml files is `oss://testBucketname/hadoop`, the local directory is `/tmp/testBucketname/hadoop`.
spark.executorEnv.HADOOP_CONF_DIR	Yes	A Spark executor parameter that specifies the local directory containing the hadoop.keytab, core-site.xml, and hdfs-site.xml files. The path must be in the `/tmp/<OSS-directory-of-the-files>` format. For example, if the OSS directory that contains the hadoop.keytab, core-site.xml, and hdfs-site.xml files is `oss://testBucketname/hadoop`, the local directory is `/tmp/testBucketname/hadoop`.
spark.kerberos.keytab	Yes	The local path to the hadoop.keytab file. The path must be in the `local:///tmp/<OSS-path-to-the-hadoop.keytab-file>` format. For example, if the OSS path of the hadoop.keytab file is `oss://testBucketname/hadoop/hadoop.keytab`, the local path is `local:///tmp/testBucketname/hadoop/hadoop.keytab`.
spark.executor.extraJavaOptions	Yes	A Spark executor parameter that specifies the local path to the krb5.conf file. The path must be in the `/tmp/<OSS-path-to-the-krb5.conf-file>` format. For example, if the OSS path of the krb5.conf file is `oss://testBucketname/hadoop/krb5.conf`, the local path is `/tmp/testBucketname/hadoop/krb5.conf`.
spark.kubernetes.kerberos.krb5.path	Yes	The OSS path to the krb5.conf file.
spark.kerberos.principal	Yes	The Kerberos principal obtained during setup.
spark.adb.eni.extraHosts	No	Specifies extra host-to-IP-address mappings for Spark to correctly resolve domain names for Hive table locations. To obtain the domain name: In the `<Hive_CONF_DIR>/core-site.xml` file of the self-managed cluster, find the value of the fs.defaultFS parameter. For example, if the value of fs.defaultFS is `hdfs://master-1.c-9c9b322d**.cn-hangzhou.emr.aliyuncs.com:9000`, the domain name is `master-1.c-9c9b322d.cn-hangzhou.emr.aliyuncs.com`. To obtain the IP address: Log on to the master node of the self-managed cluster and find the IP address-to-domain name mapping in the `/etc/hosts` file. Note** This parameter is required for a self-managed Hive cluster if its domain name cannot be resolved. It is optional for an E-MapReduce cluster that uses DNS for domain resolution. Use a space to separate an IP address and its domain name. Use a comma to separate multiple pairs. Example: `"ip0 master0, ip1 master1"`.
spark.sql.hive.metastore.jars	No	The value must be `path`. Note This parameter is required if the server-side Hive version is earlier than 2.3 and a version incompatibility occurs.
spark.sql.hive.metastore.version	No	The version number of the server-side Hive. Note This parameter is required if the server-side Hive version is earlier than 2.3 and a version incompatibility occurs.
spark.sql.hive.metastore.jars.path	No	The value must be `/tmp//.jar`. Note This parameter is required if the server-side Hive version is earlier than 2.3 and a version incompatibility occurs.
Other conf parameters	No	Corresponds to standard Apache Spark configuration properties. Specify these as `key:value` pairs within the `conf` object. For more information, see Spark application configuration parameters.

Access a Hive data source with Spark SQL

Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. Find the cluster that you want to manage and click the cluster ID.
In the left-side navigation pane, choose Job Development > SQL Development.

At the top of the editor, select the Spark engine and a Job Resource Group. Write a Spark SQL job and click Execute.

Access a Hive data source

You can connect to the metastore by using different protocols.

Thrift protocol

SET spark.adb.eni.enabled=true;
SET spark.adb.eni.vswitchId=vsw-bp1mbnyrjtf3ih1****; 
SET spark.adb.eni.securityGroupId=sg-bp180fryne3qle****;
SET spark.adb.eni.adbHostAlias.enabled=true;
SET spark.driver.resourceSpec=medium;
SET spark.executor.instances=1;
SET spark.executor.resourceSpe=medium;
SET spark.sql.catalogImplementation=hive;
SET spark.hadoop.hive.metastore.uris=thrift://master-1-1.c-9c9b32****.cn-hangzhou.emr.aliyuncs.com:9083;
SET spark.sql.hive.metastore.version=2.3.9.adb;
-- Configure the following parameters only if the EMR cluster is a high-availability cluster.
SET spark.hadoop.dfs.nameservices=<your-dfs.nameservices-name>;
SET spark.hadoop.dfs.client.failover.proxy.provider.<your-dfs.nameservices-name>=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider;
SET spark.hadoop.dfs.ha.namenodes.<your-dfs.nameservices-name>=<your-namenode-name>;
SET spark.hadoop.dfs.namenode.rpc-address.<your-dfs.nameservices-name>.<your-NameNode-name>=master-1-1.c-9c9b322****.cn-hangzhou.emr.aliyuncs.com:9000;
-- Configure the following properties to resolve incompatibilities with server-side Hive versions earlier than 2.3.
SET spark.sql.hive.metastore.jars=path;
SET spark.sql.hive.metastore.version=<your-hive-version>;
SET spark.sql.hive.metastore.jars.path=/tmp/*/*.jar;
-- Your SQL statement.
SHOW databases;

For parameter descriptions, see Parameters.

JDBC protocol

SET spark.adb.eni.enabled=true;
SET spark.adb.eni.vswitchId=vsw-bp1mbnyrjtf3ih1****; 
SET spark.adb.eni.securityGroupId=sg-bp180fryne3qle****;
SET spark.adb.eni.adbHostAlias.enabled=true;
SET spark.driver.resourceSpec=medium;
SET spark.executor.instances=1;
SET spark.executor.resourceSpe=medium;
SET spark.sql.catalogImplementation=hive;
SET spark.hadoop.javax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver;
SET spark.hadoop.javax.jdo.option.ConnectionUserName=hive_username;
SET spark.hadoop.javax.jdo.option.ConnectionPassword=hive_password;
SET spark.hadoop.javax.jdo.option.ConnectionURL=jdbc:mysql://rm-bp1h5d11r8qtm****.mysql.rds.aliyuncs.com/dbname;
SET spark.sql.hive.metastore.version=2.3.9.adb;
-- Configure the following properties to resolve incompatibilities with server-side Hive versions earlier than 2.3.
SET spark.sql.hive.metastore.jars=path;
SET spark.sql.hive.metastore.version=<your-hive-version>;
SET spark.sql.hive.metastore.jars.path=/tmp/*/*.jar;
-- Your SQL statement.
SHOW databases;

For parameter descriptions, see Parameters.

Access a Kerberos-encrypted Hive data source

SET spark.adb.eni.enabled=true;
SET spark.adb.eni.vswitchId=vsw-bp1mbnyrjtf3ih1****; 
SET spark.adb.eni.securityGroupId=sg-bp180fryne3qle****;
SET spark.adb.eni.adbHostAlias.enabled=true;
SET spark.driver.resourceSpec=medium;
SET spark.executor.instances=1;
SET spark.executor.resourceSpe=medium;
SET spark.sql.catalogImplementation=hive;
SET spark.hadoop.javax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver;
SET spark.hadoop.javax.jdo.option.ConnectionUserName=hive_username;
SET spark.hadoop.javax.jdo.option.ConnectionPassword=hive_password;
SET spark.hadoop.javax.jdo.option.ConnectionURL=jdbc:mysql://rm-bp1h5d11r8qtm****.mysql.rds.aliyuncs.com/dbname;
SET spark.kubernetes.driverEnv.ADB_SPARK_DOWNLOAD_FILES=oss://testBucketname/hadoop/hadoop.keytab, oss://testBucketname/hadoop/core-site.xml, oss://testBucketname/hadoop/hdfs-site.xml;
SET spark.executorEnv.ADB_SPARK_DOWNLOAD_FILES=oss://testBucketname/hadoop/krb5.conf, oss://testBucketname/hadoop/hadoop.keytab, oss://testBucketname/hadoop/core-site.xml, oss://testBucketname/hadoop/hdfs-site.xml;
SET spark.kubernetes.driverEnv.HADOOP_CONF_DIR=/tmp/testBucketname/hadoop;
SET spark.executorEnv.HADOOP_CONF_DIR=/tmp/testBucketname/hadoop;
SET spark.kerberos.keytab=local:///tmp/testBucketname/hadoop/hadoop.keytab;
SET spark.kubernetes.kerberos.krb5.path=oss://testBucketname/hadoop/krb5.conf;
SET spark.executor.extraJavaOptions=-Djava.security.krb5.conf=/tmp/testBucketname/hadoop/krb5.conf;
SET spark.kerberos.principal=hadoop/master-1-1.c-49f95900****.cn-beijing.emr.aliyuncs.com@EMR.C-49F95900****.COM;
SET spark.sql.hive.metastore.version=2.3.9.adb;
-- Configure the following properties to resolve incompatibilities with server-side Hive versions earlier than 2.3.
SET spark.sql.hive.metastore.jars=path;
SET spark.sql.hive.metastore.version=<your-hive-version>;
SET spark.sql.hive.metastore.jars.path=/tmp/*/*.jar;
-- Your SQL statement.
SHOW databases;

For parameter descriptions, see Parameters.