Kyuubi Gateway uses Kerberos for secure authentication and access control to ensure data security in distributed computing environments. - E-MapReduce

You can enable Kerberos for the Kyuubi Gateway in Serverless Spark to enforce secure authentication and access control. Once configured, clients must use Kerberos to authenticate when submitting jobs to the gateway, enhancing job execution security.

Limitations

The EMR on ECS cluster and the Serverless Spark workspace must be in the same region.

Prerequisites

An EMR on ECS cluster with Kerberos authentication enabled has been created. For more information, see Create a cluster.
A Serverless Spark workspace with Kerberos authentication enabled has been created. For more information, see Enable Kerberos authentication.
A Kyuubi Gateway has been created in the Serverless Spark workspace. The gateway does not need to be running.

Establish network connectivity

To use Kerberos with Kyuubi Gateway, you must configure PrivateLink to establish network connectivity between Serverless Spark and the Kerberos cluster. This configuration cannot be set up through self-service. To complete the setup, submit a ticket with the required information, and our technical support team will assist you.

Create an endpoint

An endpoint is created and managed by the service consumer. You can associate it with an endpoint service to establish a private network connection for accessing external services through PrivateLink. For more information, see Endpoints.

Log on to the Endpoint console.

On the Create Endpoint page, configure the endpoint with the following parameters, and then click Create.

Parameter	Description
Region	Select the region for the endpoint. Ensure the region matches that of the Kerberos cluster and the Serverless Spark workspace.
Node Name	Enter a custom name for the endpoint.
Endpoint Type	Select Interface Endpoint.
Endpoint Service	Click Select Service and then select or enter the ID of the target endpoint service. Note To obtain the endpoint service ID, submit a ticket with the following information: The Serverless Spark workspace ID, such as `w-f8cfXXXXXX`. The VPC ID of the Kerberos cluster that accesses the Kyuubi Gateway. This VPC must have available vSwitches in two zones. Example: `vpc-bp1tXXXXXX`. The two selected zones, such as `I,J`. To find out which zones your region supports, ask our support team in the ticket. The Kyuubi Gateway ID, such as `kb-564e********266f`.
VPC	Select the VPC of the Kerberos cluster that accesses the Kyuubi Gateway.
Security Group	Select the security group to associate with the elastic network interface (ENI) of the endpoint. Note You can associate up to nine security groups with an endpoint.
Zone and vSwitch	Select the zones and their corresponding vSwitches that you specified earlier.
IP Version	The following network types are supported: IPv4: Allows clients to use IPv4 addresses for access. Dual-stack: Allows clients to use both IPv4 and IPv6 addresses for access. Note You can select the dual-stack option only after the service provider completes the dual-stack configuration.
Resource Group	Select the resource group to which the endpoint belongs.
Tags	Select or enter a Tag Key and a Tag Value.

On the Basic Information page, if the Status is Status, the endpoint is ready for use. The endpoint service domain name is ep-xxxxxxxxxxx.epsrv-xxxxxxxxxxx.cn-hangzhou.privatelink.aliyuncs.com.
Log on to the Kerberos cluster and test the network connection.

Configure DNS resolution (Optional)

The default endpoint service domain name is long. To simplify it, you can configure a custom internal authoritative domain name. For more information, see Internal authoritative domain names.

Log on to . On the Authoritative Zone page, click the User-defined Zones tab, and then click Add Zone.
Enter an Internal Authoritative Domain Name, select the VPC where the domain name will apply, and then click OK. In this example, kyuubi-kerberos.abc is used.
Note
If the Domain Type option is available, select Private Authoritative Acceleration Zone. If the Domain Type option is not available, you do not need to make a selection, and a Private Authoritative Acceleration Zone domain is created by default.
On the User-defined Zones tab, find the target domain name and click Settings in the Actions column. Then, click Add Record and select Form Editor Mode in the dialog box.
Set Record Type to CNAME. For Hostname, enter a prefix. In this example, test is used. For Record Value, enter the endpoint service domain name ep-xxxxxxxxxxx.epsrv-xxxxxxxxxxx.cn-hangzhou.privatelink.aliyuncs.com. Clicking OK maps the endpoint service domain name to test.kyuubi-kerberos.abc.
Log on to the Kerberos cluster and test the network connection.
```
ping test.kyuubi-kerberos.abc
```

Create a keytab file

Log on to the Kerberos cluster.
Run the following command to start the kadmin.local tool for Kerberos:
```
kadmin.local
```
Create a principal in the format kyuubi/<fqdn>@<REALM>. We recommend that you use the endpoint domain name ep-xxxxxxxxxxx.epsrv-xxxxxxxxxxx.cn-hangzhou.privatelink.aliyuncs.com for the fqdn part. If you have configured a CNAME record to resolve a custom domain name, use the custom domain name, such as test.kyuubi-kerberos.abc.
```
addprinc -randkey kyuubi/ep-xxxxxxxxxxx.epsrv-xxxxxxxxxxx.cn-hangzhou.privatelink.aliyuncs.com@EMR.C-DFD4*****C204.COM
```

Export the keytab file and exit the kadmin.local tool.

xst -kt /root/kyuubi.keytab kyuubi/ep-xxxxxxxxxxx.epsrv-xxxxxxxxxxx.cn-hangzhou.privatelink.aliyuncs.com@EMR.C-DFD4*****C204.COM
quit

Upload the generated keytab file to an Object Storage Service (OSS) bucket.

hadoop fs -put /root/kyuubi.keytab oss://<YOUR_BUCKET>.<region>.oss-dls.aliyuncs.com/

Configure Kyuubi Gateway

To use Kerberos with Kyuubi Gateway, add the following Kyuubi Configuration.

kyuubi.authentication              KERBEROS
kyuubi.kinit.principal             kyuubi/ep-xxxxxxxxxxx.epsrv-xxxxxxxxxxx.cn-hangzhou.privatelink.aliyuncs.com@EMR.C-DFD43******7C204.COM
kyuubi.kinit.keytab                /opt/kyuubi/work-dir/kyuubi.keytab
kyuubi.files                       oss://bucket/path/to/kyuubi.keytab

Parameter	Description
kyuubi.authentication	Specifies the authentication mode used by Kyuubi Gateway. Set this parameter to `KERBEROS`.
kyuubi.kinit.principal	Specifies the principal that Kyuubi Gateway uses for Kerberos authentication. The format is `<user>/<host>@<realm>`.
kyuubi.kinit.keytab	Specifies the keytab file that Kyuubi Gateway uses. Note: The path is fixed. You only need to replace the keytab file name.
kyuubi.files	The OSS path to the keytab file that you uploaded in the "Create a keytab file" section.

If you need to connect to a Kerberos-enabled Hive Metastore (HMS), add the following Spark Configuration.

spark.hadoop.hive.metastore.uris                    thrift://master-1-1.c-1d36*****e840c.cn-hangzhou.emr.aliyuncs.com:9083
spark.hadoop.hive.imetastoreclient.factory.class    org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClientFactory
spark.hive.metastore.kerberos.principal             hive/_HOST@EMR.C-DFD4*****C204.COM
spark.hive.metastore.sasl.enabled                   true
spark.emr.serverless.network.service.name           <network_name>

Parameter	Description
spark.hadoop.hive.metastore.uris	The address of the Hive Metastore (HMS).
spark.hadoop.hive.imetastoreclient.factory.class	Specifies the factory class for creating an HMS client.
spark.hive.metastore.kerberos.principal	The principal for the HMS in a Kerberos environment.
spark.hive.metastore.sasl.enabled	Specifies whether to enable Kerberos authentication.
spark.emr.serverless.network.service.name	The name of the network connection.

Note

In a high availability (HA) cluster, you can configure multiple Thrift addresses for metastore.uris. The addresses must be separated by commas, and you must use hostnames, not IP addresses.
If you specify only one Thrift address for metastore.uris, you can use an IP address. However, metastore.kerberos.principal must be in the hive/<hostname of HMS>@<REALM> format.
You can simplify metastore.kerberos.principal to the hive/_HOST@<REALM> format only when metastore.uris uses a hostname.

Save the configurations and start the Kyuubi Gateway.

Submit a job

Run a show databases command to verify that the Kerberos cluster can connect to Kyuubi Gateway and start a Spark job.

Prepare a Kerberos user that has the required permissions and export its keytab file.
1. Log on to the Kerberos cluster.
2. Run the following commands to export the keytab file.
```
kadmin.local
addprinc -randkey hadoop
xst -kt /root/hadoop.keytab hadoop
quit
```
Authenticate with Kerberos by using the keytab file.
```
kinit -kt hadoop.keytab hadoop
```

Run the following command to connect to Kyuubi Gateway and start a Spark job.

/opt/apps/KYUUBI/kyuubi-1.9.2-1.0.0/bin/kyuubi-beeline -u 'jdbc:hive2://ep-xxxxxxxxxxx.epsrv-xxxxxxxxxxx.cn-hangzhou.privatelink.aliyuncs.com:10009/;principal=kyuubi/_HOST@EMR.C-DFD43*****7C204.COM'

After you connect, run show databases.

If your Spark job needs to connect to a Kerberos-enabled Hive Metastore (HMS) or Hadoop Distributed File System (HDFS) service, you must modify the core-site.xml configuration file on the HMS or HDFS cluster. Add the following properties to this file to allow the kyuubi user to impersonate other users when accessing the service. Otherwise, the connection may fail.

hadoop.proxyuser.kyuubi.hosts = *
hadoop.proxyuser.kyuubi.groups = *

Newer versions of EMR DataLake clusters include these properties by default. After adding these properties, you must restart the HDFS or HMS service.