Configure cross-realm mutual trust for Kerberos clusters - E-MapReduce

Set up cross-realm mutual trust between two EMR Kerberos clusters so that one cluster can access services running on the other. This topic covers cluster-to-cluster trust between MIT Key Distribution Centers (KDCs), not trust between an EMR cluster and Active Directory (AD).

The following steps configure one-way access from Cluster-A to Cluster-B. To enable access in both directions, repeat all steps with the cluster roles swapped.

Prerequisites

Before you begin, ensure that you have:

Two EMR clusters with Kerberos enabled (referred to as Cluster-A and Cluster-B)
SSH access to the master node of each cluster
Root access on those master nodes

How it works

Each Kerberos cluster has its own KDC and realm. To access a service in Cluster-B from Cluster-A, Cluster-A's KDC must trust Cluster-B's realm. You establish this trust by creating a cross-realm krbtgt principal on both KDCs and updating the Kerberos configuration on all nodes in Cluster-A.

Once trust is established, Cluster-A obtains a Ticket Granting Ticket (TGT) from its KDC and uses it to request service tickets in Cluster-B's realm.

Step 1: Collect cluster information

On the Configure tab of the Kerberos service page for each cluster, click the krb5.conf subtab and record the kdc_hosts and realm values.

This topic uses the following example values:

Parameter	Cluster-A	Cluster-B
Hostname	`master-1-1.1234.cn-hangzhou.emr.aliyuncs.com`	`master-1-1.6789.cn-hangzhou.emr.aliyuncs.com`
Realm	`EMR.1234.COM`	`EMR.6789.COM`

Step 2: Create the cross-realm principal

Log in to the master-1-1 node of Cluster-A via SSH as the root user.
Create the cross-realm principal for Cluster-A:
Parameter Description
123456 Initial password. Use the same password on both clusters.
EMR.1234.COM Realm of Cluster-A
EMR.6789.COM Realm of Cluster-B
```
kadmin.local -q "addprinc -pw 123456 krbtgt/EMR.6789.COM@EMR.1234.COM"
```
Log in to the master-1-1 node of Cluster-B via SSH as the root user and run the same command with the realm order swapped:
```
kadmin.local -q "addprinc -pw 123456 krbtgt/EMR.1234.COM@EMR.6789.COM"
```

Parameter	Description
`123456`	Initial password. Use the same password on both clusters.
`EMR.1234.COM`	Realm of Cluster-A
`EMR.6789.COM`	Realm of Cluster-B

Step 3: Update krb5.conf on Cluster-A

SSH into every node in Cluster-A and add the following sections to /etc/krb5.conf. No service restart is required after making these changes.

[realms] section — add the remote cluster's KDC address:

EMR.C-BE49B6BBAEEA****.COM = {
  kdc = 192.168.xx.xx:88
  admin_server = 192.168.xx.xx:749
}

[domain_realm] section — map hostnames to realms:

.1234.cn-hangzhou.emr.aliyuncs.com = EMR.1234.COM
.6789.cn-hangzhou.emr.aliyuncs.com = EMR.6789.COM

[capaths] section — define the trust path between realms:

EMR.1234.COM = {
   EMR.6789.COM = .
}
EMR.6789.COM = {
   EMR.1234.COM = .
}

Note

If you run jobs in Cluster-A that access resources in Cluster-B, restart YARN in Cluster-B after completing this step.

Step 4: Verify access

After Cluster-A obtains a TGT from its KDC, it can access services in Cluster-B.

Create a test principal and generate a ticket:

kadmin.local -q "addprinc -pw 123456 test"
kinit test

Access the Hadoop Distributed File System (HDFS) service of Cluster-B from Cluster-A:
```
hdfs dfs -ls hdfs://master-1-1.6789.cn-hangzhou.emr.aliyuncs.com:9000/
```
A successful directory listing confirms that cross-realm trust is working.