All Products
Search

This Product

    E-MapReduce:Enable Kerberos authentication

    Document Center

    E-MapReduce:Enable Kerberos authentication

    Last Updated:Apr 13, 2026

    Enable Kerberos authentication in a Serverless Spark workspace to require all clients to authenticate before submitting Spark tasks. This reduces unauthorized access and improves task execution security.

    Limitations

    • A workspace can be bound to only one Kerberos cluster.

    • Kerberos authentication is supported only for Spark batch jobs.

    Important

    Kerberos authentication requires network connectivity between EMR Serverless Spark and your Virtual Private Cloud (VPC), and UDP port 88 must be open for Kerberos traffic. Complete the network setup in Step 1 before configuring authentication.

    Prerequisites

    Before you begin, ensure that you have:

    • A Kerberos principal created, with its keytab file exported and uploaded to Object Storage Service (OSS)

    • A Serverless Spark workspace. See Manage workspaces

    Step 1: Prepare the network

    Establish network connectivity between EMR Serverless Spark and your VPC. See Establish network connectivity between EMR Serverless Spark and other VPCs.

    Note

    When adding a security group rule, open UDP port 88 for Kerberos traffic. If you configure the network connection over TCP instead, add udp_preference_limit = 1 under [libdefaults] in your krb5.conf file in Step 2.

    Step 2: Configure Kerberos authentication

    This step binds your Serverless Spark workspace to a Kerberos cluster and enables authentication enforcement.

    1. In the EMR console, choose EMR Serverless > Spark in the left navigation pane.

    2. On the Spark page, click the name of the target workspace.

    3. In the left navigation pane, click Security > Kerberos Authentication.

    4. Click Bind Kerberos.

    5. On the Bind Kerberos page, configure the following parameters and click OK.

      ParameterDescription
      Kerberos NameEnter a custom name.
      Network ConnectionSelect the network connection created in Step 1.
      Kerberos krb5.confEnter the content of your krb5.conf file. See the instructions below.

      How to get the krb5.conf content

      The krb5.conf file is typically located at /etc/krb5.conf on the Kerberos server.

      • EMR DataLake cluster: Log on to the master node (see Log on to a cluster), run cat /etc/krb5.conf, and copy the output into the Kerberos krb5.conf field.

      • Other EMR clusters or self-managed Kerberos: Replace the hostname value in the file with the private IP address of your VPC.

      If you opened UDP port 88 in Step 1, no changes to krb5.conf are needed. If you used TCP, add udp_preference_limit = 1 under [libdefaults].

      image

    6. In the Actions column, click Enable, then click OK in the confirmation dialog.

    Step 3: Submit a Spark batch job with Kerberos authentication

    After enabling Kerberos authentication, every Spark batch job must include the Kerberos credentials. Submitting a job without them returns the error: spark.kerberos.keytab and spark.kerberos.principal not configured.

    1. Create a Spark batch job. See PySpark Quick Start.

    2. On the development tab, set Network Connection to the connection added in Step 1, then add the following Kerberos parameters to Spark Configuration and click Run.

      spark.files oss://<bucketname>/path/test.keytab
spark.kerberos.keytab test.keytab
spark.kerberos.principal <username>@<REALM>
      ParameterDescription
      spark.filesThe full path of the keytab file that is uploaded to OSS.
      spark.kerberos.keytabThe name of the keytab file.
      spark.kerberos.principalThe principal in the keytab file, used for identity authentication with Kerberos. Run klist -kt <keytab_file> to view the principal.

      (Optional) Connect to a Kerberos-enabled Hive Metastore from the job To query metadata from a Kerberos-protected Hive Metastore, add these parameters to Spark Configuration:

      spark.hive.metastore.sasl.enabled true
spark.hive.metastore.kerberos.principal hive/<hostname>@<REALM>

      Set spark.hive.metastore.kerberos.principal to the principal from the keytab file used by Hive Metastore. To find it:

      • In the EMR on ECS console, go to the Hive service Configuration page and open the hive-site.xml tab. Find the value of hive.metastore.kerberos.keytab.file.

      • Run klist -kt <path_to_Hive_Metastore_keytab_file> to retrieve the principal.

      The principal format is hive/<hostname>@<REALM>, where <hostname> is the fully qualified domain name (FQDN) of the node running Hive Metastore (run hostname -f to get it) and <REALM> is the KDC realm.

      If the Hive Metastore endpoint uses a hostname, use hive/_HOST@<REALM>. Spark automatically replaces _HOST with the hostname from the Hive Metastore endpoint. This format is required when configuring multiple Hive Metastores.

    3. After the job runs, go to the Execution Records section and click Details in the Actions column.

    4. In Job History, view logs on the Log Exploration page.

      image

    Step 4 (Optional): Connect the workspace data catalog to a Kerberos-enabled Hive Metastore

    If the workspace data catalog needs to fetch metadata from a Kerberos-protected Hive Metastore, specify the keytab file path and principal when adding an external Hive Metastore.

    image
    FieldDescription
    Kerberos keytab fileThe path to the Kerberos keytab file.
    Kerberos principalThe principal name from the keytab file, used for identity authentication with Kerberos. Run klist -kt <keytab_file> to view the principal.