E-MapReduce:Configure and enable Ranger authorization for a Livy Gateway

Step 1: Prepare the network

Before you begin, you must configure the network to ensure connectivity between Serverless Spark and your Virtual Private Cloud (VPC). This allows the Ranger Plugin to connect to your Ranger Admin service and retrieve permission information. For more information, see Interconnect EMR Serverless Spark with other VPCs.

Step 2: Configure Ranger Plugin

To enable Ranger authorization for a Livy Gateway, you must first stop the gateway and then add the following configurations.

1. From the Normal Network Connection drop-down list, select the name of the connection that you created.

2. In livy.conf, add the following configuration:

   livy.impersonation.enabled   true

3. In spark-defaults.conf, add the following configurations:

   spark.ranger.plugin.enabled                true
   spark.emr.serverless.user.defined.jars     /opt/ranger/ranger-spark.jar
   ranger.plugin.spark.policy.rest.url        http://<ranger_admin_ip>:<ranger_admin_port>

   The following table describes the parameters.

   Parameter	Description
   spark.ranger.plugin.enabled	Set this to true to enable Ranger authorization.
   spark.emr.serverless.user.defined.jars	The path to a custom JAR file. Set this to /opt/ranger/ranger-spark.jar to use the Ranger Plugin included with Serverless Spark.
   ranger.plugin.spark.policy.rest.url	The address of the Ranger Admin service. The format is http://<ranger_admin_ip>:<ranger_admin_port>. Replace <ranger_admin_ip> and <ranger_admin_port> with the internal IP address and port of your Ranger Admin service. If you are connecting to the Ranger service on an EMR on ECS cluster, set <ranger_admin_ip> to the internal IP address of the master node and <ranger_admin_port> to 6080.

4. After you finish editing, restart the gateway to apply the configuration changes.

Step 3: (Optional) Configure Ranger Audit

You can configure Ranger to send audit information to storage destinations such as Solr and HDFS. By default, Ranger Audit is disabled in Serverless Spark. To enable this feature, add the required Ranger Audit parameters in Spark Configuration.

For example, to configure a connection to Solr on EMR, add the following properties in Spark Configuration.

xasecure.audit.is.enabled                  true
xasecure.audit.destination.solr            true
xasecure.audit.destination.solr.urls       http://<solr_ip>:<solr_port>/solr/ranger_audits
xasecure.audit.destination.solr.user       <user>
xasecure.audit.destination.solr.password   <password>

Parameters:

• xasecure.audit.is.enabled: Enables or disables Ranger Audit.

• xasecure.audit.destination.solr: Enables storing audit information in Solr.

• xasecure.audit.destination.solr.urls: The URL of the Solr service. Replace <solr_ip> and <solr_port> with the IP address and port of your Solr service. Configure the rest of the URL based on your actual requirements.

• xasecure.audit.destination.solr.user and xasecure.audit.destination.solr.password: If Basic Authentication is enabled for the Solr service, provide the username and password.

  If you connect to Ranger in an EMR on ECS cluster, you can find the values for xasecure.audit.destination.solr.urls, xasecure.audit.destination.solr.user, and xasecure.audit.destination.solr.password in the ranger-spark-audit.xml configuration file of the Ranger-plugin service.

  

After the configuration is complete, you can submit a job on EMR Serverless Spark, navigate to the Ranger UI, and view user access audit information on the Access tab in Ranger Audit. For information about how to access the Ranger UI, see Access the web UI for open-source components from the console.

Step 4: Test connection

Use a Jupyter Notebook to test the connection. If you attempt to access a database, table, or other resource for which you do not have permissions, a Permission denied message is returned.