E-MapReduce:Configure Ranger authentication for a Spark Thrift Server

1. Network preparation

Before starting the configuration, ensure network connectivity is configured so Serverless Spark can communicate with your virtual private cloud (VPC). This allows the Ranger plugin to connect to the Ranger Admin service and retrieve permissions. For detailed instructions, see Network Connectivity Between EMR Serverless Spark and Other VPCs.

2. Configure Ranger Plugin

To enable Ranger authentication for a Spark Thrift Server session, first stop the session. In the Network Connectivity drop-down list, select the established connection. In Spark Configuration, add the necessary configuration items. After editing, restart the session to apply the changes.

spark.ranger.plugin.enabled                true
spark.jars                                 /opt/ranger/ranger-spark.jar
ranger.plugin.spark.policy.rest.url        http://<ranger_admin_ip>:<ranger_admin_port>

spark.jars                                 oss://<bucket>/path/to/user-ranger-spark.jar
spark.ranger.plugin.class                  <class_name>
spark.ranger.plugin.enabled                true
ranger.plugin.spark.policy.rest.url        http://<ranger_admin_ip>:<ranger_admin_port>

3. (Optional) Configure Ranger Audit

Ranger allows you to choose a service to store audit logs, such as Solr or HDFS. By default, Serverless Spark does not enable Ranger audit. If required, add Ranger audit parameters in Spark Configuration.

For instance, to configure a connection to Solr in EMR, include the following configuration in Spark Configuration.

xasecure.audit.is.enabled                  true
xasecure.audit.destination.solr            true
xasecure.audit.destination.solr.urls       http://<solr_ip>:<solr_port>/solr/ranger_audits
xasecure.audit.destination.solr.user       <user>
xasecure.audit.destination.solr.password   <password>

The parameters are as follows:

• xasecure.audit.is.enabled: Indicates whether Ranger audit is enabled.

• xasecure.audit.destination.solr: Determines if audit logs are stored in Solr.

• xasecure.audit.destination.solr.urls: This parameter specifies the URL of the Solr service. Replace <solr_ip> with the Solr service's IP address and <solr_port> with the corresponding port. Fill in other URL details as required by your setup.

• If Solr has basic authentication enabled, provide the username and password with xasecure.audit.destination.solr.user and xasecure.audit.destination.solr.password.

  For EMR on ECS, find the configuration for xasecure.audit.destination.solr.urls, xasecure.audit.destination.solr.user, and xasecure.audit.destination.solr.password in the Ranger-plugin service's ranger-spark-audit.xml file.

  

After successful configuration, audit logs for user access can be viewed on the Ranger UI's Access tab. For instructions on accessing the Ranger UI, see Access the Web Interface of Open Source Components Through the Console.

4. Connection test

When accessing resources without the necessary permissions via Spark Beeline, an error like the one below will occur.

0: jdbc:hive2://pre-emr-spark-gateway-cn-hang> create table test(id int);
Error: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.kyuubi.plugin.spark.authz.AccessControlException: Permission denied: user [test] does not have [create] privilege on [database=testdb/table=test]
	at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:44)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:325)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:230)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:79)
	at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:63)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:43)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:230)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:225)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:239)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)