Configure and enable Ranger authentication for Kyuubi Gateway - E-MapReduce

Apache Ranger provides a centralized permission management framework. You can use Apache Ranger and plug-ins integrated with Spark to implement fine-grained access control on accessing databases, tables and columns through Spark SQL. This is to enhance data security. Kyuubi Gateway supports configuring the Ranger Plugin to enable control for data access.

Prerequisites

You have created a Kyuubi Gateway. For more information, see Manage Kyuubi Gateway.

We recommend the following engine versions:

esr-4.x: esr-4.3.0 and later versions.
esr-3.x: esr-3.3.0 and later versions.
esr-2.x: esr-2.7.0 and later versions.

Precautions

Ranger is primarily responsible for authentication services. User identity verification requires authentication services such as LDAP. For more information, see Configure and enable LDAP authentication for Kyuubi Gateway.

Procedure

Step 1: Prepare the network

You need to configure network connectivity between EMR Serverless Spark and your virtual private cloud (VPC). This is to allow the specified Ranger plug-in to connect to Admin and get permissions. For more information, see Network connectivity between EMR Serverless Spark and other VPCs.

Step 2: Configure the Ranger plug-in

You need to stop the Kyuubi Gateway before giving Ranger authentication to it. Select the created connection name from the Network Connection drop-down list, and add the following configuration items in Spark Configuration. After you modify the configuration, you must restart it to apply the changes.

spark.ranger.plugin.enabled                true
spark.emr.serverless.user.defined.jars     /opt/ranger/ranger-spark.jar
ranger.plugin.spark.policy.rest.url        http://<ranger_admin_ip>:<ranger_admin_port>

The following table describes the parameters:

Parameter	Description
`spark.ranger.plugin.enabled`	Set to `true` to enable Ranger authentication.
`spark.emr.serverless.user.defined.jars`	The path of the custom JAR package. Set to `/opt/ranger/ranger-spark.jar` to use the built-in Ranger plugin in Serverless Spark.
`ranger.plugin.spark.policy.rest.url`	The address of the Ranger Admin service. The format is `http://<ranger_admin_ip>:<ranger_admin_port>`. Replace `<ranger_admin_ip>` and `<ranger_admin_port>` with the private IP address and port of your Ranger Admin service. If you are connecting to the Ranger service of an Alibaba Cloud EMR on ECS cluster, you can set `<ranger_admin_ip>` to the private IP address of the master node and `<ranger_admin_port>` to 6080.

Step 3: (Optional) Configure Ranger Audit

Ranger allows you to specify how to store audit information, like Solr and Hadoop Distributed File System (HDFS). Serverless Spark does not enable Ranger Audit by default. If you need this feature, you can add Ranger Audit-related parameters in Spark Configuration.

For example, to configure a connection to EMR Solr, you can add the following configuration information to Spark Configuration.

xasecure.audit.is.enabled                  true
xasecure.audit.destination.solr            true
xasecure.audit.destination.solr.urls       http://<solr_ip>:<solr_port>/solr/ranger_audits
xasecure.audit.destination.solr.user       <user>
xasecure.audit.destination.solr.password   <password>

The parameters are as following:

xasecure.audit.is.enabled: Specifies whether to enable Ranger Audit.
xasecure.audit.destination.solr: Specifies whether to store audit information in the Solr service.
xasecure.audit.destination.solr.urls: The URL information of the Solr service. You need to configure <solr_ip> and <solr_port> with the IP address and port information of the Solr service. Other URL information should be filled in according to actual requirements.
xasecure.audit.destination.solr.user and xasecure.audit.destination.solr.password: If Basic authentication is enabled for the Solr service, you need to fill in the username and password.
If you are connecting to Ranger in EMR on ECS, you can find xasecure.audit.destination.solr.urls, xasecure.audit.destination.solr.user, and xasecure.audit.destination.solr.password in the ranger-spark-audit.xml configuration file of the Ranger-plugin service.

You can access the Ranger UI after a successful configuration, after submitting a task on EMR Serverless Spark. You can also view the audit information of user access on the Access tab of Ranger Audit. For more information about accessing the Ranger UI, see Access the web UIs of open source components from the console.

Note

You can view audit information on the web UI of Ranger only if you use Solr to store audit information. If you use HDFS or other storage methods that do not support viewing through the Ranger UI, you cannot access them.

Step 4: Test

Use Spark Beeline to test the connection. For more connection methods, see Connect to Kyuubi Gateway. If you do not have permissions to some databases, tables, or other resources but try to access them, you can see a Permission denied message.

Note

When verifying permissions, please take note of some default permissions in Ranger. For example, all users have permissions to switch and create databases, and the owners of databases and tables have full permissions on their databases and tables. We recommend you to verify the permissions of User B on resources created by User A, such as databases and tables. If you verify permissions of users on resources they created themselves, you may wrongly find that some settings do not take effect, because of permission strategy of the owner.
If Ranger Admin is incorrectly configured, SQL statements may execute successfully without errors but actually not taking effect.