Configure and enable LDAP authentication for Livy Gateway - E-MapReduce

By enabling LDAP authentication, you can enhance the security of Livy, ensuring that clients can connect to Livy Gateway and perform operations only after providing the correct username and password, effectively preventing unauthorized access to sensitive data and features.

Prerequisites

You have created a Livy Gateway and Token. For more information, see Manage Livy gateways.
Optional: To use the OpenLDAP service of an Alibaba Cloud EMR on ECS cluster, you must create a cluster with the OpenLDAP service selected and add users. For more information, see Create a cluster and OpenLDAP user management.

Procedure

Step 1: Prepare the network

Before you start configuration, you must set up the network to ensure connectivity between Serverless Spark and your Virtual Private Cloud (VPC), so that Livy Gateway can connect to the LDAP service for authentication. For more information, see Network connectivity between EMR Serverless Spark and other VPCs.

Step 2: Configure Livy Gateway

For Livy Gateway that you want to enable LDAP authentication for, you must first stop the session. Then edit the Livy Gateway, select the created connection name from the Network Connectivity drop-down list, and add the following configuration items to livy.conf. After editing, you must restart the Livy Gateway to make the changes take effect.

livy.server.auth.type ldap
livy.server.auth.ldap.url ldap://<ldap_url>:<ldap_port>
livy.server.auth.ldap.base-dn <ldap_base_dn>

Parameter description:

Parameter	Description
`livy.server.auth.type`	Fixed as `ldap`, indicating that LDAP authentication is enabled.
`livy.server.auth.ldap.url`	The connection address of the LDAP service, in the format of `ldap://<ldap_url>:<ldap_port>`. If you are connecting to the OpenLDAP service of an Alibaba Cloud EMR on ECS cluster, `<ldap_url>` should be the internal IP address or domain name of the master node, and `<ldap_port>` is 10389 by default, for example, `ldap://master-1-1.c-xxxxxxxxxx.cn-hangzhou.emr.aliyuncs.com:10389`. Note If LDAP is a high availability service, multiple LDAP connection addresses should be separated by spaces, for example, `ldap://<ldap_url_1>:<ldap_port> ldap://<ldap_url_2>:<ldap_port>`.
`livy.server.auth.ldap.base-dn`	The base DN used for LDAP service authentication. If you are connecting to the OpenLDAP service of an Alibaba Cloud EMR on ECS cluster, it is typically `ou=people,o=emr`.

For more information about Livy configuration files and parameters, see Livy Gateway configuration examples.

Step 3: Connect to Livy Gateway with LDAP authentication enabled

This topic provides the following two methods to connect to Livy Gateway with LDAP authentication enabled:

The Jupyter Notebook method is suitable for scenarios that require dynamic parameter adjustment or automated operations through code. This method is slightly more complex and requires familiarity with the Sparkmagic plugin.
The Zeppelin method is more suitable for data analytics and visualization scenarios. This method is simple to use and only requires modifying the configuration file.

Method 1: Use Jupyter Notebook

For basic usage, see Interact with Serverless Spark through Jupyter Notebook.

This topic uses Method 2 as an example to demonstrate how to connect to Livy Gateway with LDAP authentication enabled.

Import the sparkmagic plugin.
```
%load_ext sparkmagic.magics
```
Set the default timeout. Adjust the startup timeout for the Livy Session (in seconds) to avoid connection failures due to network or resource issues.
```
import sparkmagic.utils.configuration as conf
conf.override("livy_session_startup_timeout_seconds", 1000)
```
Configure the Livy Gateway Token. After enabling LDAP authentication, the Authorization field in the request will be filled with the LDAP username and password. To ensure normal communication, you need to manually set the custom Token for Livy Gateway.
```
conf.override("custom_headers", {"x-acs-spark-livy-token": "<token>"})
```
Note: Replace <token> with the actual Livy Token.
(Optional) Set Spark parameters.
Configure Spark-related parameters (such as resource configuration and environment variables) based on your actual needs.
Start the Livy Session. Use the following command to create and start a Livy Session, while authenticating with the LDAP username and password.
```
%spark add -s test -l python -u https://<endpoint> -a <ldapusername> -p <ldappassword>
```
Parameters:
- <ldapusername>: Replace with your LDAP username.
- <ldappassword>: Replace with your LDAP password.
- <endpoint>: Replace with the actual endpoint of the Livy Gateway, which can be obtained from the Livy Gateway tab. For example, emr-spark-livy-gateway-cn-hangzhou.data.aliyun.com/api/v1/workspace/w-xxxxxx/livycompute/lc-xxxxxx.

Method 2: Use Zeppelin

For basic usage, see Connect to EMR Serverless Spark through Zeppelin Livy Interpreter.

The open-source Zeppelin Livy Interpreter itself does not support directly configuring LDAP username and password. However, you can implement LDAP authentication by adding custom HTTP headers.

Generate Base64-encoded authentication information. Concatenate the LDAP username and password in the format <username>:<password>, then Base64-encode this string to generate an encoded string (for example: dXNlcm5hbWU6cGFzc3dvc*****). Record this encoded result as <token>.
Configure HTTP Headers for the Livy Interpreter. Add the following custom header in the Zeppelin Livy Interpreter configuration.
```
zeppelin.livy.http.headers = x-acs-spark-livy-token:<token>;Authorization:Basic <token>
```
Parameters:
- <token>: Replace with the Base64-encoded string generated in the previous step.
- x-acs-spark-livy-token: This is the token for Livy Gateway. Ensure its value is consistent with the actual configuration.
- Authorization:Basic <token>: Used to pass LDAP authentication information.
Save and restart the Interpreter. After completing the configuration, save the changes and restart the Zeppelin Livy Interpreter to make the configuration take effect.