Configure LDAP authentication for a Livy Gateway - E-MapReduce

LDAP authentication requires clients to provide a valid username and password before connecting to the Livy Gateway, preventing unauthorized access to sensitive data and resources.

Prerequisites

You have created a Livy Gateway and a token. For more information, see Manage gateways.
Optional: To use the OpenLDAP service of an Alibaba Cloud EMR on ECS cluster, create a cluster, select the OpenLDAP service, and add users. For more information, see Create a cluster and Manage OpenLDAP users.

Procedure

Step 1: Prepare the network

You must establish network connectivity between EMR Serverless Spark and your Virtual Private Cloud (VPC) to allow the Livy Gateway to reach the LDAP service for authentication. For instructions, see Connect EMR Serverless Spark to other VPCs.

Step 2: Configure the Livy Gateway

To enable LDAP authentication for a Livy Gateway, stop and edit the gateway. In the gateway settings, select your connection from the Normal Network Connection drop-down list and add the following parameters to the livy.conf file. Finally, restart the Livy Gateway to apply the changes.

livy.server.auth.type ldap
livy.server.auth.ldap.url ldap://<ldap_url>:<ldap_port>
livy.server.auth.ldap.base-dn <ldap_base_dn>

The following table describes the parameters. Replace the placeholders with your actual values.

Parameter	Description
`livy.server.auth.type`	Set this parameter to `ldap` to enable LDAP authentication.
`livy.server.auth.ldap.url`	The connection address of the LDAP service, in the format `ldap://<ldap_url>:<ldap_port>`. If you connect to the OpenLDAP service on an Alibaba Cloud EMR on ECS cluster, set `<ldap_url>` to the internal IP address or domain name of the master node. The default `<ldap_port>` is 10389. For example: `ldap://master-1-1.c-xxxxxxxxxx.cn-hangzhou.emr.aliyuncs.com:10389`. Note For a high-availability LDAP service, separate multiple connection addresses with a space. For example: `ldap://<ldap_url_1>:<ldap_port> ldap://<ldap_url_2>:<ldap_port>`.
`livy.server.auth.ldap.base-dn`	The base DN for LDAP service authentication. If you connect to the OpenLDAP service on an Alibaba Cloud EMR on ECS cluster, this value is typically `ou=people,o=emr`.

For more information about Livy configuration files and their parameters, see Livy Gateway configuration examples.

Step 3: Connect to the Livy Gateway

Two methods are available for connecting to an LDAP-enabled Livy Gateway:

Jupyter Notebook: Best suited for adjusting parameters at runtime or automating operations through code. Requires familiarity with the Sparkmagic plugin.
Zeppelin: Best suited for data analysis and visualization. Simpler setup — you only need to modify a configuration file.

Method 1: Jupyter Notebook

For basic usage instructions, see Interact with EMR Serverless Spark by using a Jupyter Notebook.

Load the Sparkmagic plugin.
```
%load_ext sparkmagic.magics
```
Set the default timeout. Adjusting the startup timeout for the Livy session (in seconds) helps prevent connection failures caused by network or resource issues.
```
import sparkmagic.utils.configuration as conf
conf.override("livy_session_startup_timeout_seconds", 1000)
```
Configure the Livy Gateway token. Because LDAP authentication uses the Authorization header for the username and password, you must send the Livy Gateway token in a custom header.
```
conf.override("custom_headers", {"x-acs-spark-livy-token": "<token>"})
```
Note: Replace <token> with your actual Livy token.
Optional: Set Spark parameters.

As needed, configure Spark parameters, such as resource allocation and environment variables.
Start the Livy session. Use the following command to create and start a Livy session, authenticating with your LDAP username and password.
```
%spark add -s test -l python -u https://<endpoint> -a <ldapusername> -p <ldappassword>
```
Parameters:
- <ldapusername>: Replace this with your LDAP username.
- <ldappassword>: Replace this with your LDAP password.
- <endpoint>: Replace this with the endpoint of your Livy Gateway. You can obtain this from the Livy Gateway tab. For example: emr-spark-livy-gateway-cn-hangzhou.data.aliyun.com/api/v1/workspace/w-xxxxxx/livycompute/lc-xxxxxx.

Method 2: Zeppelin

For basic usage instructions, see Connect to EMR Serverless Spark by using the Zeppelin Livy Interpreter.

The open-source Livy Interpreter for Zeppelin does not natively support LDAP credentials. You can enable LDAP authentication by adding a custom HTTP header instead.

Generate a Base64-encoded authentication string. Concatenate your LDAP username and password into the format <username>:<password>, and then apply Base64 encoding to the resulting string. The output, such as dXNlcm5hbWU6cGFzc3dvc*****, is the <token> you will use.
Configure the HTTP headers for the Livy Interpreter. In the Zeppelin Livy Interpreter configuration, add the following custom header.
```
zeppelin.livy.http.headers = x-acs-spark-livy-token:<token>;Authorization:Basic <token>
```
Parameters:
- <token>: Replace this with the Base64-encoded string from the previous step.
- x-acs-spark-livy-token: The token for the Livy Gateway. Ensure that its value matches your actual configuration.
- Authorization:Basic <token>: Specifies the LDAP authentication information.
Save the configuration and restart the Zeppelin Livy Interpreter to apply the changes.