Secure Livy Gateway Access with LDAP Authentication - E-MapReduce

LDAP authentication restricts access to Livy Gateway, requiring clients to provide valid credentials before they can connect and submit jobs. This prevents unauthorized access to sensitive data and compute resources.

Prerequisites

Before you begin, make sure you have:

A Livy Gateway and token. See Manage Livy gateways
(Optional) If you plan to use the OpenLDAP service of an Alibaba Cloud EMR on ECS cluster: a cluster with OpenLDAP selected and at least one user added. See Create a cluster and OpenLDAP user management

Step 1: Prepare the network

Set up network connectivity between EMR Serverless Spark and your Virtual Private Cloud (VPC) so that Livy Gateway can reach the LDAP service. For instructions, see Network connectivity between EMR Serverless Spark and other VPCs.

Step 2: Configure Livy Gateway

Stop the session for the Livy Gateway you want to configure.
Edit the Livy Gateway, and select the connection name from the Network Connectivity drop-down list.

Add the following parameters to livy.conf:

livy.server.auth.type ldap
livy.server.auth.ldap.url ldap://<ldap_url>:<ldap_port>
livy.server.auth.ldap.base-dn <ldap_base_dn>

Parameter	Default	Description
`livy.server.auth.type`	—	Set to `ldap` to enable LDAP authentication.
`livy.server.auth.ldap.url`	—	Connection address of the LDAP service, in the format `ldap://<ldap_url>:<ldap_port>`. For the OpenLDAP service of an EMR on ECS cluster, `<ldap_url>` is the internal IP address or domain name of the master node, and `<ldap_port>` is `10389` by default — for example, `ldap://master-1-1.c-xxxxxxxxxx.cn-hangzhou.emr.aliyuncs.com:10389`.
`livy.server.auth.ldap.base-dn`	—	Base DN for LDAP authentication. For the OpenLDAP service of an EMR on ECS cluster, this is typically `ou=people,o=emr`.

For a high availability LDAP setup, specify multiple addresses separated by spaces: ldap://<ldap_url_1>:<ldap_port> ldap://<ldap_url_2>:<ldap_port>.

Save your changes and restart the Livy Gateway.

For a full list of Livy configuration parameters, see Livy Gateway configuration examples.

Step 3: Connect to Livy Gateway

Choose the method that fits your workflow:

Method	Best for
Jupyter Notebook	Automated operations or dynamic parameter adjustment; requires familiarity with the Sparkmagic plugin
Zeppelin	Data analytics and visualization; only requires editing a configuration file

Method 1: Use Jupyter Notebook

For basic Jupyter Notebook setup, see Interact with Serverless Spark through Jupyter Notebook.

This topic uses Method 2 as an example to demonstrate how to connect to Livy Gateway with LDAP authentication enabled.

Load the sparkmagic plugin:
```
%load_ext sparkmagic.magics
```
Set the session startup timeout (in seconds) to avoid connection failures caused by network or resource delays:
```
import sparkmagic.utils.configuration as conf
conf.override("livy_session_startup_timeout_seconds", 1000)
```
Set the Livy Gateway token as a custom header. When LDAP authentication is enabled, the Authorization field carries the LDAP credentials, so the Livy token must be passed separately:
```
conf.override("custom_headers", {"x-acs-spark-livy-token": "<token>"})
```
Replace <token> with the actual Livy token.
(Optional) Configure Spark parameters such as resource allocation and environment variables.

Start the Livy session and authenticate with your LDAP credentials:

Placeholder	Description
`<ldapusername>`	Your LDAP username
`<ldappassword>`	Your LDAP password
`<endpoint>`	Livy Gateway endpoint, available from the Livy Gateway tab — for example, `emr-spark-livy-gateway-cn-hangzhou.data.aliyun.com/api/v1/workspace/w-xxxxxx/livycompute/lc-xxxxxx`

%spark add -s test -l python -u https://<endpoint> -a <ldapusername> -p <ldappassword>

Method 2: Use Zeppelin

For basic Zeppelin setup, see Connect to EMR Serverless Spark through Zeppelin Livy Interpreter.

The open-source Zeppelin Livy Interpreter does not support LDAP credentials directly. Instead, pass them as custom HTTP headers.

Generate a Base64-encoded credential string: Concatenate your LDAP username and password in the format <username>:<password>, then Base64-encode the result — for example, dXNlcm5hbWU6cGFzc3dvc*****. Record this value as <token>.

Add the following custom header in the Zeppelin Livy Interpreter configuration:

Placeholder or header	Description
`<token>`	The Base64-encoded string from step 1
`x-acs-spark-livy-token`	Livy Gateway token; must match the token configured for the gateway
`Authorization:Basic <token>`	Passes the LDAP credentials using HTTP Basic authentication

zeppelin.livy.http.headers = x-acs-spark-livy-token:<token>;Authorization:Basic <token>

The value is a list of HTTP headers. Use ; to separate multiple headers and : to separate each header name from its value.

Save the configuration and restart the Zeppelin Livy Interpreter.

Troubleshooting

Symptom	Likely cause	Action
Authentication fails in Jupyter Notebook	The `Authorization` header is overwritten by LDAP credentials, leaving the Livy token absent	Verify that `x-acs-spark-livy-token` is set in `custom_headers` (step 3)
Authentication fails in Zeppelin	Incorrect Base64 encoding or malformed header string	Re-encode `<username>:<password>` using a reliable tool and confirm the `;` and `:` delimiters are correct
Session startup times out	Network connectivity between EMR Serverless Spark and your VPC is not established	Confirm the VPC connection is configured and retry

E-MapReduce:Configure and enable LDAP authentication for Livy Gateway