Livy Gateway configuration example - E-MapReduce - Alibaba Cloud Documentation Center

This topic describes the core configuration files and parameters of Livy in detail to help you correctly configure Livy Gateway and use related features in Alibaba Cloud EMR Serverless Spark.

spark-defaults.conf

Purpose

The spark-defaults.conf file sets default parameters for Spark jobs submitted through Livy Gateway. When users create Livy batch jobs or sessions, these default values are automatically applied if specific parameters are not explicitly defined.

Example configuration

Below are some common default parameter configurations.

spark.driver.cores 2
spark.driver.memory 7g
spark.dynamicAllocation.enabled true
spark.dynamicAllocation.initialExecutors 2
spark.dynamicAllocation.maxExecutors 100
spark.dynamicAllocation.minExecutors 1

Parameter descriptions are as follows:

spark.driver.cores and spark.driver.memory: Define the number of CPU cores and memory size for the Driver.
Dynamic resource allocation related parameters:
- spark.dynamicAllocation.enabled=true: Enable dynamic resource allocation.
- spark.dynamicAllocation.minExecutors and spark.dynamicAllocation.maxExecutors: Limit the minimum and maximum number of Executors.
- spark.dynamicAllocation.initialExecutors: Initial number of Executors to allocate.

Supported enhanced parameters

Supported enhanced parameters refer to additional features or extended configuration items provided by Alibaba Cloud EMR Serverless Spark to enhance system security, flexibility, or performance. These parameters are typically designed for specific scenarios (such as multitenancy environments, permission management, etc.) to help users better control task behavior and resource allocation.

Parameter

Description

spark.livy.proxyUser

Used to specify the proxy user when submitting Spark jobs. The default value is root.

This parameter is only effective when livy.impersonation.enabled in livy.conf is set to true. When used with spark-blacklist.conf, it can prevent users from modifying the proxyUser parameter, ensuring security.

livy.conf

Purpose

The livy.conf file is the core configuration file for Livy, used to define the global behavior of Livy Gateway, including authentication methods, timeout settings, session management, and more.

Key parameters

Parameter	Description
`livy.impersonation.enabled`	Whether to allow users to impersonate Spark users through the `proxyUser` parameter. The default value is `false`.
`livy.server.session.timeout`	Session timeout period. Inactive sessions exceeding this time will be automatically closed. The default value is `2h`.
`livy.server.session.timeout-check`	Whether to check for timed-out sessions. The default value is `true`. If set to `false`, sessions will not be automatically closed even if they time out.
`livy.server.session.timeout-check.skip-busy`	Whether to skip busy sessions when checking for timed-out sessions. If skipped, sessions that are executing tasks will not be closed. The default value is `true`.
`livy.server.auth.type`	Specifies the authentication method for Livy Gateway. For example, set to `ldap` when enabling LDAP authentication.
`livy.server.auth.ldap.url`	LDAP address. For example, `ldap://master-1-1.c-xxxxxxxxxxxxx.cn-hangzhou.emr.aliyuncs.com:10389`.
`livy.server.auth.ldap.base-dn`	LDAP base DN. For example, `ou=people,o=emr`.
`livy.server.auth.ldap.enable-start-tls`	Whether to enable TLS encryption for LDAP. The default value is `false`.
`livy.server.auth.ldap.security-authentication`	LDAP security authentication method. The default value is `simple`.
`livy.server.session.state-retain.sec`	Retention time of completed sessions in memory. The default value is `600s`.
`livy.server.session.max-creation`	Maximum number of sessions that can be created simultaneously. The default value is `100`.
`livy.server.session.user.max-creation`	Limits the number of sessions that each user can create. This parameter takes effect only when `livy.impersonation.enabled` is set to `true`. For example, `livy.server.session.user.max-creation {"alice": 6, "bob": 2, "default": 5}` means that Alice can create a maximum of six concurrent sessions, Bob can create a maximum of two, and other users can create a maximum of five.

livy-client.conf

Purpose

The livy-client.conf file defines the runtime parameters of the Livy HTTP client, mainly used to optimize the interaction behavior between the client and Livy Gateway.

Key parameters

Parameter	Description
`livy.rsc.retained-statements`	Maximum number of statements retained in session memory. The default value is 100.
`livy.rsc.sql.num-rows`	Maximum number of rows returned by the SQL interpreter. The default value is 1000.

spark-blacklist.conf

Purpose

The spark-blacklist.conf file restricts parameters that users are not allowed to override in tasks submitted through Livy Gateway. Parameters listed in the blacklist will be forcibly ignored by the system, and users cannot modify these parameters.

Example configuration

Below are some examples of common blacklisted parameters.

spark.driver.cores
spark.driver.memory
spark.executor.cores
spark.executor.memory

This configuration indicates that users cannot set the resources (such as CPU cores and memory size) for Spark Driver and Executor themselves, thus preventing resource abuse.

Supported enhanced parameters

The spark-blacklist.conf also supports the following enhanced parameters to further improve job security and stability.

Parameter	Description
`spark.livy.ttl`	Prevents users from setting the time to live (TTL) for sessions. Ensures that session TTL is uniformly managed by administrators, avoiding resource waste or task interruption caused by users setting TTL too long or too short.
`spark.livy.queue`	Prevents users from setting the queue for job submission. All jobs use the default Livy Gateway queue, avoiding uneven resource allocation caused by users arbitrarily selecting queues.
`spark.livy.proxyUser`	Prevents users from setting the proxy user. The default is root user.