Livy Gateway is a REST-based service that simplifies interaction with Apache Spark. It supports submitting jobs and querying results over HTTP and is compatible with multiple programming languages. With Livy Gateway, you can integrate tools such as thelivy_operator of Airflow orspark_magic of Jupyter Notebook to easily submit jobs to Serverless Spark and get status updates.
Create a Livy Gateway
Go to the Gateway page.
Log on to the EMR console.
In the navigation pane on the left, choose .
On the Spark page, click the name of the target workspace.
On the EMR Serverless Spark page, in the navigation pane on the left, click .
On the Livy Gateway page, click Create Livy Gateway.
On the Create Livy Gateway page, configure the following parameters and click Create.
Parameter
Description
Name
The name of the gateway. The name can contain only lowercase letters, digits, and hyphens (-), and must start and end with a letter or digit.
Livy Gateway Resources
Default:
1 CPU, 4 GB.Livy Version
Default: 0.8.0, the latest version.
Engine Version
The engine version that the gateway uses. For more information, see engine versions.
Use Fusion Acceleration
Fusion can accelerate Spark workloads and reduce the total cost of jobs. For billing information, see Product Billing. For more information about the Fusion engine, see Fusion engine.
Associated Queue
Jobs submitted through the gateway use the creator's identity.
Authentication Method
Only token authentication is supported.
After you create a gateway, you must generate a unique authentication token for it. This token is used for authentication and access control in subsequent requests. See Token management for instructions on creating a token.
Mount Integrated File Directory
This feature is disabled by default. To use this feature, add a file directory on the Artifacts page, on the Integrated File Directory tab. For more information, see Manage the integrated file directory.
Enabling this feature mounts the managed file directory to the application, allowing it to directly read from and write to files in the directory.
The mount operation consumes driver compute resources. The amount consumed is the greater of the following two values:
Fixed resources: 0.3 vCPUs + 1 GB memory.
Dynamic resources: 10% of the
spark.driverresources (that is, 10% of the cores and memory ofspark.driver).
For example, if
spark.driveris configured with 4 cores and 8 GB of memory, the dynamic resources are 0.4 vCPUs + 0.8 GB of memory. In this case, the actual consumed resources aremax(0.3 vCPUs + 1 GB, 0.4 vCPUs + 0.8 GB), which is 0.4 vCPUs + 1 GB of memory.NoteMount scope: By default, the file directory is mounted only to the driver. To mount it to executors as well, enable Mount to Executor.
Multiple directories: You can mount multiple integrated file directories. However, CPFS directories cannot be used together with other types. For example, you can mount multiple OSS and NAS directories together, but you cannot mount CPFS with OSS or NAS directories.
Network requirements: When you mount a NAS or CPFS file directory, you must configure a network connection. The VPC of the network connection must be the same as the VPC of the NAS or CPFS mount target.
Network Connection
Select an existing network connection to access data sources in a VPC or external services. For more information about how to create a network connection, see Network connectivity between EMR Serverless Spark and other VPCs.
Environment
When you submit jobs through the Livy Gateway, the job uses the resources pre-configured in the selected runtime environment.
Endpoint(Public)
Disabled by default.
If you enable this feature, the system accesses Livy through a public endpoint. Otherwise, it uses an internal endpoint.
Automatic Stop
Disabled by default.
If you enable this feature, the system automatically stops the gateway if the gateway is inactive for 45 minutes.
spark-defaults.conf
This file sets the global default parameters for Spark jobs.
livy.conf
The core configuration file for the Livy server. It defines the global behavior of the Livy Gateway, including the authentication method (such as LDAP), session management, and timeout settings. This file controls the runtime parameters of the Livy server and affects all jobs submitted through this gateway.
livy-client.conf
This file defines the interaction behavior between the client and the Livy Gateway.
spark-blacklist.conf
The system ignores parameters in this blacklist, and users cannot override them.
For more information about Livy configuration files and their parameters, see Livy Gateway configuration examples.
On the Livy Gateway page, in the Start column for your gateway, click Start.
Token management
To use a token, add the--header 'x-acs-spark-livy-token: ' header to your request.
On the Livy Gateway page, in the Token Management column for the target gateway, click Tokens.
Click Create Token.
In the Create Token dialog box, configure the following parameters and click OK.
Parameter
Description
Name
The name of the new token.
Expired At
Set the token's expiration time. The value must be at least 1 day. By default, the token expires in 365 days.
Copy the token information.
ImportantCopy the token immediately after it is created. You cannot view it again. If a token is lost or expires, you must create a new one or reset the existing one.
Session information
When you create a Spark session using Livy, the system displays the session's details, such as its ID and status.
On the Livy Gateway page, click the name of the target gateway.
Click the Sessions tab.
This page shows the Spark sessions created through this Livy Gateway.
The session list includes the following information: Session ID, Session Status, Creation time, Resource configuration (in JSON format, including parameters such as
kind,driverMemory,driverCores,executorMemory,executorCores, andnumExecutors), a Spark UI link, and Actions (to close the session).
Related documentation
For Livy Gateway use cases, see the following documents: