A Livy gateway is a REST-based service that simplifies interactions with Apache Spark. It supports job submission and result querying through HTTP and is compatible with multiple programming languages. With a Livy gateway, you can use tools such as the livy_operator of Airflow or the spark_magic of Jupyter Notebook to easily submit jobs to Serverless Spark and obtain status updates.
Create a Livy gateway
Go to the Gateways page.
Log on to the EMR console.
In the left-side navigation pane, choose .
On the Spark page, click the name of the target workspace.
On the EMR Serverless Spark page, click in the left-side navigation pane.
On the Livy Gateway page, click Create Livy Gateway.
On the Create Livy Gateway page, configure the parameters and click Create.
Parameter
Description
Name
The name of the Livy gateway. The name can contain only lowercase letters, digits, and hyphens (-). It must start and end with a letter or digit.
Livy Gateway Resource
Default value:
1 CPU, 4 GB.Livy Version
The Livy version. Default value: 0.8.0.
Engine Version
The version of the Spark engine that is used by the Livy gateway. For more information about the engine version, see Engine version introduction.
Use Fusion Acceleration
Fusion can accelerate the running of Spark workloads and reduce the total cost of tasks. For billing information, see Billing. For more information about the Fusion engine, see Fusion engine.
Associated Queue
The queue in which the Livy gateway is deployed. When a Spark job is submitted by using a gateway, the Spark job is submitted by using the identity of the gateway creator.
Authentication Method
The authentication mode. You can select only Token.
After you create a gateway, you must generate a unique authentication token for the gateway. This way, you can use the token for identity authentication and access control when you submit requests over the gateway. For more information about how to create a token, see Gateway management.
Network Connection
The network connection that is used to access the data sources or external services in a virtual private cloud (VPC). For information about how to create a network connection, see Establish network connectivity between EMR Serverless Spark and other VPCs.
Environment
The runtime environment. When you use a Livy gateway to submit a job, the resources used to run the job are pre-installed based on the runtime environment.
Endpoint(Public)
By default, this feature is disabled.
If you turn on this switch, the system accesses Livy through a public endpoint. Otherwise, the system accesses Livy through an internal endpoint by default.
Automatic Stop
By default, this feature is disabled.
After you turn on the switch for a gateway, the system automatically stops the gateway if no activity is detected in the gateway in the previous 45 minutes.
spark-defaults.conf
The default configuration file of Spark, which is used to set global default parameters for Spark jobs.
livy.conf
The core configuration file of the Livy server, which is used to define the global behavior of the Livy gateway, including the authentication method (LDAP), session management, and timeout settings. This file controls the runtime parameters of the Livy server, which affects all jobs submitted through the gateway.
livy-client.conf
The configuration file for the runtime parameters of the Livy HTTP client, which is used to define the interaction behavior between the client and the Livy gateway.
spark-blacklist.conf
A security-related configuration file that is used to restrict the configuration items that users are not allowed to modify when they submit Spark jobs. The parameters in the blacklist are forcibly ignored by the system, and users cannot override these parameters.
For more information about Livy configuration files and their parameters, see Livy configuration file examples.
On the Livy Gateway page, click Start in the Actions column of the created Livy gateway.
Manage tokens
When you use a token, add --header `x-acs-spark-livy-token: token` to the header of the request.
On the Livy Gateway page, click Tokens in the Actions column of the target gateway.
Click Create Token.
In the Create Token dialog box, configure the parameters and click OK.
Parameter
Description
Name
The name of the token.
Expired At
The validity period of the token. The validity period must be greater than or equal to 1 day. By default, this parameter is enabled and set to 365 days.
Copy the token.
ImportantAfter the token is created, you must immediately copy it. You will no longer be able to view the token after leaving the page. If your token expires or is lost, you can reset it or create a new one.
View session information
After you create a Spark session using the Livy interface, you can view information about the session, such as its ID and status, on the Sessions tab of a specified Livy gateway.
On the Livy Gateway page, click the name of the target gateway.
Click the Sessions tab.
On the Sessions tab, you can view information about the Spark session that was created using the Livy interface.
References
For more information about applications of Livy gateways, see the following topics: