Livy is a service that interacts with Spark using a RESTful API or a remote procedure call (RPC) client library. You can use Livy to submit Spark jobs or code snippets, synchronously or asynchronously retrieve results, and manage Spark contexts. Livy simplifies the interaction between Spark and application servers, enabling Spark to be used for interactive web or mobile applications.
Background
Livy also provides the following features:
Maintains long-running Spark contexts that can be used in multiple Spark jobs by multiple clients.
Shares cached Resilient Distributed Datasets (RDDs) across multiple jobs and clients.
Manages multiple Spark contexts simultaneously.
Submits jobs using a precompiled JAR package, code snippets, a Java API, or a Scala API.
Supports security mechanisms.
Supported versions
EMR 5.6.0 and earlier versions support the Livy component by default. If you are using EMR 5.8.0 or later, you need to install Livy yourself and evaluate compatibility and security risks.
Submit a job
You can use one of the following methods to submit a job: