Livy is a service that interacts with Spark by using a RESTful API or a remote procedure call (RPC) client library. Livy can be used to submit Spark jobs or snippets of Spark code, synchronously or asynchronously retrieve results, and manage Spark contexts. Livy simplifies the interaction between Spark and application servers so that Spark can be used for interactive web or mobile applications.

Background information

Livy also provides the following features:
  • Has long-running Spark contexts that can be used in multiple Spark jobs by multiple clients.
  • Shares cached Resilient Distributed Datasets (RDDs) across multiple jobs and clients.
  • Manages multiple Spark contexts at the same time.
  • Submits jobs by using a precompiled JAR package, code snippets, a Java API, or a Scala API.
  • Supports security mechanisms.
The following figure shows the basic architecture of Livy. Architecture diagram

Submit a job

You can use one of the following methods to submit a job: