Livy is a service that interacts with Spark using a RESTful API or a remote procedure call (RPC) client library. Use Livy to submit Spark jobs or code snippets, synchronously or asynchronously retrieve results, and manage Spark contexts. Livy simplifies the interaction between Spark and application servers, enabling Spark to be used for interactive web or mobile applications.
Key capabilities
| Capability | Description |
|---|---|
| Persistent Spark contexts | Maintain long-running Spark contexts shared across multiple jobs and clients, eliminating per-request startup overhead. |
| Cached RDD sharing | Share cached Resilient Distributed Datasets (RDDs) across multiple jobs and clients. |
| Concurrent context management | Run multiple Spark contexts simultaneously. |
| Flexible job submission | Submit jobs using a precompiled JAR package, code snippets, a Java API, or a Scala API. |
| Security integration | Connect with security mechanisms to control access to Spark resources. |
Supported versions
| EMR version | Livy support |
|---|---|
| 5.6.0 and earlier | Included by default |
| 5.8.0 and later | Not included. Install Livy manually and evaluate compatibility and security risks before use. |
Submit a job
Livy supports four methods for submitting Spark jobs. Choose based on your integration:
| Method | Best for | Documentation |
|---|---|---|
| REST API | Language-agnostic integrations, shell scripts, and HTTP-based workflows | Official REST API reference |
| Programmatic API | JVM applications that submit jobs without going through HTTP | Official programmatic API reference |
| Java API | Type-safe Java clients for submitting jobs and retrieving results | Java API reference |
| Scala API | Idiomatic Scala clients for submitting jobs and managing sessions | Scala API reference |