Connect Jupyter Notebook to EMR Serverless Spark - E-MapReduce

Background information

Apache Livy uses a REST API to interact with Spark, simplifying communication between Spark and application servers. For more information about the Livy API, see REST API.

To interact with EMR Serverless Spark in Jupyter Notebook, use the sparkmagic plugin or a Docker image. Choose the method that best suits your needs.

Method	Use case
Method 1: Use a Docker image to quickly start the environment	To quickly set up a self-contained development environment or replicate the same setup across machines, use a Docker image.
Method 2: Use the sparkmagic plugin to start the environment	The sparkmagic plugin for Jupyter Notebook interacts with Spark through a REST API. Currently, sparkmagic supports the Livy, Livy Lighter, and Ilum protocols. Configure the sparkmagic plugin in Jupyter Notebook and use the Livy API of EMR Serverless Spark to create an efficient development environment for interacting with a remote Spark cluster.

Prerequisites

Complete the prerequisites for your chosen method:
- Method 1: Use a Docker image to quickly start the environment: Docker is installed. For more information, see the official Docker documentation.
- Method 2: Use the sparkmagic plugin to start the environment: Jupyter Notebook is installed and running. For more information, see Project Jupyter | Installing Jupyter.
  
  The examples in this topic use Jupyter Notebook with Python 3.8.
A workspace is created. For more information, see Create a workspace.

Method 1: Use a Docker image

Step 1: Create a gateway and an access token

Create and start a gateway.
1. Go to the gateway page.
  1. Log on to the EMR console.
  2. In the left-side navigation pane, choose EMR Serverless > Spark.
  3. On the Spark page, click the name of your target workspace.
  4. On the EMR Serverless Spark page, click O&M Center > Gateway in the left-side navigation pane.
2. Click the Livy Gateway tab.
3. On the Livy Gateway page, click Create Livy Gateway.
4. On the Create Gateway page, enter a Name, such as Livy-gateway, and click create.
  
  You can adjust other parameters based on your requirements. For more information, see Manage Gateways.
5. On the Livy Gateway page, find the gateway you created and click START in the Actions column.
Create a token.
1. On the Gateway page, find Livy-gateway and click Token Management in the Actions column.
2. Click Create Token.
3. In the Create Token dialog box, enter a Name, such as Livy-token, and click OK.
4. Copy the token information.
  
  Important
  After a token is created, you must copy it immediately. You cannot view the token information later. If the token expires or is lost, create a new one or reset it.

Step 2: Pull and start the Docker image

Run the following command to pull the Docker image:

docker pull emr-registry-registry.cn-hangzhou.cr.aliyuncs.com/serverless-spark-public/emr-spark-jupyter:latest

Run the following command to start the image:

docker run -p <host_port>:8888 emr-registry-registry.cn-hangzhou.cr.aliyuncs.com/serverless-spark-public/emr-spark-jupyter:latest <endpoint> <token>

The following table describes the parameters.

Parameter	Description
`<host_port>`	Replace this placeholder with your host port.
`<endpoint>`	Replace this placeholder with the endpoint of the Livy Gateway. On the Livy Gateway page, click the name of the Livy Gateway you created. You can find the endpoint on the Overview tab.
`<token>`	Replace this placeholder with the token you copied in the previous step.

After the image starts, you will see output similar to the following:

[I 2024-09-23 05:38:14.429 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-09-23 05:38:14.432 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-09-23 05:38:14.436 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-09-23 05:38:14.439 ServerApp] notebook | extension was successfully linked.
[I 2024-09-23 05:38:14.439 ServerApp] Writing Jupyter server cookie secret to /root/.local/share/jupyter/runtime/jupyter_cookie_secret
[I 2024-09-23 05:38:14.596 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-09-23 05:38:14.624 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-09-23 05:38:14.625 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-09-23 05:38:14.626 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-09-23 05:38:14.627 LabApp] JupyterLab extension loaded from /root/miniforge3/envs/livy/lib/python3.8/site-packages/jupyterlab
[I 2024-09-23 05:38:14.627 LabApp] JupyterLab application directory is /root/miniforge3/envs/livy/share/jupyter/lab
[I 2024-09-23 05:38:14.628 LabApp] Extension Manager is 'pypi'.
[I 2024-09-23 05:38:14.637 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-09-23 05:38:14.640 ServerApp] notebook | extension was successfully loaded.
[I 2024-09-23 05:38:14.640 ServerApp] Serving notebooks from local directory: /root
[I 2024-09-23 05:38:14.640 ServerApp] Jupyter Server 2.14.2 is running at:
[I 2024-09-23 05:38:14.640 ServerApp] http://6eca53b95ca2:8888/lab?token=258c0dd75e22a10fb6e2c87ac738c2a7ba6a314c6b******
[I 2024-09-23 05:38:14.640 ServerApp]     http://127.0.0.1:8888/lab?token=258c0dd75e22a10fb6e2c87ac738c2a7ba6a314c6b******

Access the Jupyter UI.

Copy the http://127.0.0.1:8888/lab?token=258c0dd75e22a10fb6e2c87ac738c2a7ba6a314c6b****** URL from the output and open it in your browser. You can now use the Jupyter service to connect to EMR Serverless Spark.
Note
- If you connect by using a remote server, replace the IP address 127.0.0.1 with the server's actual IP address.
- If the host_port is not 8888 when you start the image, replace the port number with the actual port number.

Step 3: Test connectivity

On the JupyterLab page, click PySpark.

Run the following code to query all accessible databases:

spark.sql("show databases").show()

The following output is displayed:

%%spark
spark.sql("show databases").show()
Starting Spark application
ID	YARN Application ID	Kind	State	Spark UI	Driver log	User	Current session?
1	livy-pxxx	pyspark	idle	Link		None	✔
SparkSession available as 'spark'.
+--------------------+
|           namespace|
+--------------------+
|             default|
|              xxx db|
|            xxx dome|
|           hive test|
+--------------------+

Method 2: Use the sparkmagic plugin

Step 1: Create a gateway and an access token

Create and start a gateway.
1. Go to the gateway page.
  1. Log on to the EMR console.
  2. In the left-side navigation pane, choose EMR Serverless > Spark.
  3. On the Spark page, click the name of your target workspace.
  4. On the EMR Serverless Spark page, click O&M Center > Gateway in the left-side navigation pane.
2. Click the Livy Gateway tab.
3. On the Livy Gateway page, click Create Livy Gateway.
4. On the Create Gateway page, enter a Name, such as Livy-gateway, and click create.
  
  You can adjust other parameters based on your requirements. For more information, see Manage Gateways.
5. On the Livy Gateway page, find the gateway you created and click START in the Actions column.
Create a token.
1. On the Gateway page, find Livy-gateway and click Token Management in the Actions column.
2. Click Create Token.
3. In the Create Token dialog box, enter a Name, such as Livy-token, and click OK.
4. Copy the token information.
  
  Important
  After a token is created, you must copy it immediately. You cannot view the token information later. If the token expires or is lost, create a new one or reset it.

Step 2: Install and enable the sparkmagic plugin

Run the following command to install the sparkmagic plugin:
```
pip install sparkmagic
```
Enable the extension for your Jupyter environment (Jupyter Notebook or JupyterLab).
- For Jupyter Notebook users:
```
jupyter nbextension enable --py --sys-prefix widgetsnbextension
```
- For JupyterLab users:
```
jupyter labextension install "@jupyter-widgets/jupyterlab-manager"
```

For more details and advanced configuration options for the sparkmagic plugin, see the sparkmagic documentation.

Step 3: Configure and start an interactive Spark session

Access the Jupyter UI. For more information, see JupyterLab.
Load the sparkmagic extension.
```
%load_ext sparkmagic.magics
```
Modify the session startup configuration.
1. Increase the startup timeout to prevent failures caused by resource scheduling delays.
```
import sparkmagic.utils.configuration as conf
conf.override("livy_session_startup_timeout_seconds", 1000)
              
```
2. (Optional) Customize Spark resource configuration.
  
  You can add parameters such as ttl and conf. For more information, see Livy Docs - REST API.
  
  For example, the following configuration modifies the resource settings for the driver.
```
%% spark config
{
   "conf": {
       "spark.driver.cores": "1",
       "spark.driver.memory": "7g"
   }
}
```

Create a session.

Create a Spark session in Jupyter Notebook using either Python or Scala, providing the required configuration.

Python

%spark add -s <session_name> -l python -u https://<endpoint> -a username -p <token>

Scala

%spark add -s <session_name> -l scala -u https://<endpoint> -a username -p <token>

Replace the following placeholders with your actual values.

Parameter	Description
`<session_name>`	The name of the Spark session. You can specify a custom name.
`<endpoint>`	The Endpoint(Public) or Endpoint(Private) from the Overview tab. If you use a private endpoint, ensure that the machine running Jupyter is deployed in the same region as the Livy Gateway and that the `https://` prefix of the endpoint is changed to `http://`.
`<token>`	Replace this placeholder with the token that you copied in Step 1.

The following example uses Python.

%spark add -s test -l python -u https://emr-spark-livy-gateway-cn-hangzhou.data.aliyun.com/api/v1/workspace/w-0aaxxx livycompute/lc-76c4pcxxx -a username -p t6k2zy
Starting Spark application
ID  YARN Application ID  Kind     State  Spark UI  Driver log  User  Current session?
0   livy-idqxxx          pyspark  idle   Link                  None  ✓
SparkSession available as 'spark'.

Please wait for 1 to 5 minutes until the Session status changes to idle. This indicates that the Session has been successfully created and is ready. At this point, the UI displays the details of the newly created Session, and you can start your interactive PySpark development. You can also view the Session information on the Sessions tab for the target Livy Gateway in the Serverless Spark console.

Verify the session.

After the session is successfully created, you can execute code by using %%spark. For example, use the following code to view all databases in the current Spark environment.
```
%%spark
spark.sql("show databases").show()
              
```
The following output is displayed:
```
%%spark
spark.sql("show databases").show()
+---------+
|namespace|
+---------+
|  default|
|   testdb|
+---------+
```

(Optional) Step 4: Release session resources

Automatic release

The session automatically terminates after two hours of inactivity.
Manual release
- Released through the sparkmagic plugin.
```
%spark delete -s <session_name>
```
- Release from the EMR Serverless Spark console:
  
  On the Sessions tab of the target Livy Gateway, find the session you want to release and click Off in the Actions column.