AnalyticDB:Jupyter interactive job development

Connect with the AnalyticDB for MySQL image

1. Install and start Docker.

2. After Docker starts, run the following command to pull the AnalyticDB for MySQL Jupyter image:

   docker pull registry.cn-hangzhou.aliyuncs.com/adb-public-image/adb-spark-public-image:adb.notebook.0.5.pre

3. Start the interactive Jupyter development environment.

   Use the following command format:

   docker run -it \
     -p {host_port}:8888 \
     -v {host_path}:{container_path} \
     registry.cn-hangzhou.aliyuncs.com/adb-public-image/adb-spark-public-image:adb.notebook.0.5.pre \
     -d {cluster_id} \
     -r {resource_group_name} \
     -e {api_endpoint} \
     -i {access_key_id} \
     -k {access_key_secret} \
     -t {sts_token} # Provide either an STS token or an AccessKey pair.

   Parameters:

   Parameter	Required	Description
   -p	No	Maps a host port to a container port. The format is -p host_port:container_port. You can specify any host port. The container port must be 8888. This example uses -p 8888:8888.
   -v	No	If you do not mount a host folder, files edited in the Docker container are lost when the container stops. Stopping the container also terminates all running interactive Spark jobs. To avoid losing your files, use one of the following methods: When starting the interactive Jupyter environment, mount a host directory into the container and save your job files to that path. The format is -v host_path:container_path. The container path can be any path, but /root/jupyter is recommended. Before stopping the container, ensure that all files are properly copied and saved. This example uses -v /home/admin/notebook:/root/jupyter, which mounts the /home/admin/notebook directory on the host to the /root/jupyter directory in the container. Note You must save your in-progress notebook file to the /root/jupyter folder. After you stop the Docker container, the file appears in the /home/admin/notebook folder on the host. You can continue your work when you restart the container. For more information, see the Docker volume management documentation.
   -d	Yes	The cluster ID of the AnalyticDB for MySQL Enterprise, Basic, or Data Lakehouse Edition cluster. You can log on to the AnalyticDB for MySQL console and find the cluster ID in the Cluster List list.
   -r	Yes	The name of the AnalyticDB for MySQL job resource group. You can log on to the AnalyticDB for MySQL console, go to the Cluster Management > Resource Management page, and click the Resource Group Management tab to view resource group information.
   -e	Yes	The API endpoint for your AnalyticDB for MySQL cluster. For details, see Endpoints.
   -i	Conditional	The AccessKey ID and AccessKey secret of an Alibaba Cloud account or RAM user. For information about how to view your AccessKey ID and AccessKey secret, see Accounts and permissions.
   -k
   -t	Conditional	A temporary credential for a RAM role, or STS token. A RAM user with the required permissions can call the AssumeRole operation to obtain an STS token for a RAM role. This token can then be used to access Alibaba Cloud resources.

   Example:

   docker run -it -p 8888:8888 -v /home/admin/notebook:/root/jupyter registry.cn-hangzhou.aliyuncs.com/adb-public-image/adb-spark-public-image:adb.notebook.0.5.pre -d amv-bp164l******** -r test -e adb.aliyuncs.com -i LTAI**************** -k ****************

   After startup, the following message appears. Copy the URL http://127.0.0.1:8888/lab?token=1e2caca216c1fd159da607c6360c82213b643605f11ef291 into your browser to connect to AnalyticDB for MySQL Spark through the Jupyter service.

   [I 2023-11-24 09:55:09.852 ServerApp] nbclassic | extension was successfully loaded.
   [I 2023-11-24 09:55:09.852 ServerApp] sparkmagic extension enabled!
   [I 2023-11-24 09:55:09.852 ServerApp] sparkmagic | extension was successfully loaded.
   [I 2023-11-24 09:55:09.853 ServerApp] Serving notebooks from local directory: /root/jupyter
   [I 2023-11-24 09:55:09.853 ServerApp] Jupyter Server 1.24.0 is running at:
   [I 2023-11-24 09:55:09.853 ServerApp] http://419e63fc7821:8888/lab?token=1e2caca216c1fd159da607c6360c82213b643605f11ef291
   [I 2023-11-24 09:55:09.853 ServerApp]  or http://127.0.0.1:8888/lab?token=1e2caca216c1fd159da607c6360c82213b643605f11ef291
   [I 2023-11-24 09:55:09.853 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

   Note

   If an error occurs when starting the Jupyter environment, check the proxy_{timestamp}.log file to troubleshoot the problem.

Connect with a local Jupyter Notebook

jupyter lab

[I 2025-07-02 17:36:16.051 ServerApp] Serving notebooks from local directory: /home/newuser
[I 2025-07-02 17:36:16.052 ServerApp] Jupyter Server 2.16.0 is running at:
[I 2025-07-02 17:36:16.052 ServerApp] http://419e63fc7821:8888/lab?token=1e2caca216c1fd159da607c6360c82213b643605f11ef291
[I 2025-07-02 17:36:16.052 ServerApp]     http://127.0.0.1:8888/lab?token=1e2caca216c1fd159da607c6360c82213b643605f11ef291
[I 2025-07-02 17:36:16.052 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

Start and configure the Spark session

1. After connecting to AnalyticDB for MySQL Spark through the Jupyter service, click PySpark to create a new PySpark job. By default, the Spark job runs with the following configuration parameters:

   {
      "kind": "pyspark",
       "heartbeatTimeoutInSecond": "60",
       "spark.driver.resourceSpec": "medium",
       "spark.executor.resourceSpec": "medium",
       "spark.executor.instances": "1",
       "spark.dynamicAllocation.shuffleTracking.enabled": "true",
       "spark.dynamicAllocation.enabled": "true",
       "spark.dynamicAllocation.minExecutors": "0",
       "spark.dynamicAllocation.maxExecutors": "1",
       "spark.adb.sessionTTLSeconds": "1200"
   }

2. To modify the Spark application configuration parameters, use the %%configure -f command.

   1. Restart the kernel.

      At the top of the page, click Kernel > Restart Kernel and Clear All Outputs to ensure that no Spark applications are running in the current Jupyter notebook.

   2. Enter the custom Spark application configuration parameters in a Jupyter Notebook cell.

      Important

      When customizing Spark application configuration parameters, you must set spark.dynamicAllocation.enabled to false.

      This configuration allocates 32 executors for the Spark job. Each executor has a medium specification (2 cores, 8 GB), and the job can be allocated a total of 64 ACUs of computing resources.

      %%configure -f
      {
         "spark.driver.resourceSpec":"large",
         "spark.sql.hive.metastore.version":"adb",
         "spark.executor.resourceSpec":"medium",
         "spark.adb.executorDiskSize":"100Gi",
         "spark.executor.instances":"32",
         "spark.dynamicAllocation.enabled":"false",
         "spark.network.timeout":"30000",
         "spark.memory.fraction":"0.75",
         "spark.memory.storageFraction":"0.3"
      }

      For a detailed description of Spark application configuration parameters, see Spark application configuration parameters and the official Spark documentation.

3. Click the button to apply the Spark application configuration parameters.

   Important

   • Custom configuration parameters do not persist after you close the Jupyter Notebook page. You must reconfigure the Spark application parameters when you reopen the page; otherwise, the job runs with the default parameters.

   • When you run Spark jobs in a Jupyter Notebook, all configuration options are written directly into the JSON structure, not into a conf JSON object as required for submitting batch jobs.

Run a job

1. Run the spark command to start a SparkSession.

   spark

   The following output indicates that the Spark session has been created successfully and is in an idle state:

   Starting Spark application
   ID    YARN Application ID    Kind    State    Spark UI    Driver log    User    Current session?
   2xxx    s202xxxxxxxx11018    pyspark    idle    Link        None    ✓
   SparkSession available as 'spark'.
   <pyspark.sql.session.SparkSession object at 0x7fa7a1f9c278>

   Note

   Click Link in the output to open the Spark UI, where you can view job logs and other information.

2. Run Spark SQL in a Jupyter Notebook cell to query the list of available databases in your AnalyticDB for MySQL cluster.

   Important

   You must prefix Spark SQL code with %%sql; otherwise, it is parsed as Python code by default. You can run %%help to view available magic commands and their usage.

   %%sql
   show databases

   The result includes a namespace column containing two databases: ADB_External_TPCH_10GB and demo. The query result is consistent with the databases in AnalyticDB for MySQL.