All Products
Search

This Product

    Platform For AI:Connect to EMR to process big data

    Document Center

    Platform For AI:Connect to EMR to process big data

    Last Updated:Jun 21, 2026

    You can connect a DSW instance to an EMR cluster to submit and run Spark jobs. This lets you use the high performance of EMR to efficiently process data and seamlessly transition to model development, integrating big data with AI.

    Background information

    Data pre-processing is a crucial, yet often time-consuming and complex, step in machine learning and for large language models. It includes key steps such as data cleaning, data transformation, and feature engineering. To streamline this process, DSW and the open-source big data platform EMR provide an all-in-one solution that integrates big data and AI.

    EMR is a fully managed big data processing service on Alibaba Cloud that integrates Apache Spark. You can use it to easily build, manage, and use Spark clusters in the cloud for large-scale data processing, real-time computing, machine learning tasks, and graph processing.

    Applicability

    • Only the following type of DSW instance can be connected to an EMR cluster:

      A pay-as-you-go DSW instance created in a public resource group.

    • Only the following types of EMR clusters are supported:

      • DataLake cluster

      • A custom cluster with Spark 3 and Hadoop installed

    Note

    Each DSW instance can be connected to only one EMR cluster. After the connection is established, you cannot switch to a different cluster.

    Prerequisites

    • You have activated EMR and created an EMR cluster. For more information, see Create a cluster.

    • You have created a DSW instance. The image's operating system must be Ubuntu 20.04 or an earlier version, and the instance must be in the same VPC and security group as the EMR cluster. For more information, see Create a DSW instance.

      Important

      The DSW instance must use the same VPC and security group as the EMR cluster. Otherwise, subsequent configuration steps will fail.

    Procedure

    Open the tutorial file in DSW

    1. Go to the DSW development environment.

      1. Log on to the PAI console.

      2. In the upper-left corner of the page, select the region where your DSW instance is located.

      3. In the left-side navigation pane, click Workspaces. On the Workspaces page, click the name of the default workspace.

      4. In the left-side navigation pane, choose Model Training > Data Science Workshop (DSW).

      5. In the Actions column of the instance that you want to open, click Open to open the DSW development environment.

    2. On the Launcher page of the Notebook tab, in the Quick Start section, click DSW Gallery under Tool to open the DSW Gallery page.

    3. On the DSW Gallery page, search for Big Data and AI Integration: Submit Spark Jobs to an EMR Cluster and click Open in DSW. DSW automatically downloads the resources and tutorial file required for this tutorial. The tutorial file opens automatically after the download is complete.

    Run the tutorial file

    In the opened tutorial file emr_connect.ipynb, you can view the tutorial content and run the tutorial directly.

    In the tutorial file, click image to run the command in a cell. After a command runs successfully, run the command in the next cell.

    from IPython.display import display, HTML, Javascript
display(Javascript('emr$cluster$selector(element)'))

    This tutorial includes the following four steps:

    1. Select an EMR cluster.

    2. Connect to the specified cluster.

    3. Submit a Spark job using spark-submit.

    4. Run an interactive PySpark application.

    FAQ

    Q: Spark job fails after DSW restart

    After you restart a DSW instance, you must repeat the steps Select an EMR cluster. and Connect to the specified cluster. to re-establish the EMR cluster connection.

    Q: spark-submit: command not found error

    After you connect to the EMR cluster, you must open a new Terminal to load the Spark-related environment variables. You can run the following command to check whether the Spark configuration is active in the current session. If an empty result is returned, the configuration is not active.

    env | grep -i SPARK_HOME

    Q: Python version mismatch error

    This error indicates a mismatch between the Python version on the driver client and the version on the cluster's worker nodes

    Q: ModuleNotFoundError error

    The Python environment on the cluster's executors lacks the packages required by your PySpark application. We recommend that you use the spark.archives configuration to synchronize your local Python environment with the remote cluster, or manually install the dependencies on each worker node.

    Q: PySpark option missing from kernel list

    On the Notebook tab toolbar, select Kernel > Restart Kernel.