DSW connects to an EMR cluster to submit and run Apache Spark jobs, combining EMR's big data processing capabilities with AI model development in a single workflow.
Background
Data preprocessing is a critical but time-consuming step in machine learning and large language model (LLM) development. It typically involves data cleaning, transformation, and feature engineering. DSW integrates with EMR, Alibaba Cloud's fully managed big data platform, to combine large-scale data processing with AI model development in one place.
EMR includes Apache Spark. EMR lets you build, manage, and use Spark clusters in the cloud for batch processing, real-time computing, machine learning, and graph processing.
Limitations
Only the following DSW instance type can connect to an EMR cluster:
A pay-as-you-go DSW instance created using a public resource group.
Only the following EMR cluster types are supported:
DataLake cluster
Custom cluster with Spark 3 and Apache Hadoop installed
Each DSW instance can connect to at most one EMR cluster. Once connected, you can't switch to a different cluster.
Prerequisites
EMR activated and an EMR cluster created. For details, see Create a cluster.
A DSW instance created with an image running Ubuntu 20.04 or earlier, and with the same VPC and security group as the EMR cluster. For details, see Create a DSW instance.
ImportantThe DSW instance and the EMR cluster must use the same VPC and security group. A mismatch causes configuration failures in subsequent steps.
Procedure
Open the tutorial file in DSW
Go to the DSW development environment.
Log in to the PAI console.
In the upper-left corner of the page, select the region where your DSW instance is located.
In the left navigation pane, click Workspaces. On the workspace list page, click the default workspace name to enter the workspace.
In the left navigation pane, choose Model Training > Data Science Workshop (DSW).
In the Actions column of the target instance, click Open to open the DSW development environment.
On the Launcher page, go to the Gallery tab. Search for
Big Data and AI Integration: Submit Spark Jobs to an EMR Cluster, then click Open in DSW. The required resources and files are downloaded to your DSW instance automatically. When the download is complete, the tutorial file opens.
Run the tutorial file
The tutorial file emr_connect.ipynb opens automatically.
Click 
to run each step. Complete one step before moving to the next.
from IPython.display import display, HTML, Javascript
display(Javascript('emr$cluster$selector(element)'))The tutorial includes the following four steps:
Select an EMR cluster.
Connect to the selected cluster.
Submit a job using spark-submit.
Run a PySpark interactive application.
FAQ
After restarting a DSW instance, Spark jobs fail. How do I fix this?
After restarting a DSW instance, re-run Select an EMR cluster. and Connect to the selected cluster. to reinitialize the EMR cluster connection.
The error "spark-submit: command not found" appears. How do I fix this?
After connecting to the EMR cluster, open a new terminal to load the Spark environment variables. Run the following command to check whether the Spark configuration is active in the current session. If the output is empty, the configuration hasn't taken effect.
env | grep -i SPARK_HOMEThe error "Python in worker has different version than that in driver " appears. How do I fix this?
This error means the Python version on the driver client doesn't match the Python version on the cluster's worker nodes.
The error "ModuleNotFoundError: No module named *" appears. How do I fix this?
The Python environment on the cluster's executors is missing packages that the PySpark application depends on. Use spark.archives to sync the local Python environment to the remote cluster, or manually install the required packages on each worker node.
After creating and initializing a PySpark kernel, PySpark doesn't appear in the kernel drop-down list when creating a new Notebook. How do I fix this?
In the Notebook tab toolbar, choose Kernel > Restart Kernel.