Use Spark clusters

Use cases

Spark clusters are suited for:

Short-running queries that need low startup latency.
Workloads that submit batch tasks frequently.

If your tasks are infrequent and startup time is not a concern, use the standard Serverless mode to submit tasks on demand and avoid the cost of an idle cluster.

Limitations

Task concurrency: A single Spark cluster supports up to 200 concurrent tasks by default. To request a higher limit, submit a ticket.
Kyuubi/Livy Gateway: Tasks submitted through Kyuubi Gateway or Livy Gateway cannot be directed to a Spark cluster.
Authentication: LDAP and Kerberos authentication are not supported.
Authorization: Ranger authorization is not supported.
Managed file directory mount: Mounting an integrated file directory is not supported.

Billing

After a Spark cluster starts, you are billed continuously for the resources it occupies (Master and Worker nodes), regardless of whether any tasks are running. Tasks submitted to the cluster incur no additional charges because their resource consumption is covered by the cluster fee.

For pricing details, see For pricing details, see ..

Create a Spark cluster

Log on to the E-MapReduce console.
In the left-side navigation pane, choose EMR Serverless > Spark.
Click the name of the target workspace to open the Spark console.
In the left-side navigation pane, choose O&M Center > Cluster Management.
On the Clusters page, click the Spark Cluster tab.
Click Create Spark Cluster.

Configure the cluster settings.

Parameter	Description
Cluster Name	A custom name for the Spark cluster.
Engine Version	The Spark engine version to use.
Service High Availability	Whether to enable service high availability. Disabled by default.
Cluster Resource Configuration	Configure Master and Worker nodes: - Resource Queue: Select the resource queue for deploying the nodes. - Specification: Select the node specification (vCPU, memory). - Number / Elastic configuration: Set the node count. Worker nodes support elastic scaling with a minimum and maximum node count.
Network Connection	The network connection for the cluster. To create a new one, go to the Network Connection page.

Click OK.

Submit tasks to a Spark cluster

Submit tasks to a Spark cluster in any of the following ways.

Note

When you submit a task to a Spark cluster through the console or workflow orchestration, the system automatically starts the cluster if it is not running.
When a task runs on a Spark cluster, the engine version, network connection, and integrated file directory mount settings are determined by the cluster configuration, not the task configuration.

Method 1: Submit from the Development page

On the Development page, create a batch task of type SQL, JAR, or PySpark.
Click the execution resource dropdown in the upper-right corner and select the target Spark cluster.
Click Publish or Run.

Method 2: Submit via spark-submit, API, or DataWorks

Add the following Spark configuration to your submission parameters to direct the task to a specific Spark cluster:

spark.emr.serverless.Spark.clusterId <your-Spark-cluster-id>

Replace <your-Spark-cluster-id> with the actual Spark cluster ID. You can find the cluster ID on the cluster list page or the cluster details page.

Note

Note Tasks submitted from DataWorks through the Serverless Spark Kyuubi path cannot be directed to a Spark cluster.

Method 3: Submit a streaming task from the Workflows page

When starting a streaming task on the Workflows page, select a Spark cluster as the execution resource.

View task history

Tasks submitted to a Spark cluster are recorded in the task history. Whether submitted from the Development page, spark-submit, or the API, all tasks appear under O&M Center > Job History.

FAQ

What is the difference between a Spark cluster and Serverless mode?

Aspect	Serverless mode	Spark cluster mode
Resource model	Resources spin up on demand and are released after use.	An always-on cluster that keeps running once started. You can stop or resize it at any time.
Startup speed	Tasks must wait for resources to spin up.	Tasks are submitted directly to a running cluster, resulting in faster startup.
Billing	Billed based on the resources each task actually uses.	Billed continuously based on the resources the cluster occupies.
Use cases	Infrequent tasks where startup time is not a concern.	Short queries, frequent submissions, and streaming tasks that need fast startup.

How do I choose a deploy queue when creating a Spark cluster?

Use a dedicated resource queue for the Spark cluster to avoid resource contention with other Serverless tasks. For testing purposes, you can use an existing queue and manually reduce the Worker node count.

What happens to tasks when a cluster stops?

When a cluster stops, any tasks that have been submitted but not yet finished fail with the message "Spark cluster stopped." You can view failure details in the task history.

What if the task concurrency limit is not enough?

The default task concurrency limit for a Spark cluster is 200. To increase it, submit a ticket to request custom cluster parameters and apply the following configuration:

master.SPARK_MASTER_OPTS   -Dspark.deploy.retainedApplications=200 -Dspark.deploy.retainedDrivers=200

How do I use a runtime environment in a Spark cluster?

Submit a ticket to request custom cluster parameters, then configure the runtime environment as follows:

When creating the cluster, add the following parameter in the cluster advanced configuration:
```
spark.standalone.pyenv.ids   <runtime-environment-id>
```
When submitting a task, select the runtime environment to use for that task.