Manage Spark configuration templates - E-MapReduce - Alibaba Cloud Documentation Center

Prerequisites

You have created a workspace. For more information, see Manage workspaces.

Create a configuration template

Navigate to the configuration management page.
1. Log on to the E-MapReduce console.
2. In the left navigation pane, choose EMR Serverless > Spark.
3. On the Spark page, click the name of the target workspace.
4. On the EMR Serverless Spark page, click Configurations in the left navigation pane.
On the Task Templates page, click Create Template.

On the Create Task Template page, configure the following parameters and click create.

Parameter	Description
Template Name	Enter a name for the template.
Engine Version	The engine version for the compute environment. For more information, see Engine versions.
Timeout Period	The maximum runtime for a job. Note When you submit a job, you can override the template's timeout setting by passing the `spark.emr.serverless.jr.timeout=-1` parameter to allow the job to run indefinitely.
spark.driver.cores	The number of CPU cores for the driver process.
spark.driver.memory	The amount of memory available to the driver process.
spark.executor.cores	The number of CPU cores to use for each executor.
spark.executor.memory	The amount of memory that each executor can use.
spark.executor.instances	The number of executors that Spark allocates.
Dynamic Resource Allocation	Disabled by default. When enabled, configure the following parameters: Minimum Number of Executors: The default value is 2. Maximum Number of Executors: If spark.executor.instances is not set, the default value is 10.
More Memory Configurations	spark.driver.memoryOverhead: The non-heap memory available for the driver. If this parameter is not set, Spark automatically allocates a value based on the default, which is `max(384 MB, 10% * spark.driver.memory)`. spark.executor.memoryOverhead: The non-heap memory available for each executor. If this parameter is not set, Spark automatically allocates a value based on the default, which is `max(384 MB, 10% * spark.executor.memory)`. spark.memory.offHeap.size: The amount of off-heap memory available to Spark. The default value is 1 GB. This parameter takes effect only when `spark.memory.offHeap.enabled` is set to `true`. When the Fusion engine is used, this feature is enabled by default with 1 GB of off-heap memory.
Spark Configuration	Custom Spark configuration properties. Use spaces to separate key-value pairs. Example: `spark.sql.catalog.paimon.metastore dlf`. EMR Serverless Spark provides several built-in, service-specific parameters. For details, see Custom Spark Conf parameters.

After you create a configuration template, you can modify or remove it by clicking Edit or Delete in the Actions column.

Use a configuration template

Data development

When you develop batch or streaming jobs in Load from template, you can select an existing configuration template from the Load from template dropdown list. The system automatically loads the parameters from the selected template.

CLI

When you submit a job by using spark-submit, you can specify a template ID with the spark.emr.serverless.templateId parameter in the --conf option. The system loads the template parameters and applies them as defaults for the Spark application.

Example:

spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master yarn \
  --conf spark.emr.serverless.templateId=<template_id> \
  /path/to/your/spark-job.jar

In the preceding command, <template_id> is the ID of the template. You can obtain the template ID on the Task Templates page.

Livy Gateway

When you create a Create Livy Gateway, you can select an existing configuration template from the Load from template dropdown list in the upper-right corner of the page. The system automatically loads the parameters from the selected template.

Kyuubi Gateway

When you create a Create Kyuubi Gateway, you can select an existing configuration template from the Load from template dropdown list in the upper-right corner of the page. The system automatically loads the parameters from the selected template.

Reset to system defaults

To reset a configuration template to the system defaults:

On the Task Templates tab, click Edit in the Actions column.
On the Edit Task Template page, click Reset to system defaults at the bottom of the page, and then click Save Changes.

Change the default template

To set a different configuration template as the default:

On the Task Templates page, click Set as default in the Actions column.
In the dialog box that appears, click OK.