E-MapReduce:Custom Spark parameters

Parameter

Description

Scenario

spark.emr.serverless.user.defined.jars

Adds uploaded JAR packages to the classpath of the Serverless Spark driver and executors.

• Supported engine versions:

  • esr-4.x: esr-4.1.0 and later.

  • esr-3.x: esr-3.1.0 and later.

  • esr-2.x: esr-2.5.0 and later.

• Upload methods:

  • For esr-4.2.0, esr-3.2.0, esr-2.6.0, and later versions:

    • Upload to Files in Serverless Spark. For details, see Manage files.

      On the Managed File Directory tab, click Copy Address in the Actions column for the target file to get its path.

    • Upload to OSS. For instructions, see simple upload.

      The path format is oss://path/to/file1.jar,oss://path/to/file2.jar.

  • For versions earlier than esr-4.2.0, esr-3.2.0, or esr-2.6.0: These versions only support uploads to OSS. For instructions, see simple upload.

    The path format is oss://path/to/file1.jar,oss://path/to/file2.jar.

Use this property to add custom JAR packages from OSS to the Spark driver and executors when you submit a Spark job by using spark-submit, a batch job, or the Airflow Serverless Spark Operator, or when you create a session.

spark.emr.serverless.fusion

Specifies whether to enable Fusion for a session or batch job launched by Kyuubi or Livy. Valid values:

• false (Default): Disables Fusion.

• true: Enables Fusion.

Set this property in the Spark Configuration for a job or session to enable or disable Fusion.

spark.emr.serverless.environmentId

Specifies the ID of the runtime environment for the computing resources.

When submitting Serverless Spark jobs by using Airflow or spark-submit, use this property to specify a runtime environment that has third-party libraries pre-installed.

spark.emr.serverless.network.service.name

Specifies the name of a network connection, which allows computing resources to communicate with data sources in other VPCs.

When submitting a Serverless Spark job, use a network connection to access data sources in other Virtual Private Cloud (VPC) networks.

spark.emr.serverless.excludedModules

Removes built-in libraries from Serverless Spark.

• For esr-2.7.0, esr-3.3.0, esr-4.3.0, and later versions, you can remove the following modules: paimon,hudi,iceberg,delta,Celeborn,dlf,fusion,jindo,odps,mysql,doctor.

• For esr-2.8.0, esr-3.4.0, esr-4.4.0, and later versions, you can also remove the kafka module.

To avoid conflicts with custom JAR packages, remove the built-in libraries when you submit jobs by using the Serverless Spark console, spark-submit, a batch job, the Airflow Serverless Spark Operator, Kyuubi, or Livy, or when creating a session.

spark.emr.serverless.kyuubi.engine.queue

Specifies the name of the workspace queue where Spark applications launched by Kyuubi run.

You can set this in the Kyuubi configuration panel or specify it in the JDBC connection URL.

spark.emr.serverless.jr.timeout

Sets the maximum runtime for a job in seconds. The system automatically terminates the job if it exceeds this duration. The default value is empty, indicating no timeout limit.

The value must be an integer from -1 to 2147483647. A value of -1 or 0 means no timeout is set.

Use this property to set a job timeout when submitting jobs by using the Serverless Spark console, spark-submit, a batch job, or the Airflow Serverless Spark Operator.

spark.emr.serverless.fusion.enabled

Specifies whether to enable the Fusion data acceleration engine for a Spark application. Valid values:

• false (Default): Disables Fusion.

• true: Enables Fusion.

Use this property to enable or disable Fusion acceleration when submitting jobs by using the Serverless Spark console, spark-submit, a batch job, or the Airflow Serverless Spark Operator.

spark.emr.serverless.mount.nas.enabled

Specifies whether to mount a NAS directory to the Spark driver. If enabled, you must also specify the directory to mount by using the spark.emr.serverless.mount.nas.volume property.

• Supported engine versions:

  • esr-4.x: esr-4.7.0 and later.

  • esr-3.x: esr-3.6.0 and later.

• Valid values:

  • false (Default): Disables the mount.

  • true: Enables the mount.

Use this property to mount a managed NAS file directory to the Spark driver when submitting a job by using the Serverless Spark console, spark-submit, a batch job, or the Airflow Serverless Spark Operator. When mounted, the driver can read from and write to the NAS directory.

spark.emr.serverless.mount.nas.volume

Specifies the ID of the managed NAS directory to mount.

Supported engine versions:

• esr-4.x: esr-4.7.0 and later.

• esr-3.x: esr-3.6.0 and later.

Use this property to mount the specified managed NAS file directory when submitting a job by using the Serverless Spark console, spark-submit, a batch job, or the Airflow Serverless Spark Operator.

spark.emr.serverless.mount.nas.executor

Specifies whether to mount the NAS directory to all Spark executors.

• Supported engine versions:

  • esr-4.x: esr-4.7.0 and later.

  • esr-3.x: esr-3.6.0 and later.

• Valid values:

  • false (Default): Disables the mount.

  • true: Enables the mount.

Use this property to mount a managed NAS file directory to all Spark executors when submitting a job by using the Serverless Spark console, spark-submit, a batch job, or the Airflow Serverless Spark Operator. When mounted, executors can read from and write to the NAS directory.

spark.emr.serverless.mount.oss.enabled

Specifies whether to mount an OSS directory to the Spark driver.

If enabled, you must also specify the directory to mount by using the spark.emr.serverless.mount.oss.volume property. Valid values:

• false (Default): Disables the mount.

• true: Enables the mount.

Use this property to mount a managed OSS file directory to the Spark driver when submitting a job by using the Serverless Spark console, spark-submit, a batch job, or the Airflow Serverless Spark Operator. When mounted, the driver can read from and write to the OSS directory.

spark.emr.serverless.mount.oss.volume

Specifies the ID of the managed OSS directory to mount.

Use this property to mount the specified managed OSS file directory when submitting a job by using the Serverless Spark console, spark-submit, a batch job, or the Airflow Serverless Spark Operator.

spark.emr.serverless.mount.oss.executor

Specifies whether to mount an OSS directory to all Spark executors. Valid values:

• false (Default): Disables the mount.

• true: Enables the mount.

Use this property to mount a managed OSS file directory to all Spark executors when submitting a job by using the Serverless Spark console, spark-submit, a batch job, or the Airflow Serverless Spark Operator. When mounted, executors can read from and write to the OSS directory.

spark.emr.serverless.templateId

Specifies the ID of the default configuration template for a Spark application. Referencing a predefined workspace template simplifies parameter configuration during job submission.

You can find the template ID on the Operation Center > Configurations > Spark Configuration Template page. For example: TPL-2b3859f8c0c8439faddc22f223c8****.

This property is supported only when using spark-submit.

spark.emr.serverless.livy.config.mode

Controls whether to use the settings from the spark-defaults.conf file configured in Livy Gateway when you submit a Spark job.

• If this parameter is not set (default), Livy Gateway automatically loads and applies the settings from the spark-defaults.conf file to the job.

• Set to ignore

  If you set this parameter to ignore, the configurations in the spark-defaults.conf file are ignored when you submit a Spark job. In this case, the job uses only the configuration parameters that you explicitly specify or the default Spark configurations.

When you need to fully customize the configuration of a Spark job, you can set this parameter to ignore. If you want to retain the default configuration behavior of Livy Gateway, you do not need to set this parameter.

spark.emr.serverless.tag.xxxx

Adds a tag to a batch job submitted through Livy, using the format spark.emr.serverless.tag.<key> <value>.

You can then filter jobs by tag in the job history.