E-MapReduce:Custom SparkConf parameters

Parameter

Description

Scenario

spark.emr.serverless.user.defined.jars

Adds uploaded JAR packages to the ClassPath of the Serverless Spark driver and executors.

• Supported engine versions:

  • esr-4.x: esr-4.1.0 and later

  • esr-3.x: esr-3.1.0 and later

  • esr-2.x: esr-2.5.0 and later

• Upload methods:

  • For esr-4.2.0, esr-3.2.0, esr-2.6.0, and later versions:

    • You can upload files to File Management in Serverless Spark. For more information, see Manage files.

      On the Managed File Directory tab, click Copy Address in the Actions column for an object file to copy its address.

    • Upload to Alibaba Cloud Object Storage Service (OSS). For more information, see Simple upload.

      The path format is oss://path/to/file1.jar,oss://path/to/file2.jar.

  • For versions earlier than esr-4.2.0, esr-3.2.0, or esr-2.6.0, you can only upload packages to OSS. For more information, see Simple upload.

    The path format is oss://path/to/file1.jar,oss://path/to/file2.jar.

Use this parameter to add custom JAR packages from OSS to the Spark driver and executors when you submit Spark tasks using the Spark-Submit tool, batch jobs, or Airflow Serverless Spark Operator, or when you create session resources.

spark.emr.serverless.fusion

Specifies whether to enable Fusion for sessions or batch processing tasks started by Kyuubi and Livy. Valid values:

• false (default): shutdown

• true: Indicates an enabled state.

You can use the Spark Configuration parameter in a task or session to enable Fusion.

spark.emr.serverless.environmentId

Specifies the ID of the runtime environment to use for computing resources.

Use this parameter to specify a runtime environment when you submit Serverless Spark tasks using Airflow or the Spark-Submit tool. By default, third-party dependency libraries are installed in the runtime environment.

spark.emr.serverless.network.service.name

Specifies the name of the network connection to enable network connectivity between computing resources and data sources in other VPCs.

Use this parameter to add a network connection when you submit a Serverless Spark task, allowing access to data sources in other Virtual Private Clouds (VPCs).

spark.emr.serverless.excludedModules

Removes built-in libraries from Serverless Spark.

• For esr-2.7.0, esr-3.3.0, esr-4.3.0, and later versions, you can remove the following libraries: paimon,hudi,iceberg,delta,celeborn,dlf,fusion,jindo,odps,mysql,doctor.

• For esr-2.8.0, esr-3.4.0, esr-4.4.0, and later versions, you can also remove kafka.

This parameter is typically used when you need to use custom JAR packages. It lets you remove built-in Serverless Spark libraries when you submit Spark tasks from the Serverless Spark console, the Spark-Submit tool, batch jobs, Airflow Serverless Spark Operator, Kyuubi, or Livy, or when you create session resources.

spark.emr.serverless.kyuubi.engine.queue

Specifies the name of the workspace queue where the Spark application started by Kyuubi will run.

This parameter can be set in the Kyuubi configuration section or specified in the JDBC URL when you establish a connection.

spark.emr.serverless.templateId

Specifies the ID of the default configuration template for the Spark application. By referencing a predefined workspace template, you can simplify parameter configuration when you submit a task.

You can obtain the template ID on the Operation Center > Configuration Management > Spark Configuration Templates page. For example, TPL-2b3859f8c0c8439faddc22f223c8****.

This parameter is supported only by the Spark-Submit tool.

spark.emr.serverless.livy.config.mode

Controls whether to use the settings from the spark-defaults.conf file of the Livy Gateway when you submit a Spark task.

• If this parameter is not set (default behavior), the settings from the spark-defaults.conf file are automatically loaded and applied to the task.

• Set to ignore

  If you set this parameter to ignore, the configuration information in the spark-defaults.conf file is ignored when you submit a Spark job. In this case, the job uses only the configuration parameters that you explicitly specify or the default Spark configuration.

Set this parameter to ignore to fully customize the configuration of a Spark task. If you want to retain the default configuration behavior of the Livy Gateway, do not set this parameter.

spark.emr.serverless.tag.xxxx

You can add tags to a batch job submitted through Livy in the spark.emr.serverless.tag.<key> <value> format.

Use this parameter to add tags to Spark jobs submitted through the Livy Gateway. You can then filter jobs by these tags in the job history.