E-MapReduce:Release notes for EMR Serverless Spark on September 17,2025

Feature

Description

References

Ciphertext management

Added support for workspace-level ciphertext management. This helps avoid security risks from plaintext information such as AccessKeys and passwords.

Manage sensitive information using ciphertext

Runtime environment management

Added support for configuring runtime environments based on a specified Python version.

Manage runtime environments

Managed file directory

Added a feature to mount managed file directories in data development batch jobs. After mounting, you can read and write files in the managed file directory within the batch job.

Managed file directory

Job history

Added support for filtering jobs by resource configuration, Spark configuration, and runtime parameters.

Kyuubi Gateway

• You can now create multiple Kyuubi Gateway instances in a single workspace.

• When associated with a DLF 2.5 Catalog, token creation supports user authorization. It also performs authentication and authorization for both Kyuubi and DLF 2.5.

• Manage Kyuubi Gateways

• Manage data catalogs

Livy Gateway

You can now configure and enable Ranger authorization.

Configure and enable Ranger authorization for Livy Gateway

Notebook

Added support for the R language.

Version number

Description

esr-2.7.1 (Spark 3.3.1, Scala 2.12)

esr-2.8.0 (Spark 3.3.1, Scala 2.12)

esr-3.3.1 (Spark 3.4.4, Scala 2.12)

esr-3.4.0 (Spark 3.4.4, Scala 2.12)

esr-4.3.1 (Spark 3.5.2, Scala 2.12)

esr-4.4.0 (Spark 3.5.2, Scala 2.12)

esr-4.5.0 (Spark 3.5.2, Scala 2.12)

• Fusion acceleration

  • Sort operator optimization.

  • Spill optimization.

  • JSON Datasource optimization.

  • Project merge optimization.

  • Optimization for text deduplication scenarios in large models.

  • Spark SQL now supports the OSS restore syntax.

  • Upgraded Jindo SDK to 6.10.0.

  • Fine-grained UDF fallback optimization.

  • Optimized regexp_replace.

  • Integrated DuckDB into Notebook.

• Lakehouse formats

  • [DLF 2.5] Added full support for DLF 2.5 PaimonCatalog and IcebergCatalog.

  • [Paimon] Optimized read and write operations.

  • [Paimon] Added support for federated queries and writes across multiple catalogs.

  • [Paimon] Optimized the file open cost configuration.

  • [Iceberg] Added support for multiple catalogs.

  • [Lance] Added support for DLF Lance Table.

• Spark framework

  • Upgraded esr-4.5.0 to Java 17.

  • Optimized insert overwrite for non-partitioned tables.

  • Improved compatibility between Spark and Hive.

  • Optimized the MC connector.

  • Jobs now fail automatically after consecutive executor failures.

Celeborn

• Shuffle write skew optimization.

• Stability optimization.

• Performance optimization.