E-MapReduce:Release notes for EMR Serverless Spark on June 5,2025

Feature

Description

References

Workspace management

You can create subscription workspaces to reserve resources in advance. This feature is ideal for scenarios with clear budget planning.

Subscription

Gateway

Livy Gateway now supports setting default parameters for Spark tasks and core configuration parameters for Livy.

livy.conf

You can disable the Internet Endpoint for Livy Gateway.

Manage Livy Gateway

High availability (HA) Kyuubi Gateway is now supported.

Manage Kyuubi Gateways

You can connect to Kyuubi Gateway using the REST API.

Manage Kyuubi Gateways

Runtime environment

You can batch import Python libraries using a requirements.txt file.

Create a runtime environment

Domain name management

The domain name management feature is available to resolve DNS parsing issues in the cloud environment. This ensures that tasks can access external services or resources correctly and efficiently.

Manage domain names

Configuration templates

You can create multiple configuration templates. When you submit a task, specify a template to apply its configuration.

Configuration management

Configuration file management

You can save configuration information and configuration files. When a Spark task starts, the configuration files are automatically loaded into the working directory.

Configuration management

Ecosystem

• You can enable LDAP authentication in Kyuubi and Livy Gateway to enhance security.

• You can enable Ranger authentication in Kyuubi Gateway for fine-grained access control.

• Configure and enable LDAP authentication for Livy Gateway

• Configure and enable LDAP authentication for Kyuubi Gateway

• Configure and enable Ranger authentication for Kyuubi Gateway

Custom Spark Conf parameters

New parameters are added. Remove default libraries and specify the queue for Kyuubi to start applications.

List of custom Spark Conf parameters

Version number

Description

esr-2.7.0 (Spark 3.3.1, Scala 2.12)

esr-3.3.0 (Spark 3.4.4, Scala 2.12)

esr-4.3.0 (Spark 3.5.2, Scala 2.12)

• Fusion acceleration

  • Optimized the Sort operator.

  • Optimized the Window operator.

  • Optimized spill.

  • Optimized shuffle partition.

  • Added support for the Paimon Bucket Join feature.

  • Added support for reading compressed CSV files.

  • Optimized user-defined functions (UDFs).

  • Added support for the following functions: map_from_entries, to_json, posexplode_outer, approx_percentile, regexp_extract, map_from_arrays, timestamp_seconds, and map_filter.

  • Improved the accuracy of metrics data.

• Lakehouse formats

  • [Paimon] Fixed a NullPointerException (NPE) issue with Replace Tag.

  • [Paimon] Added support for federated query and write operations across multiple catalogs.

  • [Hudi] Upgraded Hudi to version 0.15 in esr-2.7.0.

  • Added support for using Paimon and Hudi procedures at the same time.

• Spark framework

  • Tasks now fail automatically during critical garbage collection (GC) to prevent them from getting stuck.

  • Optimized the use of multiple catalogs.

Celeborn

• Added support for LocalShuffleReader.

• Optimized the performance of skew reads.

• Dynamically splits data based on worker payload to improve resource utilization.

• Optimized the quota management mechanism.

• Optimized high availability (HA) across zones.