E-MapReduce:Release notes for EMR Serverless Spark on November 12,2025

Feature

Description

References

AI Center (Beta)

Added support for AI Function. This feature provides out-of-the-box capabilities for the Qwen large model, which lets you efficiently call the large model to process massive data.

Added a model service management feature. You can register external model services and call them from Serverless Spark.

Data Development

• Streaming jobs now support log rotation.

• For streaming jobs, you can now configure automatic retry policies and retry intervals for failed jobs.

Develop batch or streaming jobs

The SparkSQL job editor now displays run records and execution results from the last three days.

Develop with SparkSQL

Data catalog

A single workspace now supports adding and using HMS, DLF 1.0, and DLF (formerly DLF 2.5) data catalogs at the same time.

Manage data catalogs

Resource Management

Queues now support a hybrid billing mode that allocates both pay-as-you-go and subscription quotas. This provides a billing policy that combines the flexibility of pay-as-you-go with the cost-effectiveness of subscriptions.

Manage resource queues

You can now decrease subscription quotas.

Manage workspaces

Resource observation now supports queries with daily granularity. You can view resource usage trends for the last 7 and 30 days.

Resource observation

Gateway

Kyuubi Application now supports viewing job overviews and exploring logs.

Manage Kyuubi Gateway

Livy Gateway now supports limiting the number of sessions a single user can create.

Livy Gateway configuration examples

Configuration management

• Spark configuration templates now include a timeout configuration item.

• Kyuubi Gateway and Livy Gateway now support loading configurations from Spark configuration templates.

Manage Spark configuration templates

Best practices

Use Serverless Spark to perform text deduplication with MinHash-LSH.

Large-scale text deduplication based on MinHash-LSH

You can now register and use Python UDFs in SparkSQL.

Use UDFs

Version number

Description

Engine esr-5.0.0 (Spark 4.0.1, Scala 2.13)

Engine esr-4.6.0 (Spark 3.5.2, Scala 2.12)

Engine esr-3.5.0 (Spark 3.4.4, Scala 2.12)

Engine esr-2.9.0 (Spark 3.3.1, Scala 2.12)

• Fusion acceleration

  • Supports shiftrightunsigned.

  • str_to_map supports last_win.

  • Parquet write optimization.

  • Commit optimization.

  • JSON Datasource optimization.

  • Sort operator optimization.

• Lakehouse formats

  • [DLF] Optimized table reads and writes.

  • [DLF] Supports password-free access to pvfs.

  • [DLF] Supports the Lance file format.

  • [Paimon] Supports password-free access for Parquet.

  • [Paimon] Supports row-level lineage.

  • [Paimon] Optimized merge into.

  • [Paimon] Optimized compaction.

• Spark framework

  • Supports Spark 4.0.

  • Supports Python UDFs.

  • MC Connector supports max_pt and map_agg.

  • Supports Fast Fail.

  • Improved compatibility with Hive.

  • Provides the distcp feature.

• DataWorks

  • Supports RDD lineage.

• DuckDB

  • Supports reading from and writing to OSS.

Celeborn

• Stability optimization

  • Optimized the Shuffle Read retry mechanism.

  • Optimized Shuffle resource allocation.