E-MapReduce:Release notes for EMR Serverless Spark on January 20, 2025

Feature

Description

Sales

Computing resource plans are provided for pay-as-you-go Spark workspaces. This helps reduce costs.

Stability

• The resource application efficiency and running stability of jobs are improved.

• Cross-zone high availability is supported.

Integration with other ecosystems

• Batch and streaming jobs can be used to connect to the external Ranger service.

• Kyuubi gateways that are compatible with open source Kyuubi can be used to submit SQL jobs.

Data catalog

• RAM users can be used to access Data Lake Formation (DLF).

• Access to Hive catalogs created in the DLF 2.0 console is supported.

Resource observation

Monitoring of resource consumption from the workspace and queue dimensions is supported.

Runtime environment

The --conf spark.emr.serverless.environmentId=<Runtime environment id> parameter can be used in Spark Submit to specify a runtime environment.

Engine version

Description

esr-4.0.0 (Spark 3.5.2, Scala 2.12)

esr-3.1.0 (Spark 3.4.3, Scala 2.12)

esr-2.5.0 (Spark 3.3.1, Scala 2.12)

• Spark 3.5.2 is supported.

• Fusion acceleration

  • CacheTable is optimized.

  • Tables in the CSV and TEXT formats can be read.

  • Data can be read from and written to files in the complex ORC format.

  • Tables in the Hudi format can be read.

  • The parse_url function is supported.

  • The concat_ws function is supported.

  • Window operators are optimized.

  • Sort operators are optimized.

• Java Runtime

  • The performance issue caused by downloading JAR files from a driver is fixed to improve the concurrent processing capability of executors.

  • The isolation mechanism for custom JAR packages is enhanced.

  • Empty data can be inserted into an external table.

  • The configurations of memory committers are optimized.

• Paimon

  • Custom Paimon data paths are supported.

  • Views can be created and used.

  • The CREATE TABLE ... WITH LOCATION syntax is supported.

  • The performance of SHOW TABLES is improved.

Celeborn

Multi-zone high availability is supported.