Release notes for EMR Serverless Spark on September 14, 2024 - E-MapReduce

This topic describes the release notes for E-MapReduce (EMR) Serverless Spark on September 14, 2024.

Overview

On September 14, 2024, the latest version of EMR Serverless Spark is released, featuring platform improvements, ecosystem integration, improved performance, and enhanced engine capabilities.

Platform updates

Feature	Description
Workspace management	The maximum quota for a Spark workspace can be modified. A workspace can be created or deleted by using a RAM role. The status transition process for a workspace is optimized, and error messages can be returned when errors occur.
Runtime environment management	A runtime environment can be created based on your business requirements. Required libraries can be added to an environment. You can use runtime environments in notebook sessions. When a notebook session is started, the system pre-installs the related libraries based on the selected environment. For more information, see Manage runtime environments.

Engine updates

Engine version	Description
esr-2.2 (Spark 3.3.1, Scala 2.12)	Fusion acceleration: The WindowTopK operator is supported. The shuffle performance is optimized. The issue that task deserialization takes a long time due to scale-in is fixed. Automatic fallback for unsupported Paimon operators are supported. CU consumption information can be displayed in the driver logs. Java Runtime JSON data parsing is accelerated based on the single instruction multiple data (SIMD) instructions.
esr-2.3 (Spark 3.4.2, Scala 2.12) Alpha	Paimon catalogs are supported in Data Lake Formation (DLF) 2.0.