E-MapReduce:Release notes for EMR Serverless Spark on April 15, 2025

Feature

Description

References

Workspace management

You can add tags when you create a workspace or add or modify tags on the Spark page after you create a workspace.

Manage workspaces

Data development

The variable management feature is provided for SQL and batch jobs to simplify code maintenance and adjustment, and improve development efficiency.

Hadoop commands can be used in a notebook to access Object Storage Service (OSS) or OSS-HDFS.

Run Hadoop commands in a notebook to perform OSS- or OSS-HDFS-related operations

EMR Serverless Spark is connected to CloudMonitor to send alerts when a streaming or batch job times out or fails.

Subscribe to system event notifications

The link for accessing the web UI of Spark UI is returned in the results of Spark SQL jobs. You can access the web UI of Spark by using different methods.

Access the web UI of Spark

A notebook session can be used by multiple notebooks.

Get started with notebook development

Session management

The jobs run by using a specific session can be viewed.

• Manage SQL sessions

• Manage notebook sessions

• Manage Spark Thrift Server sessions

File management

OSS buckets can be mounted as file systems to notebook sessions.

• Manage notebook sessions

• Manage files and file directories

Workflow

Custom variables can be configured for workflows and nodes to simplify code maintenance and adjustment, and improve development efficiency.

The data backfill feature is provided. Time variables used by workflows and workflow nodes can be automatically replaced with specific values based on the business time that you specify when you backfill data.

Manual run

Engine version

Description

esr-3.4.0 (Spark 3.4.4, Scala 2.12)

Spark 3.4.4 is available.

esr-2.6.0 (Spark 3.3.1, Scala 2.12)

esr-3.4.0 (Spark 3.4.4, Scala 2.12)

esr-4.2.0 (Spark 3.5.2, Scala 2.12)

• Fusion acceleration

  • The performance of user-defined functions (UDFs) is optimized.

  • The performance of operations, such as Sort, First/Last,and DenseRank, is optimized.

  • Partitioned tables are supported by CSV Reader.

  • Various time zone types are supported by the from_utc_timestamp function.

  • The spilling feature is optimized.

  • Parameters of the TIMESTAMP and VARCHAR types are supported by the format_datetime function.

  • Optimization is implemented in the transaction committing phase of a write operation to improve the write performance of tables.

  • The base64 and unbase64 functions are supported.

  • The array_union function is supported.

• Paimon

  • The table directory can be automatically deleted when you execute the DROP TABLE statement.

  • The compatibility issue that occurs during incremental data reading is fixed.

  • The three-part naming convention is supported.

  • Incremental data reading is supported after you change the number of buckets.

• Java Runtime

  • Concurrent data writes to static partitions are supported.

  • The committer configurations are optimized.

Celeborn

DYN is optimized to facilitate the asynchronous transformation of OpenStream, which improves the system performance and stability.