E-MapReduce:Release notes for EMR V3.39.X

Version

Date

EMR V3.39.2

March 25, 2022

EMR V3.39.1

February 15, 2022

Service

Description

Flink

• The application performance management (APM) dashboard is optimized. Metrics such as sourceIdleTime are added.

• Alerting by CloudMonitor is supported.

Kafka

• SSL and Simple Authentication and Security Layer (SASL) configurations are supported.

• The default values of some parameters are changed.

ClickHouse

The default values of some parameters are changed.

Service

Description

SmartData

The service is no longer used.

Bigboot

RSS

• EMR Remote Shuffle Service (ESS) is upgraded to Remote Shuffle Service (RSS). For more information, see RSS.

• The service functionality and stability are enhanced.

JindoSDK

• The architecture of SmartData is upgraded to JindoData.

• EMR is integrated with JindoSDK for JindoData 4.0.0 for the first time. JindoData connects to Alibaba Cloud Object Storage Service (OSS) and the Alibaba Cloud OSS-HDFS service.

Spark

• Hive on Spark is optimized.

• Spark is adapted to JindoSDK.

Tez

Tez is adapted to JindoSDK.

Hive

Hive is adapted to JindoSDK.

Presto

• User-defined functions (UDFs) can be dynamically loaded.

• The for ... as of syntax can be used in the time travel feature to query data in a Delta Lake table.

• An independent Delta Lake catalog is added. Presto provides default configurations for a Delta connector and supports the Optimize, Z-ordering, and Data Skipping features based on the independent Delta Lake catalog.

• The issue that data in Merge on Read tables of Hudi cannot be queried by using a Hudi connector is fixed. You cannot use a Hive connector to query Merge on Read tables of Hudi.

• Presto is adapted to JindoSDK.

Delta Lake

• Metadata management

  • The built-in catalog of Spark, instead of an API operation that is called by using the Hive CLI, is used to synchronize metadata and partition information.

  • The statistics on table data are automatically reported to metastores.

• SQL

  • The syntax of the time travel feature is supported.

  • The DROP PARTITION SQL syntax is supported.

  • The ADD COLUMN statement can be used to add columns to specified locations (FIRST and AFTER).

• Enhanced table management capabilities

  • The file size can be dynamically adjusted based on the table size. By default, this feature is enabled.

  • The auto-vacuum feature is supported and enabled by default. Concurrent vacuum operations are supported.

  • The logic of automatic compaction is optimized. By default, the automatic compaction feature is disabled.

  • The Z-ordering syntax is added. Z-ordering-based data processing is accelerated.

Hudi

Hudi is updated to 0.10.0.

HDFS

HDFS is adapted to JindoSDK.

YARN

YARN is adapted to JindoSDK.

Flume

Flume is adapted to JindoSDK.

Flink

• By default, the lib directory of Flink is uploaded to your HDFS cluster. This way, you can configure the yarn.provided.lib.dirs parameter to use the directory.

• Flink is adapted to JindoSDK.

Impala

Impala is adapted to JindoSDK.

Ranger

• The issue that Spark History Server fails to start is fixed.

• Ranger is adapted to JindoSDK.

HBase

• The issue about default parameter settings is fixed.

• The issue about the date format of garbage collection (GC) logs is fixed.

• The restart issue that occurs if an IP address is configured for RegionServer is fixed.

Druid

Druid is adapted to JindoSDK.

ClickHouse

The logic of processing data when the ClickHouse component stops working is optimized.

Iceberg

• Iceberg is updated to 0.13.0.

• Default configuration items are hidden to improve user experience.

DLF-Auth

The issue that Spark History Server fails to start is fixed.

StarRocks

The service is added in the new EMR console.

The 2.0.1 version of this service is released.