E-MapReduce:Release notes for EMR V5.4.X

Version

Date

EMR V5.4.3

December 2021

EMR V5.4.2

December 2021

EMR V5.4.1

November 2021

EMR V5.4.0

October 2021

Service

Description

SmartData

• SmartData is updated to 3.8.0. For more information, see SmartData 3.8.X overview.

• Authentication and authorization based on Kerberos and Ranger can be used to manage permissions on data in OSS.

Service

Description

SmartData

SmartData is updated to 3.7.3. For more information, see SmartData 3.7.X overview.

Oozie

The issue that Jetty Server of Oozie fails to start due to JAR package conflicts in high availability (HA) scenarios is fixed.

Impala

The issue that the no such method error message appears when you query data in DLF metadata tables is fixed.

DLF-Auth

DLF-Auth is updated to 1.0.1.

Service

Description

SmartData

SmartData is updated to 3.7.2. For more information, see SmartData is updated to 3.7.2. For more information, see SmartData 3.7.X overview.

Spark

• Spark is updated to 3.1.2.

• In Spark 3.x, the Distinct computing performance is optimized for Spark SQL. The optimization feature is triggered if an aggregation operator contains multiple count(distinct case ... when ...) methods.

• The array-index out of bounds error that is returned when some required statistics for Adaptive Query Execution (AQE) are missing is fixed.

• Errors related to AQE and data caching in specific scenarios are fixed.

Hive

In JindoFS in block storage mode, the metadata of multiple Hive tables can be optimized at the same time. By default, this feature is disabled.

Presto

Storage handlers can be used to query data in Delta tables.

Delta Lake

• Delta Lake is updated to 1.0.0.

• The same Delta Lake connectors are used in Hive 2 and Hive 3.

• The error that is returned when you use Delta Lake connectors to query data from multi-level partitioned tables is fixed.

• The SQL syntax that supports various features, such as Data Skipping, Optimize, and Z-ordering, can be used.

• Metadata can be synchronized to a metastore.

Hudi

• Hudi is updated to 0.9.0.

• The issue about the compatibility of sql.extension between Delta Lake and Hudi is fixed.

HDFS

By default, the reserved space of NameNode adaptively increases. This way, NameNode enters the Safe mode in a timely manner when the disk space is insufficient.

Storm

The service is no longer used.

Zeppelin

Zeppelin is updated to 0.10.0.

Hue

• The issue that YARN Job Browser sometimes cannot present or terminate jobs is fixed.

• YARN Job Browser is accessible by default.

• The Presto protocol is supported by default.

Druid

ClickHouse

• Some default configurations are updated.

• Clusters can be scaled out.

• The MetaChecker feature is supported.

• Object Storage Service (OSS) table engines and OSS table functions can be used to read data.

Iceberg

• Iceberg is updated to 0.12.0-1.0.1.

• Errors related to the Hive runtime dependency are fixed.

Knox

DLF-Auth