E-MapReduce:Release notes for EMR V5.2.X

Service

Description

SmartData

SmartData is updated to 3.6.1. For more information, see SmartData 3.6.X.

Hive

• The issue that the output of the show create table command based on Data Lake Formation (DLF) metadata is inaccurate is fixed.

• The default parameters of Hive are optimized to improve the performance of Hive jobs.

• In the EMR console, the parameter names on the hive-env tab of the Configure tab for the Hive service are changed to uppercase. This facilitates the use of the parameters.

• The issue that user-defined functions (UDFs) cause HiveServer2 memory leak is fixed.

• The error message that is reported because of the incompatibility between the file system and Hive metastore when you write data to a Hive table is optimized.

HDFS

The data compression algorithm Zstandard is supported.

Delta Lake

• Delta Lake is updated to 0.8.0.

• Spark 3 is supported.

Flink

Flink is updated to 1.12-vvr-3.0.2.

Hudi

• Hudi is updated to 0.8.0.

• Hudi can be integrated with Spark SQL.

Spark

• Delta Lake and Hudi are supported.

• Remote Shuffle Service is supported.

• Livy is supported.

• In the EMR console, the parameter names on the spark-defaults tab of the Configure tab for the Spark service are optimized.

• The cost-based optimization (CBO), dynamic partition pruning, and Z-order features are optimized. The performance of these features is 50% higher than in Spark 3.

• Log Service, DataHub, and Message Queue for Apache RocketMQ can be used as data sources.

Tez

The default parameters of Tez are optimized to improve the performance of Tez jobs.

Ranger

• The warning error contained in logs about starting Spark in Ranger is fixed.

• The issue that user information fails to be automatically synchronized after Ranger is connected to a Lightweight Directory Access Protocol (LDAP) server is fixed.

Knox

• Knox is adapted to Kudu.

• Knox is adapted to HBase.

Kafka

• The Cruise Control component can be used to provide the balance feature for Kafka clusters.

• Disks for Kafka clusters are hot-swappable. You can replace a damaged disk without the need to stop the Kafka broker of your cluster.

• The default values of some parameters are changed.

Phoenix

The issue that no JDBC driver is found when Hive or Spark SQL is used to access Phoenix tables is fixed.

EMR Remote Shuffle Service (ESS)

Spark 3 is supported.