E-MapReduce:Release notes for EMR 5.x series

Service

Change

JindoCache

Upgraded to version 6.5.3.

StarRocks2

Upgraded to version 2.5.22.

StarRocks3

Upgraded to version 3.2.11.

Service

Change

JindoSDK

Upgrades JindoSDK to resolve a deadlock issue.

Service

Change

JindoCache

• Upgraded to version 6.5.1.

• Improves the read and write performance of Distributed Hash Table (DHT).

Spark

• Fixes an issue where partition folders cannot be deleted.

• Fixes a Hive package dependency issue to ensure that client operations do not interrupt the connection to metaStoreClient.

Trino

• Fixes an issue where some modified configurations might be unexpectedly restored during scale-out.

• Supports querying data on high-security OSS-HDFS.

• Fixes a service abnormality issue that occurs after DLF-AUTH is enabled.

Presto

Supports querying data on high-security OSS-HDFS.

HDFS

Fixes an issue where the memory of NameNode and DataNode cannot be modified.

HBaseHDFS

YARN

• ResourceManager supports sending timeline events in batches to improve processing capabilities.

• Fixes a logic issue in container and resource processing in ResourceManager.

Zookeeper

• Fixes an issue where the memory configuration of a node group cannot be modified.

• Supports refactoring the log configuration file.

Impala

Fixes an issue where customer configurations are modified during elastic scaling.

Ranger

Supports the new JindoSDK kernel to effectively reduce CPU utilization.

Knox

Fixes an issue where component URL access fails when there is only one Master-Extend node.

Kafka

Fixes a startup issue with Kafka Connect clusters.

StarRocks

Fixes an issue where new BE nodes are not visible after a scale-out.

Doris

Upgraded to version 2.1.4.

Paimon

Upgraded to version 0.9-ali-7.

EMRHOOK

Supports parsing data lineage for MaxCompute tables.

Service

Change

Spark

Supports deploying Master-Extend node groups.

Hive

Kyuubi

Paimon

Replaces the Flink dependency from the VVR version with the community version and supports DLF Catalog.

Knox

Uses JDK 8 for packaging.

Flink

Restores the DLF configurations and dependencies that were removed in EMR-5.17.0.

Service

Change

Spark

Spark3 is upgraded to version 3.4.2.

Celeborn

Upgraded to version 0.4.0.

Doris

Upgraded to version 2.1.0.

StarRocks

• StarRocks2 is upgraded to version 2.5.18.

• StarRocks3 is upgraded to version 3.2.4.

DeltaLake

Upgraded to version 3.0.0.

Iceberg

Upgraded to version 1.5.0.

Zookeeper

Upgraded to version 3.8.4.

JindoCache

Upgraded to version 6.2.5.

Flink

Upgraded to version 1.17.2.

Service

Change

Spark

• Spark History Server supports using OSS-HDFS for storage by default.

• The Spark 3 native engine supports using OSS and OSS-HDFS for storage.

Hive

Hive warehouse supports using OSS-HDFS for storage by default.

OSS-HDFS

Adds the service.

YARN

Supports using OSS-HDFS for storage by default.

HBase

• HBase HFile data supports using OSS-HDFS for storage by default.

• HBase WAL logs support using OSS-HDFS for storage.

Service

Change

Kyuubi

Upgraded to version 1.7.1.

Celeborn

Upgraded to version 0.2.2.

Paimon

• Flink-Table-Store is renamed to Paimon.

• Upgraded to version 0.4-ali-1.

StarRocks

Upgraded to version 2.5.5.

Doris

Upgraded to version 1.2.4.

ClickHouse

Upgraded to version 23.3.2.37.

Trino

Provides a simple event listener by default to obtain audit logs.

Phoenix

Supports Hive on Phoenix.

Service

Change

ClickHouse

Upgraded to version 22.8.14.53.

Trino

Adds the odps.properties connector to support queries on MaxCompute.

JindoData

Upgraded to version 4.6.5.

JindoSDK

Upgraded to version 4.6.5.

Flink-Table-Store

Upgraded to version 0.3-ali-2.

YARN

Supports Node Labels management.

Service

Change

Iceberg

Upgraded to version 1.1.0.

Hudi

• Upgraded to version 0.12.2.

• Supports CDC.

DeltaLake

• Upgraded to version 2.2.0.

• Supports recording Vacuum operations in the transaction log.

Kudu

Upgraded to version 1.16.0.

Clickhouse

The ZooKeeper service must be selected when you install the ClickHouse service.

Celeborn

• RSS is renamed to Celeborn.

• The version of Celeborn is 0.2.0.

Presto

Adds the service. The kernel is community Facebook PrestoDB 0.278.3. The default HTTP port is 8889, and the default HTTPS port is 7779.

StarRocks

Upgraded to version 2.5.1.

Doris

Upgraded to version 1.2.1.

Kafka-Manager

Upgraded to version 3.0.0.6.

Impala

Upgraded to version 4.2.0.

OpenLDAP

Upgraded to version 2.4.46.

HBase

• Supports JDK 11.

• Supports ThriftServer2.

• The default value of the hbase.block.data.cachecompressed parameter is changed to true.

Flink-Table-Store

Adds the service. The version is based on community version 0.3.

JindoData

Upgraded to version 4.6.4.

Service

Change

Kerberos

Supports connecting to an external KDC on EMR.

Kafka

Adds a configuration item for startup commands that allows users to customize the startup parameters for the service.

JindoData

• Upgraded to version 4.6.0.

• Supports rewriting OSS-HDFS access paths.

Flink

Upgraded to version 1.13_vvr_4.0.15.

RSS

Upgraded to version 0.1.4.

Service

Change

Spark

• Upgraded to version 3.3.

• Supports enabling Kerberos authentication.

Hudi

• Upgraded to version 0.12.0.

• Supports Spark 3.3.

• Supports using a cloud-based MetaStore to host metadata and enabling the acceleration feature. For more information, see Instructions on how to use Hudi MetaStore.

Flink

• Supports enabling Kerberos authentication.

• Supports automatic connection with Data Lake Formation (DLF).

Iceberg

• Upgraded to version 0.14.0.

• Supports Spark 3.3.

• Supports enabling Kerberos authentication.

JindoData

• Upgraded to version 4.5.1.

• Supports AccessKey-free access to Alibaba Cloud resources.

Hadoop-Common and HDFS

• Supports enabling Kerberos authentication.

• Fixes security vulnerability CVE-2022-25168.

Knox

Integrates with Ranger. You can access the Ranger UI from the Access Links And Ports tab.

HBase

• Upgraded to version 2.4.9.

• Supports enabling Kerberos authentication.

• Supports group configuration.

RSS

• Upgraded to version 0.1.2.

• Supports enabling Kerberos authentication.

Doris

• Upgraded to version 1.1.2.

• Supports enabling Kerberos authentication.

StarRocks

• Upgraded to version 2.3.2.

• Supports enabling Kerberos authentication.

Kafka

• Upgraded to version 2.13_3.2.1.

• Supports enabling Kerberos authentication.

DeltaLake

• Supports upgrading to version 2.1.0.

• Supports Spark 3.3.

• Supports enabling Kerberos authentication.

Impala

• Supports creating views in DLF.

• Supports enabling Kerberos authentication.

Kudu

Adds the component. The version is 1.14.0.

YARN, Ranger, Hive, Kyuubi, Tez, Zookeeper, DLF-Auth, Phoenix, Sqoop, and Presto

Support enabling Kerberos authentication.

Service

Change

Clickhouse

Modifies the default values of some parameters.

StarRocks

Upgraded to version 2.1.1.

Service

Change

SmartData

The component is offline.

BIGBOOT

RSS

• Upgrades the ESS service to RSS.

• Enhances the features and stability of the service.

JindoSDK

• Upgrades the architecture to JindoData.

• EMR integrates with JindoSDK 4.0 for the first time, supporting services such as OSS and OSS-HDFS.

Spark

• The COUNT DISTINCT function supports IF statements and optimizes the usage of CASE WHEN.

  Set the spark.sql.optimizer.rewriteConditionalDistinctAggregates parameter to true.

• Shuffle Hash Join supports fallback to Sort Merge Join.

  Set the spark.sql.join.preferSortMergeJoin parameter to false, and set the spark.sql.join.enableShuffledHashJoinFallback parameter to true.

• Supports automatic merging of small files for non-dynamic partitions.

  Set the spark.sql.adaptive.merge.output.small.files.enabled parameter to true.

• The concurrency is automatically adjusted for scenarios such as GroupingSet and Distinct.

  Set the spark.sql.execution.optimizeExpand parameter to true.

• Optimizes Hive on Spark.

• Supports Time Travel syntax.

• Adapts to JindoSDK.

Tez

Adapts to JindoSDK.

Hive

• Optimizes the batch deletion of Hive Jindo.

• Optimizes the HiveServer2 OOM issue.

• Optimizes Hive on Spark.

• Adapts to JindoSDK.

Presto

• Upgrades Presto to community version 358.

• Adds MySQL, Iceberg, Hudi, Phoenix, Kudu, and Delta connectors by default and updates the default configurations.

• Supports data lake analytics.

• Supports dynamic UDF loading.

• Adapts to JindoSDK.

Delta Lake

• Version upgrade

  • Upgraded to version 1.1.0, which is compatible with Spark 3.2.0.

  • All commercial features are migrated to version 1.1.0.

• Metadata management

  • Optimizes the synchronization of metadata modifications to the metastore.

  • Automatically reports table statistics (dataProfiling) to the metastore.

• SQL

  • Supports Time Travel syntax.

  • Supports DropPartition SQL syntax.

  • Supports dynamic partition overwrites using SQL.

  • Supports ADD COLUMN operations at specified positions (FIRST and AFTER).

• Table management enhancements

  • Supports and enables dynamic adjustment of file sizes based on table sizes by default.

  • Supports and enables automatic Vacuum by default. Concurrent Vacuum is also supported.

  • Optimizes the logic for automatic compaction. This feature is disabled by default.

  • Adds Z-order syntax and accelerates the Z-order process.

Hudi

• Upgraded to version 0.10.0.

• Supports Spark 3.2.0.

• Supports JindoFS Block mode.

HDFS

Adapts to JindoSDK.

YARN

• Adapts to RSS memory configurations.

• Adapts to JindoSDK.

Flume

Adapts to JindoSDK.

Impala

Adapts to JindoSDK.

Ranger

• Supports Spark 3.2.0.

• Supports Presto 358.

HBase

• Fixes issues with default parameters.

• Fixes an issue with the GC log date format.

Clickhouse

• Adds HDFS and OSS disk types to support hot and cold data separation. For more information, see Use HDFS for hot and cold data separation and Use OSS for hot and cold data separation.

• In Replicated*MergeTree scenarios, Zero Copy is supported for OSS, HDFS, and S3 disk types.

• Optimizes the processing logic when the ClickHouse component is stopped.

Iceberg

• Upgraded to version 0.13.0.

• Supports Presto 358.

DLF-Auth

• Supports Spark 3.2.0.

• Supports Presto 358.

Service

Change

SmartData

• SmartData is updated to 3.8.0. For more information, see SmartData 3.8.X overview.

• Authentication and authorization based on Kerberos and Ranger can be used to manage permissions on data in OSS.

Service

Change

SmartData

Upgrades SmartData to version 3.7.3. For more information, see Introduction to SmartData 3.7.x.

Oozie

Fixes an issue where the Jetty server for Oozie fails to start due to a JAR package conflict in an HA environment.

Impala

Fixes a no such method error that occurs when querying DLF metadata tables.

DLF-Auth

Upgrades DLF-Auth to version 1.0.1.

Service

Change

SmartData

Upgrades SmartData to version 3.7.2. For more information, see Introduction to SmartData 3.7.x.

Spark

• Upgrades Spark to version 3.1.2.

• In Spark 3.x, the Distinct computing performance is optimized for Spark SQL. The optimization feature is triggered if an aggregation operator contains multiple count(distinct case ... when ...) methods.

• The array-index out of bounds error that is returned when some required statistics for Adaptive Query Execution (AQE) are missing is fixed.

• Errors related to AQE and data caching in specific scenarios are fixed.

Hive

The batch metadata optimization feature is supported for Hive on JindoFS (Block). This feature is disabled by default.

Presto

Delta tables support StorageHandler queries.

DeltaLake

• Upgrades DeltaLake to version 1.0.0.

• Unifies delta-connectors for Hive 2 and Hive 3.

• Fixes an error that occurs when delta-connectors query multi-level partitioned tables.

• Supports SQL syntax for multiple features, such as DataSkipping, Optimize, and Zorder.

• Supports synchronizing metadata to the MetaStore.

Hudi

• Hudi is updated to 0.9.0.

• The issue about the compatibility of sql.extension between Delta Lake and Hudi is fixed.

HDFS

The default parameter for NameNode reserved capacity is automatically increased. This ensures that the NameNode promptly enters safe mode when disk space is insufficient.

Storm

The component is offline.

Zeppelin

Upgrades Zeppelin to community version 0.10.0.

Hue

• Fixes an issue where the YARN Job Browser fails to display or stop jobs in some cases.

• Enables the YARN Job Browser in the default configurations.

• Supports the Presto protocol in the default configurations.

Druid

ClickHouse

• Updates the default configurations.

• Supports cluster scale-out.

• Supports the MetaChecker feature.

• Supports reading data using the OSS table engine and OSS table function.

Iceberg

• Upgrades Iceberg to version 0.12.0-1.0.1.

• Fixes an error with Hive Runtime dependencies.

Knox

DLF-Auth

Service

Change

SmartData

Upgraded SmartData to version 3.7.1.

Hue

Fixed an issue where Impala could not be used in high-security clusters.

Kudu

Added support for Kerberos.

HBase

• Fixed an issue where restarting HBase in high-security clusters took too long.

• Fixed an issue where the integration of Spark 3.1.1 with HBase failed.

• Optimized the graceful stop process.

Service

Changes

SmartData

Upgraded SmartData to version 3.7.0.

Spark

Fixed a compatibility issue with Delta Lake.

Hive

Hive on JindoFS (Block mode) supports the batch metadata optimization feature.

This feature is disabled by default.

DeltaLake

• Added support for the DeltaLake partition feature.

• Fixed a compatibility issue between the desc detail command and Spark version 3.1.1.

YARN

• Added appId, CPU, and Memory resource usage information to the node Containers REST API.

• Fixed an issue where ApplicationMaster (AM) logs on released Auto Scaling nodes could not be viewed.

• Fixed an issue where historical data in the State Store caused the cluster to become unavailable.

• Added support for cleaning up released nodes after they are decommissioned by Auto Scaling.

• Improved the graceful decommission logic for Auto Scaling. A node is now marked as offline only after the NodeManager (NM) process ends.

Zookeeper

Upgraded to community version 3.6.3.

Flink

• Added the SmartData component.

• Fixed an issue where password-free access to OSS was not possible when you submit jobs to a DataFlow-Flink cluster using Secure Shell (SSH).

Impala

Fixed an issue where deleting an OSS partition directory directly caused a directory listing loop.

Hue

Fixed a user interface (UI) display issue that occurred when Hue was integrated with Oozie.

Kudu

Upgraded to community version 1.14.0.

Clickhouse

The component version is 21.3.13.9.

Iceberg

Added the Iceberg component. The component version is 0.12.0.