E-MapReduce:EMR 3.x series release notes

Version

Date

EMR-3.55.0

October 27, 2025

Service

Change

Ranger

• Jindoauth Server supports custom RAM Roles for client users to access OSS.

• Fixed a missing dependency in Ranger-yarn-plugin.

Paimon

Upgraded to version 1-ali-16.3.

JindoCache

Upgraded to version 6.10.1.

Version

Date

EMR-3.54.0

July 10, 2025

Service

Change

Hive

Fixed some known bugs.

Tez

Fixed community bugs to improve performance and stability.

Version

Date

EMR-3.53.0

April 24, 2025

Service

Change

Trino

Fixed an issue where LDAP was unavailable.

YARN

Fixed open source bugs (YARN-10213, YARN-6207, and YARN-9339).

StarRocks

Supports the creation of clusters with separated storage and compute resources.

JindoCache

Upgraded to version 6.8.2.

EMRHOOK

Enhanced stability.

Version

Date

EMR-3.52.1

December 18, 2024

EMR-3.52.0 (New purchases are not supported)

December 4, 2024

Service

Change

Spark

• Fixed a configuration issue that occurred during scale-out.

• Fixed an issue where SASL connections occasionally failed in Kerberos clusters.

Hive

Fixed a configuration issue that occurred during scale-out.

Trino

Resolved an issue where connections failed after LDAP was enabled.

Presto

Zookeeper

Supports adding custom configurations.

Ranger

Replaced the existing Spark 3 Ranger plugin with the version provided by the open source Kyuubi project.

Hudi

Upgraded to version 0.15.0.

Celeborn

Upgraded to version 0.5.2.

JindoCache

Upgraded to version 6.5.3.

StarRocks3

Upgraded to version 3.2.11.

Kyuubi

Upgraded to version 1.9.2.

StarRocks2

Upgraded to version 2.5.22.

Impala

Kudu

Kafka

Kafka-Manager

Version

Date

EMR-3.51.4

December 18, 2024

EMR-3.51.3 (New purchases are not supported)

November 29, 2024

EMR-3.51.2 (New purchases are not supported)

August 29, 2024

EMR-3.51.1 (New purchases are not supported)

June 21, 2024

EMR-3.51.0 (New purchases are not supported)

April 23, 2024

Version

Date

EMR-3.50.0

February 19, 2024

Service

Change

Hudi

Upgraded to version 0.14.0.

Flume

Upgraded to version 1.11.0.

Kyuubi

Upgraded to version 1.7.3.

Impala

Upgraded to version 4.3.0.

Celeborn

Upgraded to version 0.3.2.

JindoCache

Upgraded to version 6.2.0.

Paimon

Upgraded to version 0.7-ali-1.

Kafka

• Upgraded to version 3.6.1.

• Fixed a SASL security authentication vulnerability in the Kafka Connect component.

Spark

Fixed the Commons Text vulnerability.

StarRocks

• Upgraded StarRocks2 to version 2.5.13.

• Upgraded StarRocks3 to version 3.1.5.

Ranger

• Fixed the Commons Text vulnerability.

• Fixed the Spring Security path matching permission bypass vulnerability.

• Fixed the Spring Security forward/include authentication bypass vulnerability.

• Fixed the Spring Framework identity authentication bypass vulnerability under a special matching pattern.

• Supports modifying the period for Ranger to synchronize LDAP users.

Version

Date

EMR-3.49.1

November 16, 2023

EMR-3.49.0 (New purchases are not supported)

October 27, 2023

Service

Change

JindoCache

Added the component. The version is 6.1.1.

JindoData

JindoData is unavailable. You can use JindoCache for data caching and DLF-Auth for authentication.

Spark

Removed jdo-related configurations from hive-site.xml.

HBase

Added a configuration item. You can select the HBase Thrift Server version, including v1 and v2, as needed.

StarRocks

Upgraded StarRocks2 to version 2.5.10.

Doris

Upgraded Doris to version 1.2.7.

Celeborn

Upgraded Celeborn to version 0.3.1.

Paimon

Upgraded Paimon to version 0.6-ali-2.

ClickHouse

Upgraded ClickHouse to version 23.8.2.7.

Version

Date

EMR-3.48.2

August 17, 2023

Service

Change

Trino

• Fixed an issue where the Paimon connector could not successfully query HDFS tables.

• Fixed an issue where worker monitoring metrics could not be read.

Presto

• Upgraded to version 0.283.

• Fixed an issue where worker monitoring metrics could not be read.

ClickHouse

Granted all permissions to the default user by default.

StarRocks

• Renamed the previous StarRocks to StarRocks2.

• Added StarRocks3, version 3.1.2. By default, it is created as a storage-compute coupled version. Storage-compute separated versions are not supported.

Celeborn

Upgraded to version 0.3.0.

Version

Date

EMR-3.47.0

August 3, 2023

Service

Change

Hudi

Upgraded to version 0.13.1.

Paimon

Upgraded to version 0.5-ali-1.

StarRocks

Upgraded to version 2.5.8.

JindoData

Upgraded to version 4.6.11.

Trino

• Upgraded to version 422.

• The Hudi connector supports querying Merge On Read (MOR) tables.

• Optimized error messages for dynamic UDF loading.

Version

Date

EMR-3.46.1

July 13, 2023

EMR-3.46.0 (New purchases are not supported)

June 1, 2023

Version

Date

EMR-3.45.1

April 3, 2023

EMR-3.45.0 (New purchases are not supported)

February 28, 2023

Service

Change

Iceberg

Upgraded to version 0.14.1.

Flink

Upgraded to Flink1.15-vvr-6.0.2, which corresponds to the community Flink 1.15 major version.

Kafka

• Supports LDAP user logon authentication and authorization.

• Supports user group authorization.

Trino

• EMR Presto was renamed to its official community name, Trino.

• Supports Ranger and DLF AUTH.

• Fixed an issue where connections to worker nodes failed after LDAP was enabled with a single click.

JindoSDK

Upgraded to version 4.6.2.

JindoData

Upgraded to version 4.6.2.

HBase

• Supports Ranger.

• Fixed an issue where OSS-HDFS could not be selected as the storage mode when adding a service.

YARN

ACLs are enabled by default in high-security mode.

Starrocks

Upgraded to version 2.3.4.

Doris

Upgraded to version 1.1.5.

Hudi

The console supports configuring hudi-defaults.conf.

Ranger

Supports integration with Trino, YARN, HBase, and Kafka.

DLF-Auth

• Upgraded to version 2.0.2.

• Supports Trino and Impala.

OpenLDAP

Integrated with the Nslcd component.

Kudu

Kudu Tserver can no longer be installed in the Task node group.

Spark

Upgraded to version 3.3.1.

Tez

Upgraded to version 0.10.2.

Kyuubi

Upgraded to version 1.6.0.

Version

Date

EMR-3.43.1

November 08, 2022

EMR-3.43.0 (New purchases are not supported)

October 14, 2022

Service

Change

Hive

Supports one-click integration with LDAP.

Presto

• Upgraded to community version 389.

  Uses the standalone Delta Lake and Hudi connectors provided by the community.

  • This version of the Delta Lake connector does not support Time Travel and Z-Order.

  • This version of the Hudi connector does not support querying MOR tables.

• Supports one-click integration with LDAP.

DeltaLake

• Integrated with DLF for automated lake table management.

• Supports Ranger authorization.

• Fixed an issue where statistics could not be collected for timestamp fields.

• The optimize and vacuum commands now support returning metric information.

Hudi

Upgraded to version 0.11.1.

HadoopCommon

Added a new component to resolve the issue of HDFS, YARN, and JindoSDK configurations overwriting each other.

YARN

Enhanced elastic features.

Ranger

• Supports both Spark2 and Spark3.

• Ranger Usersync supports one-click integration with LDAP.

Kafka

CruiseControl automatically creates related topics on startup.

HBase

Added the component. The version is 1.4.9.

Phoenix

Added the component. The version is 4.14.1.

Doris

Upgraded to version 1.1.1.

StarRocks

Upgraded to version 2.2.3.

ClickHouse

Fixed a memory overflow issue when reading large files from OSS.

Service

Change

JindoData

Added the component. The version is 4.3.0.

JindoSDK

Upgraded to version 4.3.0.

Spark

Upgraded to version 3.2.1.

Hive

• Fixed a bug where TEZ repeatedly committed when Speculation was enabled.

• Fixed a bug where UDFs could only be called after reloading the function.

Presto

Fixed a bug where the Presto service could not be started after it was added when the Hadoop cluster was initialized.

DeltaLake

Fixed a compatibility issue with Streaming SQL.

Hudi

Upgraded to version 0.10.1.

Iceberg

Upgraded to version 0.13.1.

YARN

• Added a feature to restrict ApplicationMasters (AMs) to run only on CORE group nodes.

• Fixed an issue where the mareduce.map.java.opts configuration was missing taihaodoctor.

Zookeeper

Optimized JVM parameter configurations.

Flink

Adapted to JindoSDK 4.3.0.

Impala

Flume

Druid

Sqoop

Upgraded the PostgreSQL version.

Zeppelin

Resolved a startup failure issue with the JDBC Interpreter.

Ranger

The Ranger 1.2.0 Spark Plugin supports Hudi.

Oozie

Upgraded Log4j to version 2.17.2.

HBase

Fixed an issue where RegionServer could not be started in HBase 1.4.9.

DLF-Auth

Upgraded to version 2.0.0.

Version

Date

EMR-3.39.2

March 25, 2022

EMR-3.39.1 (New purchases are not supported)

February 15, 2022

Version

Date

EMR-3.38.3

December 2021

EMR-3.38.2 (New purchases are not supported)

December 2021

EMR-3.38.1 (New purchases are not supported)

November 2021

EMR-3.38.0 (New purchases are not supported)

October 2021

Version

Date

EMR-3.37.1

September 2021

EMR-3.37.0 (New purchases are not supported)

August 2021

Service

Changes

SmartData

Upgraded SmartData to version 3.6.1.

For more information, see Introduction to SmartData 3.6.x.

Hive

• Upgraded Hive to version 2.3.8.

• Fixed an issue where an incorrect result was returned when you execute the show create table command using Data Lake Formation (DLF) metadata.

• Optimized the default parameters of Hive to improve job performance.

• Changed the names of configuration items on the hive-env tab of the Hive service Configuration page in the E-MapReduce console to uppercase for ease of use.

• The error message that is reported because of the incompatibility between the file system and Hive metastore when you write data to a Hive table is optimized.

HDFS

Added support for the Zstandard (ZSTD) compression format.

Flink

Upgraded Flink to version 1.12-vvr-3.0.2.

Hudi

• Upgraded Hudi to version 0.8.0.

• Added support for integration with Spark SQL.

Spark

• Optimized the names of configuration items on the spark-defaults tab of the Spark service Configuration page in the E-MapReduce console.

• Optimized the performance of log output.

• Added support for the ZSTD compression format.

Impala

Fixed an issue that caused a core dump error when you use Hadoop Distributed File System (HDFS).

Tez

Optimized the default parameters of Tez to improve job performance.

Knox

• Added support for the Kudu component.

• Added support for the Impala component.

• Added support for the Hbase component.

Phoenix

Fixed an issue where a "Java Database Connectivity (JDBC) Driver not found" error was reported when you use Hive or Spark SQL to access Phoenix tables.

ClickHouse

Enabled application performance management (APM) monitoring and alerting.

Service

Change

SmartData

Upgraded to version 3.5.0.

For version details, see Introduction to SmartData 3.5.x.

Spark

• Fixed an issue where Adaptive Execution did not take effect in some scenarios.

• Fixed an issue where the behavior of statistical aggregate functions was inconsistent with that of Hive.

• Fixed an issue where data of the char type was read incorrectly from Hive ORC tables.

HDFS

Adds support for the SM4 national encryption algorithm.

Hue

Upgraded Hue to version 4.9.0.

Alluxio

Upgraded Alluxio to version 2.5.0.

Druid

• Upgraded Druid to version 0.20.1.

• Enhanced security.

Livy

Upgraded Livy to version 0.7.1.

Service

Changes

SmartData

Upgraded to version 3.4.0.

For more information, see Introduction to SmartData 3.4.x.

Spark

• Optimized some default configurations.
• Performance optimization: Added support for Window TopK pushdown.
• Enhanced compatibility for reading and writing CSV or JSON tables in Hive.
• The ANALYZE statement now supports omitting all table column names.
• Added support for enabling or disabling the Lightweight Directory Access Protocol (LDAP) feature with a single click.
• Improved the usability of the Spark Beeline tool.

Hive

• Optimized some default configurations.

• Performance optimization: Enhanced the cost-based optimizer (CBO).

• Added support for enabling or disabling the LDAP feature with a single click.

• Upgraded Calcite to version 1.12.0.

• Added the hive.security.authorization.sqlstd.confwhitelist.append parameter.

Presto

Added support for enabling or disabling the LDAP feature with a single click.

YARN

Fixed an important security threat related to unauthorized access to the Hadoop web UI. The threat occurred when accessing the YARN web UI through a Secure Shell (SSH) tunnel, which required user.name=name to be explicitly specified in the URL.

Zookeeper

Upgraded to version 3.6.2.

Flink

Updated the config.sh file during initialization to fix an issue with HADOOP_CLASSPATH.

Impala

• Upgraded Impala to version 3.4.0.

• Upgraded Shiro to version 1.7.0.

• Added support for Data Lake Formation (DLF) metadata.

• Added support for querying data in Delta format.

• Added support for enabling or disabling the LDAP feature with a single click.

Tez

Optimized the default configurations.

HAS

Fixed an issue where the admin.keytab file could not be re-initialized after an error occurred during the HAS installation flow.

Ranger

• The issue caused by filter pushdown in Spark is fixed.

• The issue that prevents Presto from being enabled after you disable Presto in Ranger is fixed.

• LDAP authentication can be enabled or disabled with a click.

Knox

Fixed an issue with the Knox link for Druid 0.20.0.

Hue

Added support for enabling or disabling the LDAP feature with a single click.

Hudi

• Added support for the SQL on Hudi feature.
• Fixed an accuracy issue that occurred when querying partial data.
• Added support for partition pruning when you query Copy On Write tables in Hudi using Spark.
• Added support for a bucketing index mechanism to improve write performance.

Delta Lake

• Fixed an issue where metadata could not be synchronized to Hive Metastore from an existing Delta table.
• Fixed an issue where the MERGE command could not parse the * character.
• Fixed an error that occurred during the creation of table metadata when transforming data from Parquet format to a Delta table.
• Fixed an issue where the OPTIMIZE command failed when there were no files to compact.
• The MERGE syntax now supports using a subquery as the source.
• Introduced a caching mechanism to improve query efficiency when you use Presto to query Delta tables.
• Added support for querying Delta tables using Impala.

Superset

• The issue that prevents the admin user from logging on to the web UI is fixed.

• Datasets are compatible with Druid clusters.

• Spark SQL datasets are no longer supported.

Sqoop

Added support for importing files in Parquet format to Object Storage Service (OSS).

Alluxio

Upgraded to version 2.4.1.

Phoenix

Hive on Phoenix now supports backing field settings.

Pig

Removed.

Service

Changes

SmartData

Upgraded to version 3.2.0.

For more information, see Introduction to SmartData 3.2.x.

Spark

• Upgraded to version 2.4.7.

• Upgraded jQuery to version 3.5.1.

• Added compatibility with Hive to automatically update table and partition sizes.

• Added support for outputting Spark metadata and job running information to DataWorks.

Hive

• Upgraded to version 2.3.7.

• HCatalog now supports Data Lake Formation.

• Added support for outputting Hive metadata and job running information to DataWorks.

Metastore

• Added the Hive Statistics feature.

• HCatalog now supports Data Lake Formation.

• Optimized the method for obtaining STS tokens.

HDFS

Upgraded jQuery to version 3.5.1.

YARN

• Upgraded jQuery to version 3.5.1.

• Adjusted the Fair Scheduler configuration.

• Optimized Timeline Server.

Zeppelin

Upgraded to version 0.9.0.

Ranger

• Added audit log configuration for Hive.

• Added audit configuration for Log4j.

OpenLDAP

• Added an audit feature.

• Enabled the SSL port (10636) by default.

• Added support for one-click startup of Presto.

Knox

• Fixed a Spring vulnerability.

• Fixed an issue with viewing the Executors page in the Spark UI.

• Fixed an issue with the Oozie job status page.

Hue

Added support for Presto.

Druid

Upgraded to version 0.20.0.

EMRHook

• Added a new software service.

• hive-hook: Supports outputting Hive metadata and job running information to DataWorks.

• spark-hook: Supports outputting Spark metadata and job running information to DataWorks.

Service

Changes

SmartData

Upgraded to version 3.1.0.

For more information, see Introduction to SmartData 3.1.x.

Alluxio

• Supports Alluxio 2.4.0.

• Default parameter settings scale with cluster node size.

• Uses HDFS in the EMR cluster as the default UnderFS. This feature is ready to use out of the box.

• Enhanced the Alluxio OSS UnderFS to support new features such as OSS multi-versioning.

• Compatible with engines such as Hadoop, Hive, Spark, and Presto.

HUDI

Supports HUDI 0.6.0.

Spark

JindoTable supports enabling or disabling the data collection feature.

Hive

• Fixed a connection pool leak issue in HiveServer.

• JindoTable supports enabling or disabling the data collection feature.

• Optimized the performance of ADD COLUMN.

• Fixed an issue where incorrect data was read from HUDI tables.

• Default parameter settings scale with cluster node size.

HDFS

Supports a larger number of snapshots.

YARN

Default parameter settings scale with cluster node size.

Tez

Default parameter settings scale with cluster node size.

Sqoop

Fixed an issue with importing files in Avro format.

Service

Updates

SmartData

Upgraded to 3.0.0.

For more information, see Introduction to SmartData 3.0.x.

Spark

• Added support for Alibaba Cloud Data Lake Formation (DLF) metadata.

• Upgraded the HAS dependency to 2.0.1.

• Fixed an issue with backticks in Streaming SQL.

• Removed the Delta JAR package. Delta is now deployed separately.

• Modified the log path to write all logs to HDFS.

Hive

• Added support for Alibaba Cloud DLF metadata.

• Resolved an issue where a DUMMY file was written when reading an empty directory in a Delta table.

• Upgraded the HAS dependency to 2.0.1.

Presto

• Added support for Alibaba Cloud DLF metadata.

• Resolved an issue that limited the reading of Delta tables.

• Fixed an issue where the JVM configuration was missing in high-security mode.

• Upgraded the HAS dependency to 2.0.1.

HDFS

• Added support for hot-swappable disk mode.

• Upgraded the HAS dependency to 2.0.1.

YARN

• Fixed an issue with YARN RMZKStateStore.

• Added support for SNAPPY files output by SLS.

• Modified the directory configuration for MapReduce Local mode to resolve a directory permission check issue.

• Added support for hot-swappable disk mode.

• Set the log path to write all logs to HDFS.

• Upgraded the HAS dependency to 2.0.1.

Zookeeper

• Added support for attaching the service port to an internal IP address at startup.

• Upgraded the HAS dependency to 2.0.1.

Flink-Vvp

• Upgraded to version 1.11-2.2.2.

• Added support for SQL and Autopilot features.

Flink

• Added support for writing to OSS in cache mode. This feature, combined with Flink Checkpoints and a resumable Source, achieves EXACTLY_ONCE semantics.

• Synchronized with Flink community version 1.11.1 features. SQL now supports multiple outputs (MULTI INSERT).

• Upgraded the HAS dependency to 2.0.1.

Impala

• Added support for custom configurations of catalogd.flgs, impalad.flgs, and statestored.flgs.

• Upgraded Shiro to version 1.6.0.

• Upgraded the HAS dependency to 2.0.1.

Tez

• Optimized the default memory parameters for the Application Master (AM).

• Upgraded the HAS dependency to 2.0.1.

HAS

Upgraded the HAS dependency to 2.0.1.

Storm

Zeppelin

Ranger

OpenLDAP

Oozie

Knox

Kafka

HUE

HBase

Druid

Service

Changes

Bigboot

• Upgraded to version 2.7.301.

• Jindo DistCp now supports writing data to OSS with the Archive or Infrequent Access storage class.

• Enhanced the FUSE feature to support multiple namespaces.

• Improved the metadata caching feature in Cache mode.

Spark

• Upgraded Spark to 2.4.5.2.0.

• Added support for third-party metastores.

• Added the datalake metastore-client.

Hive

• Upgraded Hive to 2.3.5.6.0.

• Added support for third-party metastores.

• Added the datalake metastore-client.

Presto

Upgraded to version 338.

Ranger

• Upgraded the software package to 1.2.0-1.5.0.

• Added support for Presto 338.

• Added descriptions to configuration files.

Hadoop Distributed File System (HDFS)

Enabled adaptive configuration for the reserved space size of datanodes.

Knox

Impala, later versions of Flink, and PAI are supported.

Druid

Upgraded to version 0.18.1.

SmartData

Upgraded to version 2.7.301.

Service

Changes

Bigboot

• Releases the first version of JindoTable, which provides hotspot statistics for tables and partitions.

• Adds support for complete storage policies in Block mode and tiered storage policies, such as Infrequent Access and Archive.

• Adds the Jindo DistCp data migration tool.

• Improves and fixes Jindo Fuse.

• Improves the integration of the JFS scheme with the Hive engine and Jindo JobCommitter in Cache mode.

• Adds a feature to set a read ratio in Block mode for reading data directly from OSS. This reduces the overhead of reading from the local cache.

• Decouples JindoFS software modules into Bigboot (control layer), Smartdata (distributed service), and the JindoFS SDK. Each module can be independently upgraded and maintained.

Service

Changes

Flink

Upgrades open source Flink to Ververica Platform Enterprise Edition. The platform is heavily customized based on open source Flink 1.10 and provides value-added features, such as the self-developed Gemini storage engine.

Bigboot

Upgrades to version 2.7.0.

Delta

• Upgrades to version 0.6.0.

• Decouples the Delta code from the Spark code.

Spark

• Upgrades to version 2.4.5.

• Supports streaming-sql scripts from DataFactory.

• Supports Delta 0.6.0.

Hive

Supports Delta 0.6.0.

Ranger

• Supports custom deployments of Hadoop Distributed File System (HDFS), Hive, and Spark.

• Supports the configuration of ranger-admin-site and ranger-ugsync-site in the console.

HDFS

Now prints DataNode exception information when an HDFS write fails due to no available DataNodes (HDFS-9023).

Hue

• Supports installing the Hue component on Gateway clusters.

• Supports deploying multiple Hue instances on a single node.

DataFactory

Supports Delta 0.6.0.

Druid

Upgrades to version 0.18.0.

Knox

• Upgrades to version 1.1.0-1.0.7.

• Supports the HBase UI.

Version

Date

EMR-3.27.0

April 29, 2020

EMR-3.27.1 (New purchases are not supported)

May 8, 2020

EMR-3.27.2 (New purchases are not supported)

May 20, 2020

Feature

Change

Custom component deployment

Added support for custom deployment of components on master nodes. The following components are supported:

• Hadoop

• Spark

• Hive

• Zookeeper

• Presto

Graceful shutdown for Auto Scaling

When graceful shutdown is enabled, nodes are not released immediately. They are released after tasks are completed within a specified time period.

Service

Change

Spark

• CUBE now supports date type partition fields.

• Increased the stack depth of Spark-Submit.

Delta

• Enhanced Data Definition Language (DDL) syntax, including commands such as CREATE, SHOW, and DESCRIBE.

• Delta now supports the Optimize syntax with ZOrder.

Knox

• Adapted for the Druid User Interface (UI).

• Multi-master deployment is supported.

Hive

• hcatalog tables now support the magic committer.

• Removed some outdated default configurations.

Bigboot

• Upgraded to version 2.6.3.

• Multi-master deployment is supported.

SmartData

• Upgraded to version 2.6.3.

• Multi-master deployment is supported.

Ranger

• Ranger now supports the Solr component.

• Ranger now supports PrestoSQL version 311.

Tez

Tez now supports setting scratchdir on OSS.

Presto

Upgraded to version 331.

Druid

Upgraded to version 0.17.1.

Superset

Upgraded to version 0.35.2.

Sqoop

• The MySQL Java Database Connectivity (JDBC) JAR package is upgraded to version 5.1.48.

• The MySQL direct export mode supports setting a custom encoding using --mysql-charset.

Version

Date

EMR-3.26.3 (New purchases are not supported)

April 16, 2020

Service

Changes

Bigboot

• Upgraded to version 2.6.3.

• Added support for OTS metadata and Namespace HA.

SmartData

Hive

HCatalog tables now support the direct committer.

YARN

Changed the default committer to JindoOssCommitter.

HDFS

Upgraded JindoFS-related configurations.

Spark

Changed the default committer to JindoOssCommitter.

Service

Changes

Ranger

• Initialized the RangerAdmin database for high-availability (HA) clusters.

• Fixed a security issue in the RangerUserSync startup script.

Spark

• Added support for configuring Delta-related parameters, such as spark.sql.extensions, in the console.

• Added support for Hive to read Delta tables without setting the input format.

• Added support for the ALTER TABLE SET TBLPROPERTIES and UNSET TBLPROPERTIES statements.

Delta

Hive

Fixed an issue where MapReduce (MR) task execution failed in automatic local mode.

Presto

• Upgraded to version 310.

• Upgraded the joda-time version to 2.10.5.

Tez

• Upgraded to version 0.9.2.

• Fixed an issue where the application progress was not displayed correctly in the Tez user interface (UI).

• Fixed an issue where the application history could not be viewed in the Tez UI.

Impala

Fixed an issue where Impala could not access LZO tables.

HDFS

Removed mongo-hadoop related JAR packages.

Zookeeper

Upgraded to version 3.5.6.

YARN

Adapted for the Tez UI. The yarn-site tab now supports adding the configuration item yarn.resourcemanager.system-metrics-publisher.enabled=true.

Bigboot

• Upgraded to version 2.2.3.

• Added support for rename operations in OSS Cache mode.

SmartData

Knox

Upgraded dependency package versions.

Oozie

Upgraded dependency package versions.

Service

Changes

Delta

• Supports SQL syntax, including ALTER, CONVERT, CREATE, CTAS, DELETE, DESC, INSERT, MERGE, OPTIMIZE, UPDATE, and VACUUM.

• Built-in and optimized the OPTIMIZE command.

• Supports the Hive connector.

• Supports other existing open-source features.

Grafana

Added as a new component for standalone Flink clusters. Version: 6.4.2.

Prometheus

Added as a new component for standalone Flink clusters. Version: 2.13.0.

AlertManager

Added as a new component for standalone Flink clusters. Version: 0.19.0.

TensorFlow on spark

• Supports running TensorFlow on Spark. This deeply integrates Spark with the deep learning framework. The integration includes optimized task scheduling and data exchange. It provides a complete workflow, from data pre-processing to deep learning training.

• Supports streaming tasks.

Service

Changes

SmartData

• Optimized JindoFS usage modes. The usage of Block mode is unchanged. Cache mode now supports its original usage and is also compatible with the original OSS file system usage. It supports data and metadata caching. These features can be enabled or disabled separately through configuration and are disabled by default.

• Optimized read and write performance for Block mode and Cache mode.

• Optimized disk cleanup. This provides more accurate statistics and more timely cleanup for hot data cached on local disks. It strictly ensures that disk usage does not exceed the quota.

• Improved support for Gateway clusters. Block mode and Cache mode can now be used on a Gateway.

• Supports a deployment mode where one storage cluster is separated from multiple compute clusters.

Spark

• Added support for Delta-related parameters.

• Added support for Ranger Spark plugin configuration.

• Upgraded JindoCube to version 0.3.0.

Hive

• Added logic for the SQL compatibility check feature.

• Released a combination of Hive 2.3.5 and Hadoop 2.8.5.

• When restarting the component, the content of hiveserver2-site.xml is no longer synchronized to hive-site.xml under spark-conf.

• Supports using the MSCK command to add incremental folders.

• Fixed a bug that occurred when Hive reused a Tez container.

• Supports using the MSCK command to optimize column-based folders.

Bigboot

Upgraded to 2.2.1. Fixed issues with native code support on some machine models.

Ranger

• Refactored the deployment method for the Spark plugin.

• Fixed a bug where header2 in an HA cluster did not obtain the keytab.

Kudu

Fixed the startup logic.

Zookeeper

Added configuration for four-letter words. This is enabled by default.

HDFS

Added compatibility with JindoFS.

YARN

• Changed the default value of the yarn.scheduler.capacity.node-locality-delay configuration to -1.

• Added compatibility with JindoFS.

Has

Integrated with OpenLDAP as the backend.

OpenLDAP

Added compatibility with Has.

Presto

Upgraded to version 0.228.

Kafka

Removed D1 bad disks.

Druid

Upgraded to 0.16.0.

Flume

Upgraded to 1.9.0.

Flink

• Upgraded to 1.9.1.

• Supports standalone Flink clusters (released to a whitelist).

Service

Changes

Druid

• Upgraded to 0.15.1.

• Added the router component.

• Upgraded fastjson.

Spark

• Updated Spark Thrift Server to fix a class loader issue.

• Refactored Spark transaction code to improve stability.

• Fixed an issue with reading and writing files in ORC format after the built-in Hive was upgraded to version 2.3.

• Added support for the MERGE INTO syntax.

• Added support for the SCAN and STREAM syntax.

• The Structured Streaming Kafka sink now supports exactly-once semantics (EOS).

• Updated Delta Lake to 0.4.0.

Hive

• Removed the old version of the Hive hook.

• Added an optimization to handle data skew for multiple COUNT(DISTINCT) fields.

• Fixed an issue where data was lost when joining tables with different bucket versions.

Flink

Upgraded to 1.8.2.

Bigboot

• Updated the small file tool.

• Updated the OSS JAR package to fix a non-daemon thread issue.

Kafka

• Added support for the Deployment Set awareness feature.

• Removed the fastjson dependency.

HDFS

• Optimized the deployment logic for the SmartData OSS JAR package.

• Updated the SmartData OSS JAR package.

Flume

Upgraded fastjson.

TensorFlow on Spark

Added this service.

HAS

Upgraded fastjson.

Livy

Upgraded fastjson.

Service

Change

Kudu

• Added Kudu as a new component. Kudu fills a gap in the Hadoop ecosystem. It provides fast data inserts and random access similar to HBase, and lets you modify data. It also provides large-scale data analytics and query capabilities similar to Hadoop Distributed File System (HDFS) or Parquet.

  • Provides C++ and Java APIs for custom development.

  • Integrates with Impala, Spark, and Hive Metastore.

• This version of Kudu is based on Apache Kudu 1.10.0.

OpenLDAP

• Added OpenLDAP as a new component to replace ApacheDS. ApacheDS is now offline.

• Supports high availability (HA).

Component

Details

JindoFileSystem

• Multiple storage modes

  • Block mode: Data is stored as blocks in the backend OSS. The local Namespace service maintains metadata. Block mode provides better metadata and data performance. Block mode supports different storage policies, including WARM (local replicas, OSS replicas), COLD (OSS replicas only), HOT (multiple local replicas, OSS replicas), TEMP (local replicas only), and ALL_HDD (multiple local replicas). The default policy is WARM. You can set different storage policies for folders based on your application scenario.

  • Cache mode: This mode is compatible with existing OSS storage methods. In Cache mode, files are stored as objects in OSS. Data and metadata for each file are cached locally based on access frequency. This improves data and metadata access performance. Cache mode provides different metadata synchronization policies to meet the needs of different scenarios.

• External client support

  • The client software development kit (SDK) lets you access the EMR JindoFS file system from outside an EMR cluster. You can use the client to access the Namespace in Block mode. However, external clients cannot use the data cache built by EMR JindoFS within the EMR cluster. This results in lower performance compared to using it within the EMR cluster.

  • Cache mode retains the original OSS storage semantics. It uses JindoFS to accelerate data caching within the EMR cluster. Therefore, you can directly access data from outside the EMR cluster using an OSS client, such as the OSS SDK or EMR OssFileSystem.

• Ecosystem component support

  • JindoFS now supports many compute engines on EMR, such as Spark, Flink, Hive, MapReduce, Impala, and Presto.

  • For scenarios that separate computing and storage, you can also store job logs in JindoFS, such as YARN Container logs and Spark Event logs.

  • JindoFS can be used as the HFile backend storage for HBase to expand its storage capacity.

OssFileSystem

• Added logic to OssFileSystem to automatically detect bad disks. This fixes an issue where cache writes failed during OSS writes due to bad disks.

• Completed the related configurations for OssFileSystem.

Bigboot

• Upgraded to version 2.0.0.

• Includes several major updates, such as support for multiple Namespaces, storing local data blocks as large files, multi-mode storage, and external clients.

• Fixed an issue where the Bigboot monitor status was incorrect during a machine restart.

• Added a service spec for the Kudu component.

• Added correctness checks for all service specs.

Hadoop

• HDFS

  • Adapted for HDFS Federation. You can now create HDFS Federation clusters using custom configurations and APIs. This avoids the need for a second format operation when creating a Federation cluster.

  • Optimized the bad disk detection logic. For local disk scenarios, you can trigger bad disk detection when a DataNode block report is triggered by dfsadmin.

• YARN

  Fixed an issue where the MapReduce JobHistory job list did not update when MapReduce job Container logs were stored in JindoFS or OSS.

Spark

• Relational Cache

  Added support for Relational Cache. Relational Cache uses pre-computation to accelerate user queries. You can create a Relational Cache to pre-compute data. When a user query is executed, the Spark Optimizer automatically finds a suitable cache, rewrites the SQL execution plan, and continues the computation based on the cached data. This improves query speed. This feature is suitable for scenarios such as reports, dashboards, data synchronization, and multidimensional analysis.

  • Use Data Definition Language (DDL) to perform operations such as CACHE, UNCACHE, ALTER, and SHOW. Cached data supports all Spark data sources and data formats.

  • Supports automatic cache data updates and updates using the REFRESH command. Supports incremental updates based on partitions.

  • Supports execution plan optimization based on Relational Cache.

• Streaming SQL

  • Standardized the parameter configuration for Stream Query Writer.

  • Optimized the schema compatibility check for Kafka data tables.

  • If a Kafka data table schema does not exist, it is automatically created in SchemaRegistry.

  • Optimized the log information for when a Kafka schema is incompatible.

  • Fixed an issue where column names had to be explicitly specified when writing query results to a Kafka table.

  • Removed the restriction that streaming SQL queries only support Kafka and Loghub data sources.

• Delta

  Added Delta. You can use Spark to create a Delta data source to support scenarios such as streaming data writes, transactional reads and writes, data validation, and data history. For more information, see Delta details.

  • Supports using the DataFrame API to read data from or write data to Delta.

  • Supports using the Structured Streaming API to read from or write to Delta as a source or sink.

  • Supports using the Delta API to perform operations such as update, delete, merge, vacuum, and optimize.

  • Supports using SQL to perform operations such as creating Delta-based tables, importing data to Delta, and reading from Delta tables.

• Others

  • Added a constraint feature that supports primary keys and foreign keys.

  • Resolved JAR file conflicts, such as for servlets.

Flink

Rollback of Log4j logs

Kafka

• Log rollback for Log4j.

• Upgraded fastjson.

Zeppelin

Upgraded the dependent commons-lang3 package to version 3.7. This fixes an issue where PySpark could not write to OSS. For more information, see Spark 2.4 incompatibility with commons-lang3 in Zeppelin.

Ranger

Added support for SHOW GRANTS.

Analytics-Zoo

Fixed a NumPy installation error.

Impala

Now compatible with Apache Kudu 1.10.0.

Presto

Upgraded to version 0.221.

ZooKeeper

Upgraded to version 3.5.5.