E-MapReduce:EMR 3.x series release notes

Release version information

Update details

• Jindoauth Server supports custom RAM Roles for client users to access OSS.

• Fixed a missing dependency in Ranger-yarn-plugin.

Paimon| Upgraded to version 1-ali-16.3. JindoCache| Upgraded to version 6.10.1.

Release date

Release version information

Update details

Release date

Release version information

Update details

Release date

Update details

• Fixed a configuration issue that occurred during scale-out.

• Fixed an issue where SASL connections occasionally failed in Kerberos clusters.

Hive| Fixed a configuration issue that occurred during scale-out. Trino| Resolved an issue where connections failed after LDAP was enabled. Presto Zookeeper| Supports adding custom configurations. Ranger| Replaced the existing Spark 3 Ranger plugin with the version provided by the open source Kyuubi project. Hudi| Upgraded to version 0.15.0. Celeborn| Upgraded to version 0.5.2. JindoCache| Upgraded to version 6.5.3. StarRocks3| Upgraded to version 3.2.11. Kyuubi| Upgraded to version 1.9.2. StarRocks2| Upgraded to version 2.5.22. Impala| The service is unavailable. You can use the recommended service as an alternative or manually install the corresponding service. You can replace Impala with Presto, Trino, ClickHouse, or StarRocks. Kudu Kafka-Manager

Release date

Update details

Release date

Update details

• Upgraded to version 3.6.1.

• Fixed a SASL security authentication vulnerability in the Kafka Connect component.

Spark| Fixed the Commons Text vulnerability. StarRocks|

• Upgraded StarRocks2 to version 2.5.13.

• Upgraded StarRocks3 to version 3.1.5.

Ranger|

• Fixed the Commons Text vulnerability.

• Fixed the Spring Security path matching permission bypass vulnerability.

• Fixed the Spring Security forward/include authentication bypass vulnerability.

• Fixed the Spring Framework identity authentication bypass vulnerability under a special matching pattern.

• Supports modifying the period for Ranger to synchronize LDAP users.

Release date

Update details

Release date

Update details

• Fixed an issue where the Paimon connector could not successfully query HDFS tables.

• Fixed an issue where worker monitoring metrics could not be read.

Presto|

• Upgraded to version 0.283.

• Fixed an issue where worker monitoring metrics could not be read.

ClickHouse| Granted all permissions to the default user by default. StarRocks|

• Renamed the previous StarRocks to StarRocks2.

• Added StarRocks3, version 3.1.2. By default, it is created as a storage-compute coupled version. Storage-compute separated versions are not supported.

Celeborn| Upgraded to version 0.3.0.

Release date

Update details

• Upgraded to version 422.

• The Hudi connector supports querying Merge-On-Read (MOR) tables.

• Optimized error messages for dynamic UDF loading.

Release date

Update details

Release date

Update details

Release date

Update details

• Supports LDAP user logon authentication and authorization.

• Supports user group authorization.

Trino|

• EMR Presto was renamed to its official community name, Trino.

• Supports Ranger and DLF AUTH.

• Fixed an issue where connections to worker nodes failed after LDAP was enabled with a single click.

JindoSDK| Upgraded to version 4.6.2. JindoData| Upgraded to version 4.6.2. HBase|

• Supports Ranger.

• Fixed an issue where OSS-HDFS could not be selected as the storage mode when adding a service.

YARN| ACLs are enabled by default in high-security mode. Starrocks| Upgraded to version 2.3.4. Doris| Upgraded to version 1.1.5. Hudi| The console supports configuring hudi-defaults.conf. Ranger| Supports integration with Trino, YARN, HBase, and Kafka. DLF-Auth|

• Upgraded to version 2.0.2.

• Supports Trino and Impala.

OpenLDAP| Integrated with the Nslcd component. Kudu| Kudu Tserver can no longer be installed in the Task node group. Spark| Upgraded to version 3.3.1. Tez| Upgraded to version 0.10.2. Kyuubi| Upgraded to version 1.6.0.

Release date

EMR-3.44.0 was released on December 1, 2022.

Update details

Release date

Update details

• Upgraded to community version 389.Uses the standalone Delta Lake and Hudi connectors provided by the community.

  • This version of the Delta Lake connector does not support Time Travel and Z-Order.

  • This version of the Hudi connector does not support querying MOR tables.

• Supports one-click integration with LDAP.

DeltaLake|

• Integrated with DLF for automated lake table management.

• Supports Ranger authorization.

• Fixed an issue where statistics could not be collected for timestamp fields.

• The optimize and vacuum commands now support returning metric information.

Hudi| Upgraded to version 0.11.1. HadoopCommon| Added a new component to resolve the issue of HDFS, YARN, and JindoSDK configurations overwriting each other. YARN| Enhanced elastic features. Ranger|

• Supports both Spark2 and Spark3.

• Ranger Usersync supports one-click integration with LDAP.

Kafka| CruiseControl automatically creates related topics on startup. HBase| Added the component. The version is 1.4.9. Phoenix| Added the component. The version is 4.14.1. Doris| Upgraded to version 1.1.1. StarRocks| Upgraded to version 2.2.3. ClickHouse| Fixed a memory overflow issue when reading large files from OSS.

Release date

EMR-3.42.0 was released on August 5, 2022.

Update details

• Fixed a bug where TEZ repeatedly committed when Speculation was enabled.

• Fixed a bug where UDFs could only be called after reloading the function.

Presto| Fixed a bug where the Presto service could not be started after it was added when the Hadoop cluster was initialized. DeltaLake| Fixed a compatibility issue with Streaming SQL. Hudi| Upgraded to version 0.10.1. Iceberg| Upgraded to version 0.13.1. YARN|

• Added a feature to restrict ApplicationMasters (AMs) to run only on CORE group nodes.

• Fixed an issue where the mareduce.map.java.opts configuration was missing taihaodoctor.

Zookeeper| Optimized JVM parameter configurations. Flink| Adapted to JindoSDK 4.3.0. Impala Flume Druid Sqoop| Upgraded the PostgreSQL version. Zeppelin| Resolved a startup failure issue with the JDBC Interpreter. Ranger| The Ranger 1.2.0 Spark Plugin supports Hudi. Oozie| Upgraded Log4j to version 2.17.2. HBase| Fixed an issue where RegionServer could not be started in HBase 1.4.9. DLF-Auth| Upgraded to version 2.0.0.

Release date

EMR-3.40.0 was released on April 21, 2022.

Update details

Release date

Update details

Release date

Update details

Release date

Updates

• Upgraded Hive to version 2.3.8.

• Fixed an issue where an incorrect result was returned when you execute the show create table command using Data Lake Formation (DLF) metadata.

• Optimized the default parameters of Hive to improve job performance.

• Changed the names of configuration items on the hive-env tab of the Hive service Configuration page in the E-MapReduce console to uppercase for ease of use.

• The error message that is reported because of the incompatibility between the file system and Hive metastore when you write data to a Hive table is optimized.

HDFS| Added support for the Zstandard (ZSTD) compression format. Flink| Upgraded Flink to version 1.12-vvr-3.0.2.__Note Flink is removed from Hadoop clusters. Hudi|

• Upgraded Hudi to version 0.8.0.

• Added support for integration with Spark SQL.

Spark|

• Optimized the names of configuration items on the spark-defaults tab of the Spark service Configuration page in the E-MapReduce console.

• Optimized the performance of log output.

• Added support for the Zstandard (ZSTD) compression format.

Impala| Fixed an issue that caused a core dump error when you use Hadoop Distributed File System (HDFS). Tez| Optimized the default parameters of Tez to improve job performance. Knox|

• Added support for the Kudu component.

• Added support for the Impala component.

• Added support for the Hbase component.

Phoenix| Fixed an issue where a "Java Database Connectivity (JDBC) Driver not found" error was reported when you use Hive or Spark SQL to access Phoenix tables. ClickHouse| Enabled application performance management (APM) monitoring and alerting.

Release date

EMR-3.36.1 was released on July 16, 2021.

Updates

• Fixed an issue where Adaptive Execution did not take effect in some scenarios.

• Fixed an issue where the behavior of statistical aggregate functions was inconsistent with that of Hive.

• Fixed an issue where data of the char type was read incorrectly from Hive ORC tables.

HDFS| Adds support for the SM4 national encryption algorithm. Hue| Upgraded Hue to version 4.9.0. Alluxio| Upgraded Alluxio to version 2.5.0. Druid|

• Upgraded Druid to version 0.20.1.

• Enhanced security.

Livy| Upgraded Livy to version 0.7.1.

Release date

EMR-3.35.0 was released on April 21, 2021.

Changes

• Optimized some default configurations.

• Performance optimization: Added support for Window TopK pushdown.

• Enhanced compatibility for reading and writing CSV or JSON tables in Hive.

• The ANALYZE statement now supports omitting all table column names.

• Added support for enabling or disabling the Lightweight Directory Access Protocol (LDAP) feature with a single click.

• Improved the usability of the Spark Beeline tool.

Hive|

• Optimized some default configurations.

• Performance optimization: Enhanced the cost-based optimizer (CBO).

• Added support for enabling or disabling the LDAP feature with a single click.

• Upgraded Calcite to version 1.12.0.

• Added the hive.security.authorization.sqlstd.confwhitelist.append parameter.

Presto| Added support for enabling or disabling the LDAP feature with a single click. YARN| Fixed an important security threat related to unauthorized access to the Hadoop web UI. The threat occurred when accessing the YARN web UI through a Secure Shell (SSH) tunnel, which required user.name=name to be explicitly specified in the URL. Zookeeper| Upgraded to version 3.6.2. Flink| Updated the config.sh file during initialization to fix an issue with HADOOP_CLASSPATH. Impala|

• Upgraded Impala to version 3.4.0.

• Upgraded Shiro to version 1.7.0.

• Added support for Data Lake Formation (DLF) metadata.

• Added support for querying data in Delta format.

• Added support for enabling or disabling the LDAP feature with a single click.

Tez| Optimized the default configurations. HAS| Fixed an issue where the admin.keytab file could not be re-initialized after an error occurred during the HAS installation flow. Ranger|

• The issue caused by filter pushdown in Spark is fixed.

• The issue that prevents Presto from being enabled after you disable Presto in Ranger is fixed.

• LDAP authentication can be enabled or disabled with a click.

Knox| Fixed an issue with the Knox link for Druid 0.20.0. Hue| Added support for enabling or disabling the LDAP feature with a single click. Hudi|

• Added support for the SQL on Hudi feature.

• Fixed an accuracy issue that occurred when querying partial data.

• Added support for partition pruning when you query Copy-On-Write (COW) tables in Hudi using Spark.

• Added support for a bucketing index mechanism to improve write performance.

Delta Lake|

• Fixed an issue where metadata could not be synchronized to Hive Metastore from an existing Delta table.

• Fixed an issue where the MERGE command could not parse the * character.

• Fixed an error that occurred during the creation of table metadata when transforming data from Parquet format to a Delta table.

• Fixed an issue where the OPTIMIZE command failed when there were no files to compact.

• The MERGE syntax now supports using a subquery as the source.

• Introduced a caching mechanism to improve query efficiency when you use Presto to query Delta tables.

• Added support for querying Delta tables using Impala.

Superset|

• The issue that prevents the admin user from logging on to the web UI is fixed.

• Datasets are compatible with Druid clusters.

• Spark SQL datasets are no longer supported.

Sqoop| Added support for importing files in Parquet format to Object Storage Service (OSS). Alluxio| Upgraded to version 2.4.1. Phoenix| Hive on Phoenix now supports backing field settings. Pig| Removed.

Release date

EMR-3.34.0 was released on March 15, 2021.

Updates

• Upgraded to version 2.4.7.

• Upgraded jQuery to version 3.5.1.

• Added compatibility with Hive to automatically update table and partition sizes.

• Added support for outputting Spark metadata and job running information to DataWorks.

Hive|

• Upgraded to version 2.3.7.

• HCatalog now supports Data Lake Formation.

• Added support for outputting Hive metadata and job running information to DataWorks.

Metastore|

• Added the Hive Statistics feature.

• HCatalog now supports Data Lake Formation.

• Optimized the method for obtaining STS tokens.

HDFS| Upgraded jQuery to version 3.5.1. YARN|

• Upgraded jQuery to version 3.5.1.

• Adjusted the Fair Scheduler configuration.

• Optimized Timeline Server.

Zeppelin| Upgraded to version 0.9.0. Ranger|

• Added audit log configuration for Hive.

• Added audit configuration for Log4j.

OpenLDAP|

• Added an audit feature.

• Enabled the SSL port (10636) by default.

• Added support for one-click startup of Presto.

Knox|

• Fixed a Spring vulnerability.

• Fixed an issue with viewing the Executors page in the Spark UI.

• Fixed an issue with the Oozie job status page.

Hue| Added support for Presto. Druid| Upgraded to version 0.20.0. EMRHook|

• Added a new software service.

• hive-hook: Supports outputting Hive metadata and job running information to DataWorks.

• spark-hook: Supports outputting Spark metadata and job running information to DataWorks.

Release date

EMR-3.33.0 was released on January 15, 2021.

Updates

• Supports Alluxio 2.4.0.

• Default parameter settings scale with cluster node size.

• Uses HDFS in the EMR cluster as the default UnderFS. This feature is ready to use out of the box.

• Enhanced the Alluxio OSS UnderFS to support new features such as OSS multi-versioning.

• Compatible with engines such as Hadoop, Hive, Spark, and Presto.

HUDI| Supports HUDI 0.6.0. Spark| JindoTable supports enabling or disabling the data collection feature. Hive|

• Fixed a connection pool leak issue in HiveServer.

• JindoTable supports enabling or disabling the data collection feature.

• Optimized the performance of ADD COLUMN.

• Fixed an issue where incorrect data was read from HUDI tables.

• Default parameter settings scale with cluster node size.

HDFS| Supports a larger number of snapshots. YARN| Default parameter settings scale with cluster node size. Tez| Default parameter settings scale with cluster node size. Sqoop| Fixed an issue with importing files in Avro format.

Release date

EMR-3.32.0 was released on November 23, 2020.

Updates

• Added support for Alibaba Cloud Data Lake Formation (DLF) metadata.

• Upgraded the HAS dependency to 2.0.1.

• Fixed an issue with backticks in Streaming SQL.

• Removed the Delta JAR package. Delta is now deployed separately.

• Modified the log path to write all logs to HDFS.

Hive|

• Added support for Alibaba Cloud DLF metadata.

• Resolved an issue where a DUMMY file was written when reading an empty directory in a Delta table.

• Upgraded the HAS dependency to 2.0.1.

Presto|

• Added support for Alibaba Cloud DLF metadata.

• Resolved an issue that limited the reading of Delta tables.

• Fixed an issue where the JVM configuration was missing in high-security mode.

• Upgraded the HAS dependency to 2.0.1.

HDFS|

• Added support for hot-swappable disk mode.

• Upgraded the HAS dependency to 2.0.1.

YARN|

• Fixed an issue with YARN RMZKStateStore.

• Added support for SNAPPY files output by SLS.

• Modified the directory configuration for MapReduce Local mode to resolve a directory permission check issue.

• Added support for hot-swappable disk mode.

• Set the log path to write all logs to HDFS.

• Upgraded the HAS dependency to 2.0.1.

Zookeeper|

• Added support for attaching the service port to an internal IP address at startup.

• Upgraded the HAS dependency to 2.0.1.

Flink-Vvp|

• Upgraded to version 1.11-2.2.2.

• Added support for SQL and Autopilot features.

__Note Only Dataflow clusters support Flink-Vvp. Hadoop clusters do not support Flink-Vvp at this time. Flink|

• Added support for writing to OSS in cache mode. This feature, combined with Flink Checkpoints and a resumable Source, achieves EXACTLY_ONCE semantics.

• Synchronized with Flink community version 1.11.1 features. SQL now supports multiple outputs (MULTI INSERT).

• Upgraded the HAS dependency to 2.0.1.

Impala|

• Added support for custom configurations of catalogd.flgs, impalad.flgs, and statestored.flgs.

• Upgraded Shiro to version 1.6.0.

• Upgraded the HAS dependency to 2.0.1.

Tez|

• Optimized the default memory parameters for the Application Master (AM).

• Upgraded the HAS dependency to 2.0.1.

HAS| Upgraded the HAS dependency to 2.0.1. Storm Zeppelin Ranger OpenLDAP Oozie Knox Kafka HUE HBase Druid

Release date

EMR-3.30.0 was released on October 26, 2020.

Updates

• Upgraded to version 2.7.301.

• Jindo DistCp now supports writing data to OSS with the Archive or Infrequent Access storage class.

• Enhanced the FUSE feature to support multiple namespaces.

• Improved the metadata caching feature in Cache mode.

Spark|

• Upgraded Spark to 2.4.5.2.0.

• Added support for third-party metastores.

• Added the datalake metastore-client.

Hive|

• Upgraded Hive to 2.3.5.6.0.

• Added support for third-party metastores.

• Added the datalake metastore-client.

Presto| Upgraded to version 338. Ranger|

• Upgraded the software package to 1.2.0-1.5.0.

• Added support for Presto 338.

• Added descriptions to configuration files.

Hadoop Distributed File System (HDFS)| Enabled adaptive configuration for the reserved space size of datanodes. Knox| Impala, later versions of Flink, and PAI are supported. Druid| Upgraded to version 0.18.1. SmartData| Upgraded to version 2.7.301.

Release date

EMR-3.29.0 was released on July 29, 2020.

Updates

• Upgrades to version 0.6.0.

• Decouples the Delta code from the Spark code.

Spark|

• Upgrades to version 2.4.5.

• Supports streaming-sql scripts from DataFactory.

• Supports Delta 0.6.0.

Hive| Supports Delta 0.6.0. Ranger|

• Supports custom deployments of Hadoop Distributed File System (HDFS), Hive, and Spark.

• Supports the configuration of ranger-admin-site and ranger-ugsync-site in the console.

HDFS| Now prints DataNode exception information when an HDFS write fails due to no available DataNodes (HDFS-9023). Hue|

• Supports installing the Hue component on Gateway clusters.

• Supports deploying multiple Hue instances on a single node.

DataFactory| Supports Delta 0.6.0. Druid| Upgrades to version 0.18.0. Knox|

• Upgrades to version 1.1.0-1.0.7.

• Supports the HBase UI.

New features

• Releases the first version of JindoTable, which provides hotspot statistics for tables and partitions.

• Adds support for complete storage policies in Block mode and tiered storage policies, such as Infrequent Access and Archive.

• Adds the Jindo DistCp data migration tool.

• Improves and fixes Jindo Fuse.

• Improves the integration of the JFS scheme with the Hive engine and Jindo JobCommitter in Cache mode.

• Adds a feature to set a read ratio in Block mode for reading data directly from OSS. This reduces the overhead of reading from the local cache.

• Decouples JindoFS software modules into Bigboot (control layer), Smartdata (distributed service), and the JindoFS SDK. Each module can be independently upgraded and maintained.

Release date

EMR-3.28.0 was released on June 12, 2020.

Updates

• CUBE now supports date type partition fields.

• Increased the stack depth of Spark-Submit.

Delta|

• Enhanced Data Definition Language (DDL) syntax, including commands such as CREATE, SHOW, and DESCRIBE.

• Delta now supports the Optimize syntax with Z-order.

Knox|

• Adapted for the Druid User Interface (UI).

• Multi-master deployment is supported.

Hive|

• hcatalog tables now support the magic committer.

• Removed some outdated default configurations.

Bigboot|

• Upgraded to version 2.6.3.

• Multi-master deployment is supported.

SmartData|

• Upgraded to version 2.6.3.

• Multi-master deployment is supported.

Ranger|

• Ranger now supports the Solr component.

• Ranger now supports PrestoSQL version 311.

Tez| Tez now supports setting scratchdir on OSS. Presto| Upgraded to version 331. Druid| Upgraded to version 0.17.1. Superset| Upgraded to version 0.35.2. Sqoop|

• The MySQL Java Database Connectivity (JDBC) JAR package is upgraded to version 5.1.48.

• The MySQL direct export mode supports setting a custom encoding using --mysql-charset.

New features

• Hadoop

• Spark

• Hive

• Zookeeper

• Presto

Graceful shutdown for Auto Scaling| When graceful shutdown is enabled, nodes are not released immediately. They are released after tasks are completed within a specified time period.

Release dates

Updates

• Upgraded to version 2.6.3.

• Added support for OTS metadata and Namespace HA.

SmartData Hive| HCatalog tables now support the direct committer. YARN| Changed the default committer to JindoOssCommitter. HDFS| Upgraded JindoFS-related configurations. Spark| Changed the default committer to JindoOssCommitter.

Release dates

Updates

• Initialized the RangerAdmin database for high-availability (HA) clusters.

• Fixed a security issue in the RangerUserSync startup script.

Spark|

• Added support for configuring Delta-related parameters, such as spark.sql.extensions, in the console.

• Added support for Hive to read Delta tables without setting the input format.

• Added support for the ALTER TABLE SET TBLPROPERTIES and UNSET TBLPROPERTIES statements.

Delta Hive| Fixed an issue where MapReduce (MR) task execution failed in automatic local mode. Presto|

• Upgraded to version 310.

• Upgraded the joda-time version to 2.10.5.

Tez|

• Upgraded to version 0.9.2.

• Fixed an issue where the application progress was not displayed correctly in the Tez user interface (UI).

• Fixed an issue where the application history could not be viewed in the Tez UI.

Impala| Fixed an issue where Impala could not access LZO tables. HDFS| Removed mongo-hadoop related JAR packages. Zookeeper| Upgraded to version 3.5.6. YARN| Adapted for the Tez UI. The yarn-site tab now supports adding the configuration item yarn.resourcemanager.system-metrics-publisher.enabled=true. Bigboot|

• Upgraded to version 2.2.3.

• Added support for rename operations in OSS Cache mode.

SmartData Knox| Upgraded dependency package versions. Oozie| Upgraded dependency package versions.

New features

Ranger service: Added support for Ranger Presto operations.

Release date

EMR-3.25.0 was released on January 13, 2020.

Updates

• Optimized JindoFS usage modes. The usage of Block mode is unchanged. Cache mode now supports its original usage and is also compatible with the original OSS file system usage. It supports data and metadata caching. These features can be enabled or disabled separately through configuration and are disabled by default.

• Optimized read and write performance for Block mode and Cache mode.

• Optimized disk cleanup. This provides more accurate statistics and more timely cleanup for hot data cached on local disks. It strictly ensures that disk usage does not exceed the quota.

• Improved support for Gateway clusters. Block mode and Cache mode can now be used on a Gateway.

• Supports a deployment mode where one storage cluster is separated from multiple compute clusters.

Spark|

• Added support for Delta-related parameters.

• Added support for Ranger Spark plugin configuration.

• Upgraded JindoCube to version 0.3.0.

Hive|

• Added logic for the SQL compatibility check feature.

• Released a combination of Hive 2.3.5 and Hadoop 2.8.5.

• When restarting the component, the content of hiveserver2-site.xml is no longer synchronized to hive-site.xml under spark-conf.

• Supports using the MSCK command to add incremental folders.

• Fixed a bug that occurred when Hive reused a Tez container.

• Supports using the MSCK command to optimize column-based folders.

Bigboot| Upgraded to 2.2.1. Fixed issues with native code support on some machine models. Ranger|

• Refactored the deployment method for the Spark plugin.

• Fixed a bug where header2 in an HA cluster did not obtain the keytab.

Kudu| Fixed the startup logic. Zookeeper| Added configuration for four-letter words. This is enabled by default. HDFS| Added compatibility with JindoFS. YARN|

• Changed the default value of the yarn.scheduler.capacity.node-locality-delay configuration to -1.

• Added compatibility with JindoFS.

Has| Integrated with OpenLDAP as the backend. OpenLDAP| Added compatibility with Has. Presto| Upgraded to version 0.228. Kafka| Removed D1 bad disks. Druid| Upgraded to 0.16.0. Flume| Upgraded to 1.9.0. Flink|

• Upgraded to 1.9.1.

• Supports standalone Flink clusters (released to a whitelist).

New features

• Supports SQL syntax, including ALTER, CONVERT, CREATE, CTAS, DELETE, DESC, INSERT, MERGE, OPTIMIZE, UPDATE, and VACUUM.

• Built-in and optimized the OPTIMIZE command.

• Supports the Hive connector.

• Supports other existing open-source features.

Grafana| Added as a new component for standalone Flink clusters. Version: 6.4.2. Prometheus| Added as a new component for standalone Flink clusters. Version: 2.13.0. AlertManager| Added as a new component for standalone Flink clusters. Version: 0.19.0. TensorFlow on spark|

• Supports running TensorFlow on Spark. This deeply integrates Spark with the deep learning framework. The integration includes optimized task scheduling and data exchange. It provides a complete workflow, from data pre-processing to deep learning training.

• Supports streaming tasks.

Release date

EMR-3.24.0 was released on November 18, 2019.

Updates

• Upgraded to 0.15.1.

• Added the router component.

• Upgraded fastjson.

Spark|

• Updated Spark Thrift Server to fix a class loader issue.

• Refactored Spark transaction code to improve stability.

• Fixed an issue with reading and writing files in ORC format after the built-in Hive was upgraded to version 2.3.

• Added support for the MERGE INTO syntax.

• Added support for the SCAN and STREAM syntax.

• The Structured Streaming Kafka sink now supports exactly-once semantics (EOS).

• Updated Delta Lake to 0.4.0.

Hive|

• Removed the old version of the Hive hook.

• Added an optimization to handle data skew for multiple COUNT(DISTINCT) fields.

• Fixed an issue where data was lost when joining tables with different bucket versions.

Flink| Upgraded to 1.8.2. Bigboot|

• Updated the small file tool.

• Updated the OSS JAR package to fix a non-daemon thread issue.

Kafka|

• Added support for the Deployment Set awareness feature.

• Removed the fastjson dependency.

HDFS|

• Optimized the deployment logic for the SmartData OSS JAR package.

• Updated the SmartData OSS JAR package.

Flume| Upgraded fastjson. TensorFlow on Spark| Added this service. HAS| Upgraded fastjson. Livy| Upgraded fastjson.

Release date

EMR-3.23.0 was released on September 18, 2019.

Updates

• Multiple storage modes

  • Block mode: Data is stored as blocks in the backend OSS. The local Namespace service maintains metadata. Block mode provides better metadata and data performance. Block mode supports different storage policies, including WARM (local replicas, OSS replicas), COLD (OSS replicas only), HOT (multiple local replicas, OSS replicas), TEMP (local replicas only), and ALL_HDD (multiple local replicas). The default policy is WARM. You can set different storage policies for folders based on your application scenario.

  • Cache mode: This mode is compatible with existing OSS storage methods. In Cache mode, files are stored as objects in OSS. Data and metadata for each file are cached locally based on access frequency. This improves data and metadata access performance. Cache mode provides different metadata synchronization policies to meet the needs of different scenarios.

• External client support

  • The client software development kit (SDK) lets you access the EMR JindoFS file system from outside an EMR cluster. You can use the client to access the Namespace in Block mode. However, external clients cannot use the data cache built by EMR JindoFS within the EMR cluster. This results in lower performance compared to using it within the EMR cluster.

  • Cache mode retains the original OSS storage semantics. It uses JindoFS to accelerate data caching within the EMR cluster. Therefore, you can directly access data from outside the EMR cluster using an OSS client, such as the OSS SDK or EMR OssFileSystem.

• Ecosystem component support

  • JindoFS now supports many compute engines on EMR, such as Spark, Flink, Hive, MapReduce, Impala, and Presto.

  • For scenarios that separate computing and storage, you can also store job logs in JindoFS, such as YARN Container logs and Spark Event logs.

  • JindoFS can be used as the HFile backend storage for HBase to expand its storage capacity.

OssFileSystem|

• Added logic to OssFileSystem to automatically detect bad disks. This fixes an issue where cache writes failed during OSS writes due to bad disks.

• Completed the related configurations for OssFileSystem.

Bigboot|

• Upgraded to version 2.0.0.

• Includes several major updates, such as support for multiple Namespaces, storing local data blocks as large files, multi-mode storage, and external clients.

• Fixed an issue where the Bigboot monitor status was incorrect during a machine restart.

• Added a service spec for the Kudu component.

• Added correctness checks for all service specs.

Hadoop|

• HDFS

  • Adapted for HDFS Federation. You can now create HDFS Federation clusters using custom configurations and APIs. This avoids the need for a second format operation when creating a Federation cluster.

  • Optimized the bad disk detection logic. For local disk scenarios, you can trigger bad disk detection when a DataNode block report is triggered by dfsadmin.

• YARN Fixed an issue where the MapReduce JobHistory job list did not update when MapReduce job Container logs were stored in JindoFS or OSS.

Spark|

• Relational Cache Added support for Relational Cache. Relational Cache uses pre-computation to accelerate user queries. You can create a Relational Cache to pre-compute data. When a user query is executed, the Spark Optimizer automatically finds a suitable cache, rewrites the SQL execution plan, and continues the computation based on the cached data. This improves query speed. This feature is suitable for scenarios such as reports, dashboards, data synchronization, and multidimensional analysis.

  • Use Data Definition Language (DDL) to perform operations such as CACHE, UNCACHE, ALTER, and SHOW. Cached data supports all Spark data sources and data formats.

  • Supports automatic cache data updates and updates using the REFRESH command. Supports incremental updates based on partitions.

  • Supports execution plan optimization based on Relational Cache.

• Streaming SQL

  • Standardized the parameter configuration for Stream Query Writer.

  • Optimized the schema compatibility check for Kafka data tables.

  • If a Kafka data table schema does not exist, it is automatically created in Schema Registry.

  • Optimized the log information for when a Kafka schema is incompatible.

  • Fixed an issue where column names had to be explicitly specified when writing query results to a Kafka table.

  • Removed the restriction that streaming SQL queries only support Kafka and Loghub data sources.

• Delta Added Delta. You can use Spark to create a Delta data source to support scenarios such as streaming data writes, transactional reads and writes, data validation, and data history. For more information, see Delta details.

  • Supports using the DataFrame API to read data from or write data to Delta.

  • Supports using the Structured Streaming API to read from or write to Delta as a source or sink.

  • Supports using the Delta API to perform operations such as update, delete, merge, vacuum, and optimize.

  • Supports using SQL to perform operations such as creating Delta-based tables, importing data to Delta, and reading from Delta tables.

• Others

  • Added a constraint feature that supports primary keys and foreign keys.

  • Resolved JAR file conflicts, such as for servlets.

Flink| Rollback of Log4j logs Kafka|

• Log rollback for Log4j.

• Upgraded fastjson.

Zeppelin| Upgraded the dependent commons-lang3 package to version 3.7. This fixes an issue where PySpark could not write to OSS. For more information, see Spark 2.4 incompatibility with commons-lang3 in Zeppelin. Ranger| Added support for SHOW GRANTS. Analytics-Zoo| Fixed a NumPy installation error. Impala| Now compatible with Apache Kudu 1.10.0. Presto| Upgraded to version 0.221. ZooKeeper| Upgraded to version 3.5.5.

New features

• Added Kudu as a new component. Kudu fills a gap in the Hadoop ecosystem. It provides fast data inserts and random access similar to HBase, and lets you modify data. It also provides large-scale data analytics and query capabilities similar to Hadoop Distributed File System (HDFS) or Parquet.

  • Provides C++ and Java APIs for custom development.

  • Integrates with Impala, Spark, and Hive Metastore.

• This version of Kudu is based on Apache Kudu 1.10.0.

OpenLDAP|

• Added OpenLDAP as a new component to replace ApacheDS. ApacheDS is now offline.

• Supports high availability (HA).

Release date

EMR-3.22.0 was released on July 28, 2019.