E-MapReduce:kernel release notes

Version 3.2.15-1.97

Release date: May 28, 2025

Bug fixes

• Fixed an error that could occur when parsing properties in an Elasticsearch catalog.

• Fixed an issue where the query timeout could not be correctly set during an Optimize Job.

• Fixed an issue where a CreateLoadTask operation could cause an FE node to get stuck.

• Fixed an issue where Stream Load reported a duplicate label due to an abnormal transaction status.

Version 3.2.15-1.92

Release date: March 21, 2025

New features

• Added support for the MAX_BY() and MIN_BY() window functions.

Improvements

• Added support for setting a time range during which base compaction is prohibited for a table.

• Added support for collecting statistics on Paimon tables.

• Added node and histogram metrics to Metrics.

• Added timeout parameter configurations in StarClient.

• Added support for automatic detection of damaged replicas and marking them as bad.

Bug fixes

• Fixed an issue where 0 was returned for the loadRowsRate field after SHOW ROUTINE LOAD was executed.

• Fixed an issue where the Files() function could not correctly read unqueried columns.

• Fixed an issue where the ARRAY_MAP function caused BEs to crash.

• Fixed an issue where Metadata Cache caused BEs to crash.

• Fixed an issue where Routine Load was canceled due to a transaction timeout.

• Fixed an issue where the execution plan for the nested aggregate function MAX(COUNT(DISTINCT)) was incorrect when low-cardinality optimization was enabled.

• Fixed an issue where Stream Load was redirected to non-alive nodes.

• Fixed an issue where bRPC continuously reported errors after a BE was restarted.

• Fixed an issue where Stream Load failed to be submitted using HTTP 1.0.

• Fixed an issue where the task status was inconsistent between leader and follower FEs.

• Fixed an issue where the row count in MVs was incorrect.

Version 3.2.11-1.79

Release date: November 20, 2024

Improvements

• Enabled Paimon partition-level cache by default to improve query speed.

• Improved the statistics feature for Paimon.

• Added error messages for failed queries to the audit log.

Bug fixes

• Fixed an issue where colocate tablets migrated too frequently.

• Fixed an issue where the OSS scheme could not be found when accessing Paimon tables on OSS or OSS-HDFS.

• Fixed an issue where DLF 2.0 Paimon tables could not be created.

• Fixed a partition error that occurred when writing data to Hive partitioned tables.

• Fixed an error when the array_to_bitmap function processed constant results.

Version 3.2.11-1.76

Release date: October 30, 2024

Improvements

• Improved profile collection policies and fixed potential crash issues.

• Enhanced the exception handling mechanism of the shared-data architecture to improve system fault tolerance.

Bug fixes

• Fixed an issue where incorrect results were returned when inserting data into external partitioned tables.

• Fixed an issue where abnormal FEs were caused by thread leaks.

• Fixed a crash that occurred when PK tables used the Bitmap type in a shared-data environment.

• Fixed a ToDataCacheInfo leak issue to prevent OOM errors on FEs in the shared-data architecture.

• Fixed an issue where incorrect query results were returned due to Query Cache.

• Fixed an issue where some Bucket Shuffle operations returned incorrect results.

• Fixed an issue where a NullPointerException (NPE) was generated when accessing table functions.

Version 3.2.9-1.71

Release date: September 14, 2024

New features

• Added support for writing data to Paimon tables.

• Added support for reading from and writing to DLF 2.0 Paimon catalogs.

Improvements

• Added support for the Explain Analyze feature for Paimon tables.

• Removed pruning and statistics for Paimon tables.

• Added an SQL field to Current Query.

• Improved the performance of SELECT COUNT.

• Added an interface to get migration progress.

• Iceberg DLF now supports configuring dlf.catalog.id.

Bug fixes

• Fixed a LIKE query rewrite exception.

• Fixed an issue where SHOW DATA displayed inaccurate data.

Version 3.2.9-1.67

Release date: August 16, 2024

Improvements

• Improved the performance of Hive Sink.

Bug fixes

• Fixed a permission adaptation issue between Hive Catalog and Ranger.

• Fixed a crash caused by Count optimization.

• Fixed an issue where MV refresh efficiency was low due to obtaining snapinfo.

• Fixed an issue where metadata expanded in the Compaction Manager for shared-data tables.

• Fixed an issue that caused CNs to crash.

• Fixed an issue where Prepare Statement could not be used after an upgrade to 3.2.9.

• Fixed a data inconsistency issue that occurred after data spilling was enabled.

Version 3.2.9-1.66

Release date: August 09, 2024

New features

• Added support for writing to OSS-HDFS files when the parent directory does not exist.

• Broker Load now supports Jindo.

• Added automatic identification of different engines for OSS-HDFS.

Improvements

• Improved compaction logs for more efficient diagnosis and analysis.

• The star mgr directory is now created when CNs start.

• Improved the default redirection configurations of FEs.

• Removed unnecessary log records and configuration items to improve performance.

• Added more I/O monitoring metrics for the shared-data architecture to improve operations and maintenance capabilities.

• Improved the adaptive I/O policy for the shared-data architecture.

Bug fixes

• Added support for accessing GCS and Azure data using foreign tables.

• Fixed an error that occurred when the spill result exceeded 4 GB.

• Fixed an issue where incorrect results were returned due to spill pre-aggregation.

• Fixed an issue of unexpected MV refreshes.

• Fixed a PK index memory statistics issue.

• Fixed an issue where statistics information was inaccurate.

• Fixed an issue where insert tasks could not be canceled.

• Fixed a plan error caused by a schema change.

• Fixed an issue where external tables failed to retrieve CN information.

Version 3.2.9-1.65

Release date: July 19, 2024

New features

• Upgraded the Jindo software development kit (SDK) to version 6.5.0.

• Iceberg now supports DLF Catalog.

• Shared-data clusters now support shard rebalancing.

• Added support for the regexp_split method.

Improvements

• Shortened the default wait time of get_txn_status_internal_sec from 30 seconds to 10 seconds to prevent Stream Load from being delayed in the publish phase.

• Added support for dynamic modification of the pindex_shared_data_gc_evict_interval_seconds parameter to adjust the garbage collection (GC) interval of the local persistent index for primary key tables in shared-data clusters.

Bug fixes

• Fixed an occasional Jindo CRC mismatch issue when exporting data to OSS.

• Fixed an occasional issue where a file was reported as not existing when accessing OSS-HDFS.

• Fixed an issue where an SQL syntax error was misreported as "Lost Connection".

Version 3.2.8-1.62

Release date: June 27, 2024

New features

• Unified Catalog now supports Kudu and Paimon, and supports creating DLF-type Unified Catalogs.

• Added support for generating kinship logs.

• Added support for viewing comment information of foreign tables using the DESCRIBE and SHOW CREATE TABLE commands.

Bug fixes

• Fixed an issue where data could not be written to foreign tables in shared-data clusters.

• Fixed a potential refresh failure of Paimon MVs when a partition key column contained NULL values.

• Fixed an issue where memory statistics were inaccurate.

Version 3.2.6-1.60

Release date: June 06, 2024

New features

• Added support for categorizing SQL statements that fail to be parsed as "bad sql".

• Added Kudu connector support.

• Added the enable_pipeline_engine configuration.

Bug fixes

• Fixed an issue where a cluster migration task failed to correctly obtain the source cluster schema, causing BEs to crash.

• Fixed an issue where memory statistics were inaccurate.

• Fixed a performance issue when Paimon read from read-only tables.

Version 3.2.6-1.59

Release date: May 31, 2024

New features

• Added display of "bad sql" information in Query Details for syntax parsing errors.

• Added support for delete vector queries for Paimon.

Bug fixes

• Fixed an issue where Paimon Catalog could not use custom DLF directories.

• Fixed an issue where a warehouse had to be specified when creating a Paimon Catalog.

Version 3.2.6-1.57

Release date: May 23, 2024

New features

• Added support for optimizing the bucket storage layout of internal shared-data tables using the OPTIMIZE command.

Improvements

• Added the FE configuration fragment_profile_drop_threshold_ms, which supports default configurations in the console. The default threshold for drop fragment profile is set to 0, meaning the feature is disabled by default.

• Added an independent thread pool configuration lake_flush_thread_num_per_store for refreshing internal shared-data tables. The default value is 2 * number of CPUs. This improves write I/O throughput performance.

• Improved the I/O merge policy to allow direct reading of entire small files.

• Adjusted the default values of some cross-cluster migration configurations.

Bug fixes

• Fixed an issue where a connection to the report_exec_stat RPC failed and could not be re-established, causing INSERT INTO operations and profile collection to fail.

• Fixed an issue introduced by MV Index that caused FEs to crash frequently during metadata replay.

• Fixed an AST Cache issue introduced after version 3.2.6 that prevented some MVs from being created.

Version 3.2.6-1.52

Release date: May 08, 2024

New features

• Added support for creating DLF database and table metadata.

Bug fixes

• Fixed an error in obtaining tablet metadata after the lake_tablet_internal_parallel configuration was enabled.

• Fixed a timeout issue for INSERT INTO import operations. The thrift_rpc_timeout_ms configuration is added to BEs to adjust the RPC timeout.

• Fixed an issue where a schema change did not respond for a long time.

Version 3.2.4-1.37

Release date: March 08, 2024

New features

• Added support for cross-cluster data migration.

• Primary key (PK) tables now support size-tiered compaction.