ApsaraDB for SelectDB:Kernel release notes

New features

• Supports all features of Apache Doris 3.0.4 and Apache Doris 3.0.5.

Improvements

• Improves availability by fixing an issue of premature data reclamation.

Bug fixes

• Fixes an inverted index core dump that occurs when you upgrade from V4.0.4 to V4.0.5.

New features

• Supports all features of Apache Doris 3.0.3.

• Supports the array form of the text field type for JDBC Catalog.

• Supports the timestamp field type for MaxCompute (MC) Catalog.

Improvements

• Improves query performance by resolving high CPU consumption during the Load Segment process.

• Optimizes the file cache policy to reduce fluctuations in query response time.

• Optimizes the file cache reclamation mechanism.

• Optimizes information_schema query performance.

• Optimizes the metadata lock mechanism.

• Optimizes the delete bitmap lock mechanism.

• Optimizes memory management of Compressed Class Space.

• Removes interfaces with Server-Side Request Forgery (SSRF) risks.

• Optimizes statistical performance of Audit Log.

• Optimizes performance of the window function for paged queries.

• Optimizes performance of time-series compaction for Merge-on-Write (MoW) tables.

Bug fixes

Import and load

• Fixes an issue where data import into MoW tables fails during peak hours.

• Fixes Routine Load issues, including potential data loss, jobs getting stuck, and transaction commit failures.

• Fixes an issue where Routine Load is paused and reports errors after the instance access module is restarted.

• Fixes an issue where an error may be reported when you use the Group Commit feature.

• Fixes an issue of inconsistent results between Change Data Capture (CDC) partial column updates and Stream Load wide table appends.

• Fixes an issue of data quality validation exceptions when writing to an auto-partition table using INSERT OVERWRITE.

• Fixes an issue where an error is reported when writing data using INSERT OVERWRITE.

• Fixes an issue where running INSERT OVERWRITE on the same table is not allowed.

• Fixes an issue where no error URL is provided when an INSERT INTO SELECT statement fails due to data quality issues.

• Fixes an issue of data import errors and version limit exceeded errors caused by empty partitions after a schema change.

Query

• Fixes an issue where the amount of data to be queried increases after an instance is upgraded to kernel V4.0.4.

• Fixes an issue where no result set is returned for point queries that use precompilation.

• Fixes an issue of Prepared Statement exceptions caused by stmtId overflow in high-concurrency point query scenarios.

• Fixes an issue of a RUNTIME_ERROR caused by an uninitialized state when running the ConvertTz function after upgrading to V4.0.4.

• Fixes an issue where the NOT operator does not work for short word matching.

• Fixes an issue where incorrect results are returned due to SQL Cache.

• Fixes an issue of calculation errors in the split_part function.

• Fixes an issue where the timestamp column in the window_funnel function does not support high precision.

• Fixes an issue of query planning timeouts.

• Fixes an issue where queried data does not hit a materialized view.

• Fixes an issue of query latency caused by multi-threaded waiting for spinlocks.

• Fixes an issue where short queries report a COMPUTE_GROUPS_NO_ALIVE_BE error.

Compaction and storage

• Fixes an issue where table data becomes unavailable after the disable_auto_compaction property of an MoW table is modified.

• Fixes an issue where duplicate values may appear in the primary key field of a table when the table model is Unique.

• Fixes an issue where the query response time is prolonged due to cache misses caused by the reclamation of stale rowsets.

• Fixes an issue of compaction recovery after a schema change.

Cache

• Fixes an issue where the local file cache drops sharply due to premature eviction of cached data.

Catalog and external data

• Fixes an issue of Java Database Connectivity (JDBC) connection leaks.

• Fixes an issue of connection leaks in PostgreSQL (PG) Catalog.

• Fixes an issue where partition pruning does not take effect in MaxCompute (MC) Catalog.

• Fixes an issue of metadata exceptions when reading tables across clusters using asynchronous materialized views.

Stability

• Fixes an issue of core dumps on multiple nodes in a cluster.

• Fixes an issue of slow cluster restarts.

• Fixes an issue where cluster nodes go offline abnormally due to network jitter.

• Fixes an issue of a core dump when using the explode_variant_array function.

• Fixes an issue where error log URLs are inaccessible due to excessive length.

• Fixes a compatibility issue with field naming conventions.

New features

• Supports all features of Apache Doris 3.0.2.

• Supports large transaction import. This feature is disabled by default.

Improvements

• Disables java_udf by default.

• Optimizes performance of top n queries in log scenarios.

• Optimizes point query performance using a short path.

• Optimizes tablet pruning to improve query performance.

Bug fixes

• Fixes an issue where replaying a catalog may cause the instance to fail to start in the background.

• Fixes an issue where the audit log component fails to work after the kernel is upgraded from a 3.0.x version to a 4.0.x version.

• Fixes an issue where the permissions of the default compute cluster are restricted after the kernel is upgraded to V4.0.2.

• Fixes an issue where constant folding Remote Procedure Calls (RPCs) may cross compute clusters.

• Fixes an issue where a compute cluster may not be found when an SQL statement is forwarded.

• Fixes an issue of a core dump caused by a null pointer in the compaction delete bitmap.

• Fixes an issue where a full compaction cannot be triggered because the delete bitmap of an MoW table is too large.

• Fixes an issue where a materialized view defined by select * fails to be built.

New features

• Supports all features of Apache Doris 3.0.1.

Bug fixes

• Fixes an issue where import performance may degrade if many import jobs with the same label fail.

• Fixes an issue where a cluster scale-out may cause the instance version upgrade to get stuck.

• Fixes an issue of memory leaks in the runtime filter.

New features

• Supports kernel upgrade from a 3.0.x (x >= 9) version to a 4.0.x version.

• Supports all features of Apache Doris 3.0.0.

Bug fixes

• Fixes an issue where an instance needs to be restarted in the background to restore cluster permissions after an upgrade from kernel V3.0.9.

• Fixes an issue where queries that use inverted indexes may get stuck.

New features

• Supports multi-table materialized views (MTMVs), including transparent rewrite acceleration, auto-refresh, materialized views from external to internal tables, and direct queries on materialized views. Materialized views can also be used for data warehouse layered modeling, job scheduling, and data transformation.

• Supports the Variant and IP data types and improves analytical functions for storing and analyzing complex semi-structured data.

• Supports auto-increment columns, automatic partitioning, and server-side batch request accumulation for centralized submission, improving the efficiency of real-time large-scale data writes.

• Supports a high-speed read interface based on Arrow Flight, which increases data transmission efficiency by 100x for data science and other large-scale data reading scenarios.

• Supports all features of Apache Doris 2.1.0.

Improvements

• Improves out-of-the-box blind test query performance with no tuning required. Complex SQL query performance improves by over 100% on the TPC-DS 1 TB test dataset, reaching an industry-leading level.

• Improves data lake analytics performance by 4x to 6x compared to Trino and Spark, respectively. Introduces compatibility with multiple SQL dialects, allowing you to switch from your original system to ApsaraDB for SelectDB.

Bug fixes

• Fixes issues of occasional backlogs in Routine Load and errors when importing extended partitions from upstream topics.

• Fixes issues of BE core dumps during load_segment and abnormal BE restarts.

• Fixes issues of slow export speed and data loss for Aggregate tables.

• Fixes an issue of high JVM memory usage on the master node.

• Fixes an issue of frontend (FE) out-of-memory (OOM) errors.

• Fixes an issue of duplicate warning logs caused by repeated closing of the S3 FileWriter.

Improvements

• Optimizes compaction scheduling to improve efficiency.

• Reduces high CPU utilization and HTTP interface stuttering caused by metrics spinlocks.

Bug fixes

• Fixes an issue of inconsistent execution results for the extract_url_parameter function.

• Fixes issues of MoW BE core dumps and UDF core dumps.

• Fixes issues of write failures and scheduling task failures due to connection exceptions caused by metadata service upgrade jitters.

• Fixes an issue where exporting Parquet files using OUTFILE causes multiple BE core dumps.

Improvements

• Optimizes performance for primary key tables with a high number of delete markers. This resolves failures that previously occurred during data ingestion and file merging.

• Optimizes Routine Load, including removing the 60s limit on max_batch_interval, adding observability metrics, and resolving issues where abnormal jobs affect the scheduling of other jobs.

Bug fixes

• Fixes an issue of lost audit logs.

New features

• Supports importing compressed JSON data using Stream Load.

Improvements

• Optimizes the upload object retry policy and connection timeout to improve import stability.

Bug fixes

• Fixes an issue where the CPU utilization of the instance access module may reach 100% due to high-frequency Stream Load jobs.

• Fixes an issue of memory leaks in the instance access module during the import process.

• Fixes an issue where an import fails because writing to an index file fails during the import process.

• Fixes an issue where an import may fail to upload.

• Fixes an issue where an import fails because an MoW table holds a lock for a long time.

• Fixes an issue where a schema change may cause a Base Compaction to fail.

• Fixes an issue where a Routine Load job may get stuck.

• Fixes an issue where double values of the JSON data type are processed as 0 during import.

• Fixes an issue where permissions cannot be granted on the Audit Log table.

• Fixes an issue of duplicate times in Audit Log Query.

• Fixes an issue of duplicate SQL Hash.

• Fixes an issue where UTF-8 encoding and decoding for inverted indexes may cause errors.

• Fixes an issue where cluster nodes go offline abnormally due to network jitter.

• Fixes an issue where an execution exception is triggered when an INSERT INTO SELECT query returns an empty result.

New features

• Supports all features of Apache Doris 2.0.15.

Improvements

• Improves the file cache hit rate.

Bug fixes

• Fixes an issue where query results in an Aggregate table may be incorrect.

• Fixes an issue where queries on the struct type may become unavailable.

• Fixes an issue where the window_funnel function does not support datetime v2.

• Fixes an issue where an incorrect permission error may be reported for an INSERT INTO SELECT statement.

• Fixes an issue where a Routine Load transaction cannot be correctly aborted.

• Fixes an issue where a "bad page" error is reported during a query or compaction.

New features

• Supports all features of Apache Doris 2.0.13.

Improvements

• Optimizes inconsistent mapping distribution among multiple tablets.

Bug fixes

• Fixes an issue where the new optimizer cannot automatically retry an INSERT INTO SELECT statement.

• Fixes an issue where a node restarts due to dual-write of the nullable type during a schema change.

• Fixes an issue where the disposable file cache may be occupied by other queues, causing slow compaction or schema changes.

New features

• Supports all features of Apache Doris 2.0.11 and kernel V3.0.8.

Improvements

• Optimizes the calculation of delete bitmaps for MoW table imports to improve import speed and robustness.

• Optimizes the response speed of SHOW DATA.

• Optimizes query performance of inverted indexes and semi-structured data.

Bug fixes

• Fixes an issue where an instance OOM error occurs due to an excessive number of COPY INTO tasks.

• Fixes an issue where an instance OOM error may occur due to an excessive number of alter job tasks.

• Fixes an issue where instance metrics may be missing when multiple clusters exist.

• Fixes an issue where some tablets lack a version in extreme cases of multi-tablet import within a single transaction.

• Fixes an issue where an instance cannot immediately perceive changes after the file cache size is modified.

• Fixes several issues related to JDBC external table connections.

• Fixes an issue where Elasticsearch Catalog cannot use esquery.

• Fixes issues of uneven balance and decommission getting stuck.

New features

• Supports all features of Apache Doris 2.0.10.

• Supports the SHOW PROC statement.

• Supports retrieving all JSON keys.

Improvements

• Optimizes table creation time.

• Optimizes the calculation speed of delete bitmaps for MoW tables to improve import throughput.

• Optimizes the query speed for high-frequency imports (multiple versions) of MoW tables.

• Optimizes query efficiency.

Bug fixes

• Fixes an issue of incorrect metrics after a rename cluster operation.

• Fixes an issue of inaccurate traffic statistics after a rename cluster operation.

• Fixes an issue of incorrect format in the SHOW PROCESSLIST output.

• Fixes an issue where a schema change fails due to insufficient memory.

• Fixes an issue with import permissions for the Audit Log component.

• Fixes an issue where the datetimev2 round function returns an error.

• Fixes an issue where nullable planning may cause a core dump.

• Fixes an issue where nullable planning may lead to incorrect results.

New features

• Supports all features of Apache Doris 2.0.6.

• Supports the audit log plugin. You need to manually create an audit log table.

Improvements

• Selects an available cluster (running and with permissions) in alphabetical order of cluster names when a user has not set a default cluster.

• Optimizes the availability and performance of inverted indexes and log scenarios.

• Optimizes the ambiguous projection having syntax in the new optimizer.

• Improves the performance of compound in list for inverted indexes.

Bug fixes

• Fixes an issue where many delete operations trigger an import failure.

• Fixes an issue where the logic for automatic cluster shutdown fails because statistics information logic repeatedly triggers SQL statements.

• Fixes an issue of incorrect time zone conversion when importing with a specified UTC time zone.

• Fixes an issue where SHOW PROCESSLIST reports an error from JDBC.

• Fixes an issue where the table property ttl_seconds is reset to 0 after a restart following a checkpoint.

• Fixes an issue with the date/month/year add function.

New features

• Supports all features of Apache Doris 2.0.4.

• Supports Auto Analyze.

• Supports wide table import.

• Supports Group Commit (batch import/real-time import).

• Supports in left join infer.

Improvements

• Optimizes error messages for import failures.

• Improves stability.

Bug fixes

• Fixes an issue where a cluster restart may invalidate cluster permissions.

• Fixes an issue where an upgrade from a kernel 2.4.x version to a 3.0.x version fails.

• Fixes an issue of misaligned output layout for the SHOW PROCESSLIST command.

• Fixes an issue where the alter table TTL property may not take effect (querying the table schema shows TTL as 0).

• Fixes an issue where a query occasionally reports a "failed to initialize storage reader" error.

• Fixes an issue where an abnormal path in the Parquet Reader may cause a memory leak.

• Fixes an issue with the pushdown of the date_sub, data_add, and to_date functions for external tables.

• Fixes an issue where a colocate join plan becomes invalid because colocate is involved in balance.

• Fixes an issue where the replay of an alter colocate property fails.

• Fixes an issue where data cannot be queried because a delete condition contains special characters.

• Fixes an issue where a read query has a small probability of reporting a NOT_IMPLEMENTED_ERROR.

• Fixes an issue of incorrect data due to time zone conversion during import.

• Fixes an issue where passing a negative value to substr() causes a core dump.

New features

• Supports all features of Apache Doris 2.0.3.

We recommend that you do not enable the Auto Analyze feature.

• Provides a Stream Load compatible API.

• Opens the whitelist for show proc/statistic.

Improvements

• Improves the stability of MoW table import.

• Speeds up the creation of dynamic tables.

• Improves plan generation performance.

• Optimizes the behavior of decimal operations.

• Improves stability.

Bug fixes

• Fixes an issue where insufficient OSS buffer may cause write operations to get stuck.

• Fixes an issue where a query in a multi-cluster scenario may report a "failed to find version path" error.

• Fixes an issue where a plan may cross clusters in a multi-cluster scenario.

• Fixes an issue where a new partition cannot be created for a dynamic partition table because the length of a varchar column is changed.

• Fixes an issue where the SHOW CREATE TABLE decimal data field is incompatible with SmartBI.

• Fixes an issue where the column type conversion feature may be unavailable.

• Fixes an issue where row-oriented table properties do not take effect.

• Fixes an issue where a Base Compaction may fail.

• Fixes an issue where a point query may not find the latest data.

• Fixes an issue where a timeout is required for duplicate files in COPY INTO.

• Fixes an issue where query_parallel_instance_num may be divided by zero.

• Fixes an issue where the cache hit rate calculation may be inaccurate.

• Fixes an issue where the NGram index does not take effect.

New features

• Supports the SHOW PROC current_stmt statement.

Improvements

• Enhances access control security.

• Enhances the reliability of uploaded data validation.

• Improves the availability of the instance metadata service.

• Optimizes the error message for dynamic_partition.replication_num during table creation.

Bug fixes

• Fixes an issue where an upgrade from a kernel 2.4.x version to a 3.0.x version may cause COPY INTO to fail.

• Fixes an issue where authorization for import using an external stage may fail.

• Fixes an issue where a Routine Load import may fail.

• Fixes an issue where an MoW table import may fail.

• Fixes an issue where a real-time write insert statement may fail.

• Fixes an issue of TPC-DS performance degradation.

• Fixes an issue where non-global session variables may not take effect.

• Fixes an issue where a query with many where conditions is slow to run due to high CPU consumption.

New features

• Supports Doris Clean Label.

• Makes the Stream Load cloud API compatible with the Apache Doris community behavior.

• Builds a JDBC driver into the instance.

Improvements

• Enables runtime filter pruning by default for top n queries in the new optimizer.

Bug fixes

• Fixes an issue of slow query response due to resource leaks in the instance.

• Fixes an issue where disabling a function statement does not take effect.

• Fixes an issue where a reordering schema change fails.

New features

• Supports all features of Apache Doris 2.0.2.

• Supports all features of kernel V2.3.10 and V2.4.4.

• Adds a whitelist feature for JDBC driver URLs for OSS Load.

Improvements

• Reduces the time to retrieve metadata during planning, speeding up plan generation.

• Adds the time consumed to access metadata to the query profile statistics.

• Optimizes resource utilization in a balanced manner.

Bug fixes

• Fixes an issue of read and write errors after setting the file cache TTL.

• Fixes an issue where querying the backends or rowsets table causes the service to become unavailable.

New features

• Supports all features of Apache Doris 2.0.1.

• Supports all features of kernel V2.4.3.

Improvements

• Optimizes the time consumed by Get Version.

Bug fixes

• Fixes an issue of memory leaks on several compute nodes.

Improvements

• Optimizes the balance policy.

Bug fixes

• Fixes an issue where a query fails due to replace tmp partition.

• Fixes a deadlock issue caused by the mixed use of pthread and bthread.

Improvements

• Supports all features of kernel V2.3.9 and V2.3.10.

• Improves the availability of the import feature.

Bug fixes

• Fixes an issue where a MySQL session becomes unresponsive when requesting to truncate a table that contains many partitions.

New features

• Supports all features of kernel V2.3.8.

• Supports service suspension due to overdue payments.

Bug fixes

• Fixes an issue with time zones in a Docker environment.

New features

• Supports setting constant values for the COPY INTO import method.

Improvements

• Improves the accuracy of storage information statistics.

Bug fixes

• Fixes an issue where Cloud Stream Load may fail.

New features

• Supports all features of kernel V2.3.5, V2.3.6, and V2.3.7.

• Supports importing data using Stream Load.

• Supports automatic startup of stopped pay-as-you-go clusters when they receive query requests.

Bug fixes

• Fixes an issue where a query that uses an inverted index may fail.

New features

• Supports using session variables to control whether SQL statements use the cache.

• Supports incremental data synchronization between clusters during import.

• Supports the cache data prefetch feature based on table clusters.

• Supports querying cache usage through SQL.

• Supports automatic shutdown of idle pay-as-you-go clusters.

Improvements

• Enhances data security.

• Improves the response speed when a query does not hit the cache.

Bug fixes

• Fixes an issue where a set default cluster statement may fail.

• Fixes an issue where an upgrade process may fail.

For more information about historical kernel versions of SelectDB, see Kernel Release Notes of SelectDB Cloud.