Hologres:What's new

Execution engine

• Core engine (QE v2) upgrade

  • For scenarios with ample Worker resources but sparse Shard distribution, QE v2 optimizes internal parallelization. It supports increased concurrency for CPU-intensive operators (such as JOIN and AGG) without adding Shuffle overhead.

  • Execution plans now provide more detailed operator status, including filter conditions, join conditions, redistribution columns, and filter columns.

• Intelligent query acceleration (HBO and rewrite)

  • Query Rewrite for Dynamic Table: The optimizer supports single-table query rewriting, automatically redirecting queries from base tables to pre-computed Dynamic Tables for millisecond-level responses without SQL changes. See Dynamic Table query rewrite.

  • Expanded HBO rules: Introduced adaptive join order, adaptive segment aggregation, and adaptive runtime filter.

  • Forced tuning for large queries: For queries exceeding 20 seconds with tuning potential, subsequent executions forcefully apply HBO plans. The execution plan indicates the tuning status.

  • See also: Supported functions for incremental refresh, Optimize query performance.

• Advanced operators and real-time analytics

  • Supports UserId Encoding for high-speed UV calculation by mapping high-cardinality string IDs to integer IDs and leveraging Dynamic Table incremental computation. See UV calculation with Dynamic Table and RoaringBitmaps.

  • Window TopN shifts from full sorting to streaming sorting in hash partitioning scenarios, reducing peak memory pressure.

  • TopN Filters during the Scan process can use Result Cache, reducing scan overhead.

  • Row-store table hybrid DML uses a sampling mechanism for INSERT to dynamically determine the Join strategy, improving performance for importing large data into small tables.

Lakehouse deep integration

• EXTERNAL_FILES for schema-on-read

  • The EXTERNAL_FILES function allows direct analysis of Parquet or ORC files on OSS without creating external tables, using standard SQL. It also supports writing internal table data to OSS for cold data archiving and cross-platform data exchange. See EXTERNAL_FILES function.

• MaxCompute direct read enhancements

  • Supports partition pruning and clustering key pruning when reading logic partitioned tables, reducing I/O scan volume.

  • Supports reading MaxCompute JSON data types. See Access MaxCompute through the Common Table pathway.

• Near real-time import

  • Supports near real-time import based on a temporary Stage storage, balancing write performance, resource overhead, and data visibility.

Multi-modal retrieval and analysis

• Full-text search enhancements: Supports IK, Ngram, and Pinyin tokenizers for Chinese text retrieval, fuzzy log searches, and Pinyin-based searches. See Full-text inverted index.

• Vector search enhancements: HGraph in-memory indexes support compression, saving 50% memory with only a 5% performance trade-off. HGraph supports attaching column information to index files, enabling column values to be retrieved directly from the index without querying the target table. See HGraph index guide (Recommended).

Enterprise-grade operations and system stability

• Query thread pool isolation: Query execution logic is isolated in independent thread pools to prevent interference between query loads, real-time writes, and storage engine control links.

• Adaptive Serverless enhancement: During high load, traffic is automatically diverted to the Serverless Computing resource pool. See Adaptive serverless computing.

• Virtual warehouse switching via SQL: Switch connected virtual warehouses using standard SQL commands. See Connect to a virtual warehouse.

• Rebuild enhancement: Write impact during table structure changes is reduced to under 10 seconds, and the table remains queryable during execution. See REBUILD.

• Lock wait timeout: The hg_experimental_lock_wait_timeout_ms parameter allows customizing the lock wait timeout for non-FixedQE writes.

• Fine-grained memory management: Full-text indexes load via Block Cache instead of permanent memory. Optimized batching algorithms and streaming ConcatRecordBatch support address errors caused by Arrow structures exceeding 2 GB.

Enhanced AI and retrieval capabilities

• (Beta) AI functions: LLM-powered AI functions for searching and analyzing unstructured data, including text and images. All models are fully hosted in Hologres AI nodes to ensure data security, performance, scalability, and compliance. Use cases include vector and full-text search with embedding models and Object Tables, insights from text and images, natural language filtering and classification, sentiment analysis, and document parsing for RAG processes.

• (Beta) HGraph vector search: Over 10x performance improvement. Supports hybrid search for scalar and vector data. Hybrid in-memory and on-disk indexes cut memory usage by 80% with a 5% QPS trade-off on VectorDBBench.

• (Beta) Full-text search: Inverted indexes and built-in tokenizers for keyword, phrase, and natural language search. Supports BM25 scoring for text similarity. Combines full-text search with vector or scalar data. See Full-text inverted index.

• (Beta) Global secondary index: Efficient point queries on non-primary key columns for feature stores and e-commerce platforms. See Accelerate point queries with global secondary index.

Engine enhancements

• TopN runtime filter to accelerate data queries in TopN scenarios.

• (Beta) Time Travel for internal tables to query historical data at any point within a defined period.

• (Beta) History-Based Optimization (HBO): collects execution details from slow queries, analyzes query plans, and automatically adjusts them.

Dynamic Table

• (Beta) Supports writing processed data back to Paimon in full or incremental modes. Dynamic Tables now support near real-time data processing for warehouse-to-warehouse, lake-to-warehouse, warehouse-to-lake, and lake-to-lake use cases. Combined with serverless instances, they enable ultra-low-cost data lake processing. See Dynamic tables and Hologres Serverless.

Syntax

• Supports the QUALIFY clause to filter window function results. See QUALIFY (Beta).

Function and ecosystem

• ClickHouse compatibility: supports time trunc functions such as toDayOfMonth, toDayOfYear, and toHour, improving performance by 50% compared to extract(field from timestamp). See Date and time functions.

Serverless and elasticity

• (Beta) Virtual warehouse instances deliver write isolation. Any virtual warehouse can perform batch writes without loading a table group.

• (Beta) Seamless virtual warehouse upgrades with no impact on running SQL queries and automatic client reconnection.

• (Beta) Enhanced virtual warehouse hot-update with lossless SQL execution and instant reconnections.

• Switch connected virtual warehouses in SQL. See Switch the virtual warehouse for the current connection using SQL.

• Serverless Computing can access DLF foreign tables. See Access OSS data lake acceleration using DLF, Run read and write tasks with Serverless Computing.

• See also: Architecture, Manage permissions on virtual warehouses.

Data lake analytics

• (Beta) MaxCompute data mirroring: Mirrors data from MaxCompute to Hologres with zero ETL. Performance is similar to querying Hologres internal tables.

Engine enhancements

• Supports adaptive CTE expression reuse or inlining. See CTE reuse strategy optimization.

• Optimizes BETWEEN expression calculations.

Dynamic Table

• Incremental refresh mode supports ARRAY_AGG and STRING_AGG. See ARRAY_AGG, STRING_AGG.

• DataWorks Data Map supports lineage analysis on Dynamic Tables.

Enhanced serving capabilities

• Fixed Plan supports simple expressions for high-QPS point queries and writes in more scenarios. See Accelerate SQL execution using Fixed Plan.

Function and ecosystem extensions

• Supports Lambda expressions and higher-order array functions. See LAMBDA expressions and related functions.

Serverless capabilities

• Serverless Computing supports reading and writing encrypted tables, including internal tables and MaxCompute external/foreign tables. See Serverless Computing guide.

• Enhanced Query Queue automatically routes SQL requests for certain tables to Serverless Computing. See Query Queue.

Data lake analytics capabilities

• Paimon lake table mirroring to accelerate data lake queries. See Data lake table mirroring.

• DLF 2.5 integration for metadata management, supporting Apache Paimon catalogs through DLF REST APIs. See Access a Paimon catalog using DLF.

• Apache Paimon catalogs mirroring replicates data lake data to mirrored internal tables through zero ETL.

• Time Travel for Paimon tables by specifying a timestamp or tag.

• Supports reading data from a specific or fallback branch of Paimon tables.

• Supports disabling full table scans on partitioned tables.

• TPC-H benchmark on a 1 TB dataset: 2x faster execution for Paimon table queries.

Enhanced ecosystem capabilities

• Supports trimming and compressing binary logs, reducing I/O during binary log consumption.

Dynamic Table

• Supports dynamic partitioning of logical partitioned tables, simplifying partitioned table usage.

• Added auto-refresh mode: specify data freshness, and the engine automatically optimizes the refresh strategy.

• Incremental refresh supports joins on two data streams.

• Incremental refresh supports RoaringBitmap for incremental calculations in UV and PV scenarios.

• Full refresh mode supports Adaptive Execution (Beta) with low latency, reduced OOM probability, dynamic resource estimation, and execution plan adjustments.

• See CREATE DYNAMIC TABLE, Incremental refresh.

Serverless capabilities

• Serverless Computing supports INSERT OVERWRITE, resharding, CREATE TABLE AS, stored procedures, Rebuild, and encrypted tables.

• Lossless virtual warehouse scaling (Beta) for business continuity during scaling.

• Automatic throttling (Beta) dynamically limits query queue concurrency based on workload.

• Adaptive routing automatically executes large queries using serverless resources.

• Supports configuring a daily usage limit for serverless computing resources. See Daily usage limit.

• Optimized cache reuse in high-concurrency scenarios.

• Supports RAM roles for scaling virtual warehouses.

• See also: Serverless Computing guide, Release notes for Serverless Computing, Manage virtual warehouses, Scheduled scaling (beta), Automatic throttling (Beta).

Performance optimization and query capabilities

• Restructured query engine with QEv2 and computation on lightweight encoding. TPC-H 1 TB benchmark: 33% performance improvement. See Test results.

• Engine adaptive optimization and automatic aggregation plan pushdown based on a cost model, reducing JOIN data and latency.

• Automatic NOT NULL attribute deduction for JOIN fields, pushing down NOT NULL conditions. Automatic elimination of constant fields in GROUP BY clauses.

• Query cache. See Query cache.

• Enhanced hg_stats_missing view with autovacuum_enabled and reason fields. See HG_STATS_MISSING view.

• AUTO ANALYZE optimization with improved persistence and reduced unnecessary clears from schema changes (such as RENAME, cold storage transition).

Data management and write optimization

• (Beta) Logical partitioned tables for flexible metadata and data management. See Logical partitioned table (Beta).

• (Beta) Stored generated columns for pre-computation. See Generated columns.

• (Beta) Rebuild tool for lightweight indexes (distribution key, clustering key, segment key) and table structure modifications. See REBUILD.

• Primary key tables support partial column updates in COPY operations. If a fixed frontend (FE) node is used, it does not consume the original FE connection count. See COPY.

• Native INSERT OVERWRITE syntax for regular and logical partitioned tables. See INSERT OVERWRITE.

• See also: Best practices for batch writing data to Hologres.

Function and ecosystem extensions

• Property association funnel function (finder_funnel) and dimension grouping funnel function (finder_group_funnel). See finder_funnel, finder_group_funnel.

• Functions compatible with Spark and Presto. See Spark compatible functions, Presto compatible functions.

• Roaring bitmap functions partially support 64-bit. See Roaring bitmap functions.

• Remote UDFs via Function Compute (FC). See User-defined functions (Remote UDFs).

Enterprise-level features

• Security tokens in PostgreSQL protocol connection options. RAM role login via JDBC or PSQL.

• Table recycle bin for recovering accidentally deleted tables. See Table recycle bin.

• Data masking for computation results and non-TEXT field types. See Data masking.

Data lake analytics capabilities

• External database integration with Quick BI, Tableau, and Superset. Supports metadata refresh intervals and ANALYZE/AUTO ANALYZE for external databases. See ANALYZE and auto-analyze.

• INSERT INTO writes to Paimon primary key tables and Iceberg tables.

• (Beta) MaxCompute remote querying upgraded to 2.0 using C++ Native SDK. See MaxCompute access acceleration.

• (Beta) Direct reads of MaxCompute Delta Tables. See Access MaxCompute Delta tables from Hologres.

• (Beta) Direct reads of dynamically masked MaxCompute data.

• Direct reads of MaxCompute tables after schema changes (add columns, delete columns, modify types, adjust order).

• DataWorks Data Development supports one-click mapping and import of MaxCompute data to Hologres. See Map MaxCompute metadata to Hologres, Synchronize MaxCompute data to a Hologres internal table.

Engine enhancements

• COUNT DISTINCT performance automatically optimized at runtime. See Optimize query performance.

• Row Group Filter mechanism for column-oriented tables records min/max values per row group, filtering data without reading the table.

• Runtime Filter supports multi-column joins. See Runtime Filter.

• Manual full compaction to merge small files. See Compact data files (beta).

• Range-based funnel analysis functions. See Funnel function.

• BSI (Bit-Sliced Index) extension for high-cardinality tag and join query scenarios. See BSI function, DataQ - Profiler - BSI optimization solution (Beta).

• Clustering keys support descending sort. See Clustering key.

• Optimized Infrequent Access storage caching. See Data tiered storage.

• CREATE TABLE WITH and ALTER TABLE SET replace set_table_property syntax. See CREATE TABLE.

• Batch writes to tables without primary keys use row locks instead of table locks, supporting concurrency with Fixed Plan. See Locks.

• Proxima vector processing optimized for index creation after data import. See Proxima vector computing.

• Array functions on HQE. KeyValue function. IF function. See Release notes for Hologres functions, KeyValue function, IF.

O&M and stability improvements

• EXPLAIN ANALYZE results recorded in slow query logs. See Viewing and analyzing slow query logs.

• Fixed plan diagnostics: affected_rows, result_rows, and result_bytes reported to the metadata warehouse. See Accelerate SQL execution using Fixed Plan.

• hg_relation_size function for table storage size details. See Table stored function.

• Load balancing and automatic failover with primary/secondary instances. See JDBC-based load balancing.

• New OpenAPI operations for instance management. See API overview.

Ecosystem extension

• Data lake acceleration supports Paimon format. See OSS data lake acceleration.

Behavior changes

• Data Map, data lineage, and Transmission Encryption completed Beta and are now generally available. See Data Map, Data lineage, Transmission Encryption.

• Consuming Hologres Binlog requires only read permission on the target table. See Consume Hologres binlog using JDBC.

• Bulkload imports to tables without distribution keys may have performance deterioration.

• See Changes to default behaviors.

Engine enhancements

• Runtime Filter reduces scanned data and I/O overhead, improving performance by over 20% in multi-table join scenarios. See Runtime Filter.

• Lazy Create Fragment Instance mechanism improves performance when querying large tables with row limits.

• EXPLAIN and EXPLAIN ANALYZE display format optimized. See EXPLAIN and EXPLAIN ANALYZE.

• Multiple DML statements in one transaction. See SQL transaction capabilities.

• Supports dropping columns. See Delete column (Beta).

• CREATE TABLE AS syntax. See CREATE TABLE AS.

• Streaming COPY for higher write throughput without batching. See COPY.

• Bitmap index for JSONB columns in column-oriented storage. See Columnar JSONB.

• DATE type as primary key and partition key, with optimized partition pruning for IN Array clauses exceeding the threshold (default 100). See CREATE PARTITION TABLE.

• Internal storage optimizations:

  • Tablet Lazy Open: disables memory for tables idle over 24 hours, uses LRU policy when open table count exceeds a threshold.

  • Meta tablet for schema storage management reduces memory overhead.

  • Quick recovery in repair mode with logical recovery, shortening recovery time by over 5x for tens of thousands of partitions.

• New array functions: array_max, array_min, array_contains, array_except, array_distinct, array_union. See ARRAY functions.

• HQE Table Function support framework reconstructed to support generate_series (INT, BIGINT, NUMERIC). PQE function support framework reconstructed to support functions such as left, right, text::timestamp, and timestamp::text.

• max_by and min_by aggregate functions. See MAX_BY, MIN_BY.

• See also: Release notes for Hologres functions.

O&M and stability improvements

• hg_stat_activity view with runtime diagnostic information including execution stage, engine type, resource usage, and locks. See Query the hg_stat_activity view.

• Shard-level replica for high availability, load balancing, and high throughput. See Single-instance shard-level replicas.

• Distributed Auto Analyze for foreign tables, lakehouse acceleration clusters, and incremental partitioned table data.

• Single-table storage encryption. See Table-level storage encryption settings.

• Data lineage with cross-engine lineage analysis in DataWorks. See Data lineage.

Ecosystem extension

• MaxCompute foreign table query acceleration engine upgraded.

• DLF metadata management with Multi-Catalog support for metadata isolation. See OSS data lake acceleration.

• OSS-HDFS (JindoFS) data lake acceleration. See Access data in an OSS-HDFS data lake using DLF.

• ClickHouse-compatible functions. See Import from ClickHouse.

Behavior changes

• Column store no longer supports the Segment format. Instances using Segment format cannot upgrade to V2.0. See Change the data storage format of column-oriented tables.

• Maximum total shard count limit imposed for Table Groups from V2.0. See Table Group and shard count operation guide.

• DataHub writes fully switched to JDBC mode.

• Binary log extension configured by default. WAL Senders quota increased 10x from 200 to 2,000 Slot/32C. See Consume Hologres binary logs using JDBC.

• Backup and recovery, and tiered storage completed Beta. See Local backup and recovery, Data tiered storage.

• See Default behavior changes.

Engine enhancements

• (Beta) Real-time materialized views for real-time aggregation scenarios. See Manage materialized views with SQL.

• JSONB column-oriented storage optimization for improved query statistics and data compression. See JSON and JSONB types.

• Dynamic partition management for automatic creation and deletion of partition sub-tables. See CREATE PARTITION TABLE.

• UNIQ precise deduplication function for improved efficiency and reduced memory. See Optimize query performance.

• FixedPlan INSERT writes directly to partitioned parent tables. See Accelerate SQL execution with Fixed Plan.

• Filtering with aggregation expressions (string_agg(), array_agg()).

• RowType support with row() and row_to_json(). See JSON functions.

• Schema modification. See ALTER TABLE.

• CTE Reuse operator for improved WITH expression performance.

• MaxCompute three-layer model reading (project.schema.table). See CREATE FOREIGN TABLE, IMPORT FOREIGN SCHEMA.

• MaxCompute Transactional table reads/writes, Schema Evolution table reads, and Array/Date type write-back. See Accelerate queries on MaxCompute data using a foreign table, Export to MaxCompute.

O&M and stability improvements

• Self-service configuration of shared storage secondary instances. See Multi-instance high-availability deployment with read/write splitting (shared storage).

• table_info table in the metadata warehouse. See Viewing and analysis of table statistics information.

• Continuous memory optimization.

• Automatic periodic and manual backups. See Local backup and recovery, Recover.

Ecosystem extension

• Production-grade PostGIS extension. See PostGIS spatial functions.

• Oracle extension package. See Oracle-compatible functions.

• Hudi and Delta format foreign tables via DLF. Writing CSV, Parquet, SequenceFile, and ORC data to OSS foreign tables via DLF. See OSS data lake acceleration.

• 99%+ pass rate in Tableau compatibility test (TDVT).

Behavior changes

See Default behavior change notes.

O&M improvements

• (Beta) Resource group isolation for thread-level workload isolation across users. See Single-instance computing resource isolation (Beta).

• Online hot upgrades without affecting reads and queries. See How to get more online support.

Engine enhancements

• Row-column hybrid storage for point queries and OLAP with a single data copy. See CREATE TABLE.

• (Beta) JDBC Binlog consumption for real-time data. See Consuming Hologres binary logging with JDBC.

• Dynamic Binlog configuration modification. See Subscribing to Hologres binary logging.

• Column renaming. See ALTER TABLE.

• (Beta) JSONB index for accelerated JSON queries. See JSON and JSONB types.

• Optimized metadata management with caching and compression.

Ecosystem enhancements

• OSS data in CSV, Parquet, SequenceFile, and ORC formats via DLF. See OSS data lake acceleration.

• (Beta) Cross-database queries and federated queries across Hologres instances. See Cross-database query (beta).

Security enhancements

• (Beta) Storage encryption for Hologres internal tables. See Data storage encryption.

• (Beta) Reading encrypted MaxCompute data. See Querying MaxCompute encrypted data.

Behavior changes

• Auto Analyze enabled by default.

• New MaxCompute engine enabled by default.

• Resharding function completed Beta, with updated function names.

• See Changes in default behavior.

Engine enhancements

• Automatic collection of table statistics during writes and updates. See ANALYZE and AUTO ANALYZE.

• (Beta) Shard-level multi-replica with millisecond-level switching and query retries. See Single-instance shard with multiple replicas (Beta).

• RoaringBitmap extension with native Bitmap data type. See RoaringBitmap functions.

• bit_construct and bit_match functions for user targeting. See Detailed user segmentation function.

• range_retention_count and range_retention_sum functions for retention analysis. See Funnel function.

• Built-in Resharding for modifying shard count without recreating tables. See Table Group and shard count operation guide.

• Default compression format optimized to AliORC, increasing compression ratio by 30% to 50%. See Change the data storage format of a column-oriented table.

Foreign table query features

• (Beta) MaxCompute foreign table query performance improved by 30% to 100%. See Optimizing query performance for MaxCompute foreign tables.

• (Beta) DLF integration for reading OSS data. See OSS data lake acceleration.

Performance optimization

• Point query throughput: row store increased by 100%, column store by 30%.

• Update/Delete performance improved by 30%.

• Query Plan cache optimized.

Enterprise-level O&M and security

• Slow query exposure with query status history for the last month. See Viewing and analyzing slow query logs.

Engine enhancements

• New data types: JSON, JSONB, interval, timetz, time, inet, money, name, uuid, oid, bytea, bit, varbit. See Data type summary.

• New functions: array_length, array_positions, pg_relation_size, pg_database_size. See Array functions, Table stored functions.

• Export to MaxCompute using SQL. See Export to MaxCompute.

• (Beta) Hologres Binlog subscription. See Subscribe to Hologres binary logging.

• Dynamic bitmap index and dictionary encoding modification with automatic dictionary encoding. See ALTER TABLE.

• Hologres Client Library for batch synchronization and high-QPS point queries. See Holo Client.

• Optimized JDBC write pipeline and query optimizer.

• Improved BI connectivity with Tableau Server and Superset.

Security enhancements

• STS account login via role. See RAM role authorization mode.

Engine enhancements

• CREATE VIEW for views based on internal tables, foreign tables, or other views. See VIEW.

• New data types: SERIAL, DATE, TIMESTAMP, VARCHAR(n), CHAR(n). Array type mapping for MaxCompute foreign tables. See Data type summary.

• INSERT ON CONFLICT for upsert based on primary key configuration. See INSERT ON CONFLICT (UPSERT).

• TRUNCATE support.

• (Beta) Proxima vector search engine for massive dataset vector search. See Proxima vector computation.

Security enhancements

• Data masking with configurable policies for sensitive information. See Data masking.

• CloudMonitor integration for custom metrics and one-click alerts.

MaxCompute foreign table query constraints

• Maximum partitions per scan: 512 (up from 50 in earlier versions).

• Maximum underlying data per query: 200 GB (up from 100 GB in earlier versions).

• See Constraints and limitations.