Hologres:Release notes

In Hologres V2.0, the following features are added or updated:

• Features related to engines

  • The Runtime Filter feature is provided to optimize the filter operation in multi-table join queries. This feature helps reduce the amount of data that is scanned, decrease I/O overheads, and improve query performance by more than 20% in typical multi-table join queries. For more information, see Runtime Filter.

  • The Lazy Create Fragment Instance mechanism is added to the Hologres query engine. This mechanism helps reduce query overheads and significantly improve query performance in scenarios in which data in a large table is queried and an upper limit for the number of returned rows is configured. This mechanism is commonly used in preview scenarios.

  • The EXPLAIN and EXPLAIN ANALYZE statements are provided to optimize the display of execution plans. This improves the readability of execution plans and simplifies SQL performance optimization. For more information, see EXPLAIN and EXPLAIN ANALYZE.

  • The distributed transaction capability is optimized. Multiple DML statements can be executed in one transaction. For more information, see SQL transaction capabilities.

  • Columns can be dropped. For more information, see Drop columns (in public preview).

  • The CREATE TABLE AS statement can be executed to simplify the iterative optimization of table schemas. For more information, see CREATE TABLE AS.

  • The COPY statement can be executed to import or export data in real time. You do not need to perform batch operations. This improves the write throughput. For more information, see COPY.

  • A bitmap index can be created for columns that contain JSONB-formatted data in column-oriented storage mode. This helps accelerate point queries. For more information, see Column-oriented storage for JSONB-formatted data.

  • A column of the DATE data type can be configured as the primary key or the partition key of a partitioned table. For more information, see CREATE PARTITION TABLE. Partition pruning is optimized. After the optimization, you can still perform partition pruning even if the number of values in an array of a partition key column exceeds a threshold. The threshold is 100 by default.

  • The storage engine is optimized in the following aspects:

    • Tablet Lazy Open mechanism of the storage engine: This mechanism is optimized for both the primary and secondary instances. After the optimization, the memory overhead is automatically disabled for tables that have not been accessed for more than 24 hours. When the amount of data in an opened table exceeds a threshold, the system dynamically selects a tablet based on the least recently used (LRU) policy and disables the tablet. This reduces the resident memory overhead in scenarios in which multiple tables are opened.

    • Schema storage management mechanism of the storage engine: After the optimization, a meta tablet is used for storage management. This helps reduce the schema resident memory overhead and resource overhead in scenarios in which multiple tables and shards exist.

    • Quick restoration capability of the storage engine: Quick restoration in repair mode can be enabled if data in specific tables cannot be restored by using routine data restoration methods. By default, metadata management supports logical restoration. Logical restoration helps significantly shorten the restoration time if data in a large number of partitions needs to be restored. If data in tens of thousands of partitions needs to be restored, logical restoration shortens the restoration time by more than five times.

  • Function capabilities are enhanced in the following aspects:

    • HQE supports more functions to improve function performance. For more information, see Function release notes.

      • The table function framework is reconstructed to enable HQE to support generate_series functions that contain data of the INT, BIGINT, or NUMERIC data type.

      • The PostgreSQL Query Engine (PQE) function framework is reconstructed to enable HQE to support the following functions: left, right, text::timestamp, and timestamp::text.

    • The following array functions are supported: array_max, array_min, array_contains, array_except, array_distinct, and array_union. For more information, see Array functions.

    • The max_by and min_by aggregate functions are supported to simplify window sorting. For more information, see MAX_BY and MIN_BY.

• Features related to O&M and service stability

  • The hg_stat_activity view is introduced on the basis of the pg_stat_activity view to provide more detailed runtime information about SQL statements for diagnostics. The runtime information includes the execution stage, execution engine type, resource usage, and runtime lock. For more information, see Query the hg_stat_activity (pg_stat_activity) view.

  • The shard-level replication capability is improved to support high availability, load balance, and high throughput for a single instance. The shard-level replication feature is applicable when some servers become faulty or hot data is not evenly distributed. For more information, see Shard-level replication for a Hologres instance.

  • The auto-analyze feature is refactored to enable distributed and automatic analysis. This way, data in foreign tables, data in tables of Shared Cluster (Lakehouse Acceleration Edition) instances, and incremental data of partitioned tables can be automatically analyzed. This helps resolve issues such as analysis failures of ultra-large tables or ultra-wide columns. The number of tables that lack statistical information is significantly reduced. The execution plan is more stable, less resources are consumed, and the performance is more stable.

  • The storage encryption configuration is optimized to support flexible single-table encryption configuration. For more information, see Encrypt data in Hologres.

  • The data lineage mechanism is optimized to allow you to use DataWorks to perform lineage analysis on data in MaxCompute and Hologres. You can use expressions such as common table expressions (CTEs) to parse data lineage. For more information, see Data lineage.

• Features related to ecosystem extension

  • The engine for query acceleration by using MaxCompute foreign tables is upgraded to improve the compatibility and stability.

  • The multi-catalog feature of Data Lake Formation (DLF) can be used to isolate metadata in the test environment, development environment, and cross-department instances in lakehouse acceleration scenarios. For more information, see Use DLF to read data from and write data to OSS.

  • A data lake can be built based on OSS-HDFS (JindoFS) to accelerate data reads and data writes. This helps meet business requirements for data lake-based computing in the big data ecosystem such as Hadoop and AI. For more information, see Use DLF to access data in OSS-HDFS.

  • Some ClickHouse functions are supported to simplify data and job migration. For more information, see Import data from ClickHouse.

• Default behavior changes

  • In Hologres V2.0 and later, data in the segment format cannot be stored in column-oriented storage mode. Hologres instances that contain data in the segment format cannot be upgraded to V2.0 or later. You can use the hg_convert_segment_orc function to convert multiple tables in the segment format into tables in the Optimized Row Columnar (ORC) format at a time. For more information, Update the data storage format of existing column-oriented tables.

  • An upper limit is imposed on the number of shards for a table group or an instance. This helps avoid resource waste caused by misuse of table groups. For more information, see User guide of table groups and shard counts.

  • Data is written to DataHub in Java Database Connectivity (JDBC) mode instead of the SDK mode. The JDBC mode is more stable than the SDK mode and supports various data types.

  • By default, the binary log extension is configured. When you consume binary log data in JDBC mode, you do not need to manually create the binary log extension. If you consume binary log data in JDBC mode, the maximum number of walsenders that can be used is increased by 10 times. For an instance that is configured with 32 CPU cores, the maximum number of walsenders is increased from 200 to 2,000. Walsenders are counted as slots. For more information, see Use JDBC to consume Hologres binary logs.

  • The public preview of the backup and restoration feature and the tiered storage feature is complete, and the features can be used in production environments.

  • For more information about default behavior changes, see Default behavior changes.

In Hologres V1.3, the following features are added or updated:

• Features related to engines

  • The real-time materialized view feature is supported to improve query efficiency in real-time aggregation scenarios. For more information, see Manage materialized views by using SQL statements.

  • Data of the JSONB type can be stored in column-oriented storage mode. This significantly accelerates queries and statistics collection and increases the efficiency of data compression. For more information, see JSON and JSONB data types.

  • Partitions in a partitioned table can be dynamically managed. This enables the system to automatically create and drop child tables. For more information, see CREATE PARTITION TABLE.

  • The UNIQ function is added for accurate deduplication. This function reduces memory consumption and improves deduplication efficiency when multiple COUNT DISTINCT operations are performed. For more information, see Optimize query performance.

  • Engines are optimized in the following aspects:

    • The INSERT statements that are supported by the fixed plan feature can be executed to write data to parent tables. For more information, see INSERT and Accelerate the execution of SQL statements by using fixed plans.

    • The string_agg() or array_agg() function can be used to concatenate input values into a string or array. The FILTER clause can be used in the function expression. For more information, see String functions.

    • The row type attribute and related functions such as row() and row_to_json() are supported. For more information, see JSON functions.

    • The schema to which a table belongs can be changed. For more information, see ALTER TABLE.

    • The reuse operator of CTEs is supported. This improves the efficiency of queries based on WITH clauses. For more information, see Optimize query performance.

  • Data can be read from MaxCompute projects that use the three-layer model. The three layers are the project, schema, and table. For more information, see CREATE FOREIGN TABLE and IMPORT FOREIGN SCHEMA.

  • Data can be read from transactional tables and tables in the schema evolution state in MaxCompute. A MaxCompute table is in the schema evolution state if one or more columns are dropped, the order of columns is changed, or the data types of one or more columns are changed. Hologres also allows data of the ARRAY and DATE types to be written back to MaxCompute tables. For more information, see Use foreign tables in Hologres to accelerate queries on MaxCompute data and Export data to MaxCompute.

• Features related to O&M and service stability

  • Configurations of secondary instances can be customized. Secondary instances share storage space with the primary instance. This provides optimized elasticity and high availability. For more information, see Configure read/write splitting for primary and secondary instances (shared storage).

  • The table_info table is provided to store table statistics. This improves the capabilities of data governance. For more information, see Query and analyze table statistics.

  • The memory usage of metadata is reduced.

  • Automatic backup on a periodic basis and manual backup are supported to help restore historical data in specific scenarios. For more information, see Local backup and Restoration.

• Features related to ecosystem extension

  • PostGIS at the production level is supported. For more information, see PostGIS for geographic information analysis.

  • Multiple Oracle functions are supported by introducing the Orafce extension. For more information, see Supported Oracle functions.

  • Data can be read from foreign tables of the Hudi and Delta formats, and data in CSV, Parquet, SequenceFile, and ORC formats can be written to OSS foreign tables by using DLF. For more information, see Use DLF to read data from and write data to OSS.

  • The compatibility of business intelligence (BI) tools are improved. The pass rate of Hologres in the compatibility test performed by using Tableau Data source Verification Tool (TDVT) is raised to over 99%.

• Default behavior changes

  For more information about default behavior changes, see Default behavior changes.

In Hologres V1.1, the following features are added or updated:

• Features related to O&M

  • Computing resources within a Hologres instance can be isolated by using resource groups. This feature is in public preview. You can create multiple resource groups and bind users to different resource groups to isolate computing resources at the thread level within a Hologres instance. This feature applies to the scenarios that involve multiple users or different business requirements. For more information, see Isolate computing resources in a Hologres instance (beta).

  • Online hot upgrades of Hologres instances are supported. You can read data during a hot upgrade. If you want to perform a hot upgrade on your instance, join a Hologres DingTalk group to apply for a hot upgrade. For more information, see Obtain online support for Hologres.

• Features related to engines

  • The row-column hybrid storage mode is supported. You can perform both point queries and online analytical processing (OLAP) on a table that uses the row-column hybrid storage mode. For more information, see CREATE TABLE.

  • Hologres binary log data can be consumed by using a JDBC driver. This feature is in public preview. For more information, see Use JDBC to consume Hologres binary logs.

  • Binary logging can be enabled and the relevant configurations can be modified based on your business requirements. For more information, see Subscribe to Hologres binary logs.

  • A table column can be renamed. For more information, see ALTER TABLE.

  • Indexes can be created for fields of the JSONB type. This feature is in public preview. You can create indexes to accelerate queries and retrieval of JSON-formatted data. For more information, see JSON and JSONB data types.

  • The management mechanism for metadata in Hologres instances is optimized. You can cache and compress metadata to manage memory in a more efficient manner.

• Features related to foreign tables

  • Data in CSV, Parquet, SequenceFile, and ORC formats can be read from OSS by using DLF. For more information, see Use DLF to read data from and write data to OSS.

  • Data queries across Hologres databases and federated queries across Hologres instances are supported. For more information, see Query data across databases (beta).

• Features related to security management

  • Storage encryption for data in Hologres tables is supported and the security in terms of data access is enhanced. This feature is in public preview. For more information, see Encrypt data in Hologres.

  • Encrypted data can be read from MaxCompute. This feature enables Hologres to be further integrated with MaxCompute and is in public preview. For more information, see Query MaxCompute data encrypted based on BYOK.

• Default behavior changes

  • By default, the auto-analyze feature is enabled in Hologres V1.1.

  • By default, the new engine for accelerated access to MaxCompute is used in Hologres V1.1.

  • The resharding feature has been proven to be available in public preview. The function related to this feature is renamed.

  For more information about default behavior changes, see Default behavior changes.

In Hologres V0.10, the following features are added or updated:

• Features related to engines

  • Table statistics can be automatically collected. Statistics can be automatically updated when data is being written to a table. This way, you can obtain a better query plan without the need to execute the ANALYZE statement. For more information, see ANALYZE and auto-analyze.

  • Point queries of key-value pairs can be performed with high reliability in milliseconds. This feature is in public preview. You can configure multiple replicas at the shard level, switch among the primary shard and follower shards in milliseconds, and retry a query in milliseconds if the query fails. This significantly improves the reliability of Hologres. For more information, see Shard-level replication for high throughout.

  • Extensions for roaring bitmaps are added, and native support is provided for the bitmap data type and relevant functions. For more information, see Roaring bitmap functions.

  • The bit_construct and bit_match functions are added and are used for user identification and attribution analysis. The functions can be used to aggregate filter conditions to filter data based on user IDs in an efficient manner. For more information, see Intended user identification functions.

  • The range_retention_count and range_retention_sum functions are added to allow you to specify a date range for user retention analysis. For more information, see Funnel functions.

  • The resharding feature is supported. Hologres provides a built-in function for resharding that allows you to change the number of shards without the need to create tables and import data again. This simplifies the resharding process and provides optimal performance. For more information, see User guide of table groups and shard counts.

  • The AliORC data storage format is used for column-oriented tables by default. This way, the data compression ratio is raised by about 30% to 50%. For more information, see Update the data storage format of existing column-oriented tables.

• Features related to queries of foreign tables

  • A new engine is adopted for accelerated access to MaxCompute data. Compared with earlier versions, Hologres V0.10 improves query performance by about 30% to 100%. This feature is in public preview. For more information, see Optimize the performance of querying MaxCompute tables in Hologres.

  • DLF is integrated to allow you to read OSS data in Hologres. This feature is in public preview. For more information, see Use DLF to read data from and write data to OSS.

• Performance optimization

  • Improved point query performance: The maximum throughput of a row-oriented table is increased by 100%, and the maximum throughput of a column-oriented table is increased by 30%.

  • Optimized SQL operations: The performance of UPDATE and DELETE operations is improved by 30%.

  • Optimized cache for query plans: The amount of time required by the optimizer is reduced.

• Features related to enterprise-level O&M and security management

  • A query history is provided for you to view the status of all queries initiated in the last month and locate slow queries and failed queries in an efficient manner. For more information, see Query and analyze slow query logs.

In Hologres V0.9, the following features are added or updated:

• Features related to engines

  • Data types of various categories are supported.

    • JSON and JSONB. For more information, see JSON and JSONB data types.

    • Time: INTERVAL, TIMETZ, and TIME.

    • Network: INET.

    • Currency: MONEY.

    • PostgreSQL system: NAME, UUID, and OID.

    • Others: BYTEA, BIT, and VARBIT.

    For more information, see Data types.

  • Various functions including extension functions and PostgreSQL functions are supported.

    • Array functions: array_length and array_positions. For more information, see Array functions.

    • Functions that you can use to query the storage size of tables or databases: pg_relation_size and pg_database_size. For more information, see Table storage functions.

  • Data can be exported from Hologres to MaxCompute by using SQL statements. This facilitates data archiving. For more information, see Export data to MaxCompute.

  • Hologres binary logs can be subscribed. This feature is in public preview. For more information, see Subscribe to Hologres binary logs.

  • The bitmap index and dictionary encoding properties of a field can be dynamically modified, and dictionary encoding can be automatically enabled for a field based on the field value characteristics. For more information, see ALTER TABLE.

  • Hologres Client Library (Holo Client for short) is released. Holo Client allows you to perform point queries with high queries per second (QPS). You can use Holo Client to synchronize data to Hologres in real time or in offline mode. Holo Client automatically batch collects data to improve the throughput. For more information, see Use Holo Client to read and write data.

  • The query optimizer and the JDBC link that is used to write data to Hologres are optimized. The efficiency of data writing is improved.

  • Users can connect to more BI tools, such as Tableau Server and Apache Superset. This can meet the requirements for different types of business analysis.

• Features related to security management

  • Hologres can be accessed by using a role that is assigned by Security Token Service (STS). Role-based single sign-on (SSO) is a more secure logon method that helps protect your Alibaba Cloud account. For more information, see RAM role authorization mode.

In Hologres V0.8, the following features are added or updated:

• Features related to engines

  • The CREATE VIEW statement is supported. You can execute this statement to create a view based on one or more Hologres tables or foreign tables, or an existing view. For more information, see VIEW.

  • The SERIAL, DATE, TIMESTAMP, VARCHAR(n), and CHAR(n) data types are supported. Hologres also supports the mapping of arrays between MaxCompute and Hologres when you create a foreign table. For more information, see Data types.

  • The INSERT ON CONFLICT statement is supported. This statement allows you to specify the policy that is used to deal with conflicts when you insert a row that contains the same primary key value as an existing row. For more information, see INSERT ON CONFLICT(UPSERT).

  • The TRUNCATE statement is supported.

  • The Proxima vector search engine is supported. Proxima allows you to query a large number of vectors at a time. Proxima is in public preview. For more information, see Vector processing based on Proxima.

• Features related to security management

  • The feature of masking sensitive data is added. You can configure custom masking policies to mask your sensitive data, such as phone numbers, addresses, and resident identity card numbers. For more information, see Mask data.

  • CloudMonitor is integrated. You can configure custom metrics and alert rules on Hologres by using CloudMonitor.

• Features related to queries of MaxCompute tables

  • A maximum of 512 partitions can be scanned during a single query. In Hologres versions earlier than V0.8, a maximum of 50 partitions can be scanned during a single query.

  • A maximum of 200 GB of data can be scanned during a single query, regardless of the number of foreign tables and fields. In Hologres versions earlier than V0.8, a maximum of 100 GB of data can be scanned during a single query.

  For more information, see Limits.