All Products
Search
Document Center

Hologres:Release notes

Last Updated:Nov 24, 2023

This topic describes the release notes for Hologres and provides links to the relevant references.

Hologres V2.1 (October 2023)

In Hologres V2.1, the following features are added or updated:

  • Features related to engines

    • The performance of executing one or more COUNT DISTINCT functions is automatically optimized to significantly improve the query efficiency. For more information, see Optimize the COUNT DISTINCT function in the "Optimize performance of queries on Hologres internal tables" topic.

    • The Row Group Filter mechanism is added to the query optimizer. For column-oriented tables, rows in a column form a row group, and the maximum and minimum values in each row group are recorded. When you query data in a column, the system filters data in each row group without reading data from the column-oriented table. This significantly decreases the query overhead and improves the query efficiency.

    • The Runtime Filter capability is optimized to support table joins based on values of multiple columns. This significantly improves the join efficiency. For more information, see Runtime Filter.

    • Full compaction can be manually triggered to merge small files. This improves the query efficiency. For more information, see Compaction (beta).

    • The range-based funnel functions are added to analyze user activity changes and compare the changes. For more information, see Funnel analysis and retention analysis functions.

    • The bit-sliced index (BSI) extension library is added to optimize the performance and usability of join queries based on user attribute tags and behavior tags and queries based on high cardinality tags. For more information, see BSI functions (beta) and BSI optimization solutions (beta).

    • Data can be sorted in descending order based on clustering keys. This helps improve the query performance. For more information, see Clustering key.

    • The cache mechanism of Infrequent Access (IA) storage is optimized to improve the query performance of IA storage. For more information, see Tiered storage of hot data and cold data.

    • The CREATE TABLE WITH and ALTER TABLE SET statements are added to replace the set_table_property function. This simplifies the process of configuring table properties. For more information, see Overview.

    • The capability of writing data to tables that are not configured with primary keys is optimized. Operations of writing data in a batch to such tables do not acquire table-level locks but acquire row-level data locks. The data locking feature can be used with the fixed plan feature. For more information, see Locks and lock troubleshooting.

    • The Proxima-based vector processing feature is optimized. It allows you to create a table, import vector data into the table, and then create vector indexes. This helps shorten the index creation time and simplify vector processing. For more information, see Vector processing based on Proxima.

    • Function capabilities are enhanced in the following aspects:

      • Hologres Query Engine (HQE) supports some array functions to improve function performance. For more information, see Function release notes.

      • The KeyValue function is added to split strings. For more information, see KeyValue function.

      • The IF function is added to simplify type detection scenarios and reduce MySQL data migration costs. For more information, see IF.

  • Features related to O&M and service stability

    • The slow query capability is enhanced to improve the efficiency of analyzing slow queries. For more information, see Query and analyze slow query logs.

      • Results of the EXPLAIN ANALYZE statement are recorded in slow query logs for you to view the execution data of each operator.

      • The fixed plan-based diagnostics capability is enhanced. Data in the rows specified by affected_rows is reported to the metadata warehouse in data write scenarios, and data in the rows specified by result_rows and result_bytes is reported to the metadata warehouse in query scenarios. For more information, see Accelerate the execution of SQL statements by using fixed plans.

    • The hg_relation_size function is added to query the storage size details of a table. For more information, see Query the storage sizes of tables and databases.

    • Hologres is compatible with native PostgreSQL behaviors and supports load balancing. Load balancing and automatic instance failover are supported in scenarios in which primary and secondary instances are configured. This improves the service availability. For more information, see JDBC-based load balancing.

    • API operations for creating, renewing, and releasing instances, and the API operation for changing instance specifications are added to improve instance O&M and management capabilities. For more information, see List of operations by function.

  • Features related to ecosystem extension

    Data can be read from Apache Paimon tables based on Data Lake Formation (DLF). For more information, see Use DLF to read data from and write data to OSS.

  • Default behavior changes

    • The public preview of the Data Map, data lineage, and SSL-encrypted transmission features is complete, and the features can be used in production environments.

    • The permissions that are required to consume Hologres binary logs are changed. After the change, only the read permissions on the desired table are required. For more information, see Use JDBC to consume Hologres binary logs.

    • If you use Bulkload to import data into a Hologres internal table that has no distribution key, the import performance may deteriorate.

    • For more information about default behavior changes, see Default behavior changes.

Hologres V2.0 (April 2023)

In Hologres V2.0, the following features are added or updated:

  • Features related to engines

    • The Runtime Filter feature is provided to optimize the filter operation in multi-table join queries. This feature helps reduce the amount of data that is scanned, decrease I/O overheads, and improve query performance by more than 20% in typical multi-table join queries. For more information, see Runtime Filter.

    • The Lazy Create Fragment Instance mechanism is added to the Hologres query engine. This mechanism helps reduce query overheads and significantly improve query performance in scenarios in which data in a large table is queried and an upper limit for the number of returned rows is configured. This mechanism is commonly used in preview scenarios.

    • The Explain and Explain Analyze statements are provided to optimize the display of execution plans. This improves the readability of execution plans and simplifies SQL performance optimization. For more information, see Explain and Explain Analyze.

    • The distributed transaction capability is optimized. Multiple DML statements can be executed in one transaction. For more information, see SQL transaction capabilities.

    • Columns can be dropped. For more information, see Drop columns (in public preview).

    • The CREATE TABLE AS statement can be executed to simplify the iterative optimization of table schemas. For more information, see CREATE TABLE AS.

    • The COPY statement can be executed to import or export data in real time. You do not need to perform batch operations. This improves the write throughput. For more information, see COPY.

    • A bitmap index can be created for columns that contain JSONB-formatted data in column-oriented storage mode. This helps accelerate point queries. For more information, see Column-oriented storage for JSONB-formatted data.

    • A column of the DATE data type can be configured as the primary key or the partition key of a partitioned table. For more information, see CREATE PARTITION TABLE. Partition pruning is optimized. After the optimization, you can still perform partition pruning even if the number of values in an array of a partition key column exceeds a threshold. The threshold is 100 by default.

    • The storage engine is optimized in the following aspects:

      • Tablet Lazy Open mechanism of the storage engine: This mechanism is optimized for both the primary and secondary instances. After the optimization, the memory overhead is automatically disabled for tables that have not been accessed for more than 24 hours. When the amount of data in an opened table exceeds a threshold, the system dynamically selects a tablet based on the least recently used (LRU) policy and disables the tablet. This reduces the resident memory overhead in scenarios in which multiple tables are opened.

      • Schema storage management mechanism of the storage engine: After the optimization, a meta tablet is used for storage management. This helps reduce the schema resident memory overhead and resource overhead in scenarios in which multiple tables and shards exist.

      • Quick restoration capability of the storage engine: Quick restoration in repair mode can be enabled if data in specific tables cannot be restored by using routine data restoration methods. By default, metadata management supports logical restoration. Logical restoration helps significantly shorten the restoration time if data in a large number of partitions needs to be restored. If data in tens of thousands of partitions needs to be restored, logical restoration shortens the restoration time by more than five times.

    • Function capabilities are enhanced in the following aspects:

      • Hologres Query Engine (HQE) supports more functions to improve function performance. For more information, see Function release notes.

        • The table function framework is reconstructed to enable HQE to support generate_series functions that contain data of the INT, BIGINT, or NUMERIC data type.

        • The PostgreSQL Query Engine (PQE) function framework is reconstructed to enable HQE to support the following functions: left, right, text::timestamp, and timestamp::text.

      • The following array functions are supported: array_max, array_min, array_contains, array_except, array_distinct, and array_union. For more information, see Array functions.

      • The max_by and min_by aggregate functions are supported to simplify window sorting. For more information, see MAX_BY and MIN_BY.

  • Features related to O&M and service stability

    • The hg_stat_activity view is introduced on the basis of the pg_stat_activity view to provide more detailed runtime information about SQL statements for diagnostics. The runtime information includes the execution stage, execution engine type, resource usage, and runtime lock. For more information, see Query the hg_stat_activity (pg_stat_activity) view.

    • The shard-level replication capability is improved to support high availability, load balance, and high throughput for a single instance. The shard-level replication feature is applicable when some servers become faulty or hot data is not evenly distributed. For more information, see Shard-level replication for high throughout.

    • The auto-analyze feature is refactored to enable distributed and automatic analysis. This way, data in foreign tables, data in tables of Shared Cluster (Lakehouse Acceleration Edition) instances, and incremental data of partitioned tables can be automatically analyzed. This helps resolve issues such as analysis failures of ultra-large tables or ultra-wide columns. The number of tables that lack statistical information is significantly reduced. The execution plan is more stable, less resources are consumed, and the performance is more stable.

    • The storage encryption configuration is optimized to support flexible single-table encryption configuration. For more information, see Encrypt data in Hologres.

    • The data lineage mechanism is optimized to allow you to use DataWorks to perform lineage analysis on data in MaxCompute and Hologres. You can use expressions such as common table expressions (CTEs) to parse data lineage. For more information, see Data lineage (Beta).

  • Features related to ecosystem extension

    • The engine for query acceleration by using MaxCompute foreign tables is upgraded to improve the compatibility and stability.

    • The multi-catalog feature of Data Lake Formation (DLF) can be used to isolate metadata in the test environment, development environment, and cross-department instances in lakehouse acceleration scenarios. For more information, see Use DLF to read data from and write data to OSS.

    • A data lake can be built based on OSS-HDFS (JindoFS) to accelerate data reads and data writes. This helps meet business requirements for data lake-based computing in the big data ecosystem such as Hadoop and AI. For more information, see Build a data lake based on OSS-HDFS.

    • Some ClickHouse functions are supported to simplify data and job migration. For more information, see Migrate data from ClickHouse to Hologres.

  • Default behavior changes

    • In Hologres V2.0 and later, data in the segment format cannot be stored in column-oriented storage mode. Hologres instances that contain data in the segment format cannot be upgraded to V2.0 or later. You can use the hg_convert_segment_orc function to convert multiple tables in the segment format into tables in the Optimized Row Columnar (ORC) format at a time. For more information, Update the data storage format of existing column-oriented tables.

    • An upper limit is imposed on the number of shards for a table group or an instance. This helps avoid resource waste caused by misuse of table groups. For more information, see User guide of table groups and shard counts.

    • Data is written to DataHub in Java Database Connectivity (JDBC) mode instead of the SDK mode. The JDBC mode is more stable than the SDK mode and supports various data types.

    • By default, the binary log extension is configured. When you consume binary log data in JDBC mode, you do not need to manually create the binary log extension. If you consume binary log data in JDBC mode, the maximum number of walsenders that can be used is increased by 10 times. For an instance that is configured with 32 CPU cores, the maximum number of walsenders is increased from 200 to 2,000. Walsenders are counted as slots. For more information, see Use JDBC to consume Hologres binary logs.

    • The public preview of the backup and restoration feature and the tiered storage feature is complete, and the features can be used in production environments.

    • For more information about default behavior changes, see Default behavior changes.

Hologres V1.3 (July 2022)

In Hologres V1.3, the following features are added or updated:

  • Features related to engines

    • The real-time materialized view feature is supported to improve query efficiency in real-time aggregation scenarios. This feature is in public preview. For more information, see Real-time materialized view.

    • Data of the JSONB type can be stored in column-oriented storage mode. This significantly accelerates queries and statistics collection and increases the efficiency of data compression. For more information, see JSON and JSONB data types.

    • Partitions in a partitioned table can be dynamically managed. This enables the system to automatically create and drop child tables. For more information, see CREATE PARTITION TABLE.

    • The UNIQ function is added for accurate deduplication. This function reduces memory consumption and improves deduplication efficiency when multiple COUNT DISTINCT operations are performed. For more information, see Optimize performance of queries on Hologres internal tables.

    • Engines are optimized in the following aspects:

    • Data can be read from the three-layer model of MaxCompute. The three layers are the project, schema, and table. For more information, see CREATE FOREIGN TABLE and IMPORT FOREIGN SCHEMA.

    • Data can be read from transactional tables and tables in the schema evolution state in MaxCompute. A table in the schema evolution state refers to a MaxCompute table with one or more columns dropped, the order of columns changed, or the data types of one or more columns changed. Hologres V1.3 also allows data of the ARRAY and DATE types to be written back to MaxCompute tables. For more information, see Create a foreign table in Hologres to accelerate queries of MaxCompute data and Export data to MaxCompute by executing SQL statements.

  • Features related to O&M and service stability

    • Configurations of secondary instances can be customized. Secondary instances share storage space with the primary instance. This provides optimized elasticity and high availability. For more information, see Configure read/write splitting for primary and secondary instances (shared storage).

    • The table_info table is provided to store table statistics. This improves the capabilities of data governance. For more information, see Query and analyze table statistics.

    • The memory usage of metadata is reduced.

    • Automatic backup on a periodic basis and manual backup are supported to help restore historical data in specific scenarios. For more information, see Backup and Restoration.

  • Features related to ecosystem extension

    • PostGIS at the production level is supported. For more information, see PostGIS for geographic information analysis.

    • Multiple Oracle functions are supported by introducing the Orafce extension. For more information, see Supported Oracle functions.

    • Data can be read from foreign tables of the Hudi and Delta formats, and data in CSV, Parquet, SequenceFile, and ORC formats can be written to OSS foreign tables by using DLF. For more information, see Use DLF to read data from and write data to OSS.

    • The compatibility of business intelligence (BI) tools are improved. The pass rate of Hologres in the compatibility test performed by using Tableau Data source Verification Tool (TDVT) is raised to over 99%.

  • Default behavior changes

    For more information about default behavior changes, see Default behavior changes.

Hologres V1.1 (October 2021)

In Hologres V1.1, the following features are added or updated:

  • Features related to O&M

    • Computing resources within a Hologres instance can be isolated by using resource groups. This feature is in public preview. You can create multiple resource groups and bind users to different resource groups to isolate computing resources at the thread level within a Hologres instance. This feature applies to the scenarios that involve multiple users or different business requirements. For more information, see Isolate computing resources in a Hologres instance (beta).

    • Online hot upgrades of Hologres instances are supported. You can read data during a hot upgrade. If you want to perform a hot upgrade on your instance, join a Hologres DingTalk group to apply for a hot upgrade. For more information, see Obtain online support for Hologres.

  • Features related to engines

    • The row-column hybrid storage model is supported. You can perform both point queries and online analytical processing (OLAP) on a table that uses the row-column hybrid storage model. For more information, see Overview.

    • Hologres binary log data can be consumed by using a JDBC driver. This feature is in public preview. For more information, see Use JDBC to consume Hologres binary logs.

    • Binary logging can be enabled and the relevant configurations can be modified based on your business requirements. For more information, see Subscribe to Hologres binary logs.

    • A table column can be renamed. For more information, see ALTER TABLE.

    • Indexes can be created for fields of the JSONB type. This feature is in public preview. You can create indexes to accelerate queries and retrieval of JSON-formatted data. For more information, see JSON and JSONB data types.

    • The management mechanism for metadata in Hologres instances is optimized. You can cache and compress metadata to manage memory in a more efficient manner.

  • Features related to foreign tables

  • Features related to security management

    • Storage encryption for data in Hologres tables is supported and the security in terms of data access is enhanced. This feature is in public preview. For more information, see Encrypt data in Hologres.

    • Encrypted data can be read from MaxCompute. This feature enables Hologres to be further integrated with MaxCompute and is in public preview. For more information, see Query MaxCompute data encrypted based on BYOK.

  • Default behavior changes

    • By default, the auto-analyze feature is enabled in Hologres V1.1.

    • By default, the new engine for accelerated access to MaxCompute is used in Hologres V1.1.

    • The resharding feature has been proven to be available in public preview. The function related to this feature is renamed.

    For more information about default behavior changes, see Default behavior changes.

Hologres V0.10 (May 2021)

In Hologres V0.10, the following features are added or updated:

  • Features related to engines

    • Table statistics can be automatically collected. Statistics can be automatically updated when data is being written to a table. This way, you can obtain a better query plan without the need to execute the ANALYZE statement. For more information, see ANALYZE and auto-analyze.

    • Point queries of key-value pairs can be performed with high reliability in milliseconds. This feature is in public preview. You can configure multiple replicas at the shard level, switch among the primary shard and follower shards in milliseconds, and retry a query in milliseconds if the query fails. This significantly improves the reliability of Hologres. For more information, see Shard-level replication for high throughout.

    • Extensions for roaring bitmaps are added, and native support is provided for the bitmap data type and relevant functions. For more information, see Roaring bitmap functions.

    • The bit_construct and bit_match functions are added and are used for user identification and attribution analysis. The functions can be used to aggregate filter conditions to filter data based on user IDs in an efficient manner. For more information, see Intended user identification functions.

    • The range_retention_count and range_retention_sum functions are added to allow you to specify a date range for user retention analysis. For more information, see Funnel analysis and retention analysis functions.

    • The resharding feature is supported. Hologres provides a built-in function for resharding that allows you to change the number of shards without the need to create tables and import data again. This simplifies the resharding process and provides optimal performance. For more information, see User guide of table groups and shard counts.

    • The AliORC data storage format is used for column-oriented tables by default. This way, the data compression ratio is raised by about 30% to 50%. For more information, see Update the data storage format of existing column-oriented tables.

  • Features related to queries of foreign tables

  • Performance optimization

    • Improved point query performance: The maximum throughput of a row-oriented table is increased by 100%, and the maximum throughput of a column-oriented table is increased by 30%.

    • Optimized SQL operations: The performance of UPDATE and DELETE operations is improved by 30%.

    • Optimized cache for query plans: The amount of time required by the optimizer is reduced.

  • Features related to enterprise-level O&M and security management

    • A query history is provided for you to view the status of all queries initiated in the last month and locate slow queries and failed queries in an efficient manner. For more information, see Query and analyze slow query logs.

Hologres V0.9 (January 2021)

In Hologres V0.9, the following features are added or updated:

  • Features related to engines

    • Data types of various categories are supported.

      • JSON and JSONB. For more information, see JSON and JSONB data types.

      • Time: INTERVAL, TIMETZ, and TIME.

      • Network: INET.

      • Currency: MONEY.

      • PostgreSQL system: NAME, UUID, and OID.

      • Others: BYTEA, BIT, and VARBIT.

      For more information, see Data types.

    • Various functions including extension functions and PostgreSQL functions are supported.

    • Data can be exported from Hologres to MaxCompute by using SQL statements. This facilitates data archiving. For more information, see Export data to MaxCompute by executing SQL statements.

    • Hologres binary logs can be subscribed. This feature is in public preview. For more information, see Subscribe to Hologres binary logs.

    • The bitmap index and dictionary encoding properties of a field can be dynamically modified, and dictionary encoding can be automatically enabled for a field based on the field value characteristics. For more information, see ALTER TABLE.

    • Hologres Client Library (Holo Client for short) is released. Holo Client allows you to perform point queries with high queries per second (QPS). You can use Holo Client to synchronize data to Hologres in real time or in offline mode. Holo Client automatically batch collects data to improve the throughput. For more information, see Use Holo Client to read and write data.

    • The query optimizer and the JDBC link that is used to write data to Hologres are optimized. The efficiency of data writing is improved.

    • Users can connect to more BI tools, such as Tableau Server and Apache Superset. This can meet the requirements for different types of business analysis.

  • Features related to security management

    • Hologres can be accessed by using a role that is assigned by Security Token Service (STS). Role-based single sign-on (SSO) is a more secure logon method that helps protect your Alibaba Cloud account. For more information, see RAM role authorization mode.

Hologres V0.8 (October 2020)

In Hologres V0.8, the following features are added or updated:

  • Features related to engines

    • The CREATE VIEW statement is supported. You can execute this statement to create a view based on one or more Hologres tables or foreign tables, or an existing view. For more information, see VIEW.

    • The SERIAL, DATE, TIMESTAMP, VARCHAR(n), and CHAR(n) data types are supported. Hologres also supports the mapping of arrays between MaxCompute and Hologres when you create a foreign table. For more information, see Data types.

    • The INSERT ON CONFLICT statement is supported. This statement allows you to specify the policy that is used to deal with conflicts when you insert a row that contains the same primary key value as an existing row. For more information, see INSERT ON CONFLICT(UPSERT).

    • The TRUNCATE statement is supported.

    • The Proxima vector search engine is supported. Proxima allows you to query a large number of vectors at a time. Proxima is in public preview. For more information, see Vector processing based on Proxima.

  • Features related to security management

    • The feature of masking sensitive data is added. You can configure custom masking policies to mask your sensitive data, such as phone numbers, addresses, and resident identity card numbers. For more information, see Mask data.

    • CloudMonitor is integrated. You can configure custom metrics and alert rules on Hologres by using CloudMonitor.

  • Features related to queries of MaxCompute tables

    • A maximum of 512 partitions can be scanned during a single query. In Hologres versions earlier than V0.8, a maximum of 50 partitions can be scanned during a single query.

    • A maximum of 200 GB of data can be scanned during a single query, regardless of the number of foreign tables and fields. In Hologres versions earlier than V0.8, a maximum of 100 GB of data can be scanned during a single query.

    For more information, see Limits.