ApsaraDB for ClickHouse:Release notes

Feature

Description

References

Community-compatible Edition V23.8-1.3.2

The following issue is fixed: ReplicatedMergeTree tables can only be read due to abnormal restarts.

None

Community-compatible Edition V23.8-1.3.1

• Community-compatible Edition V23.8-1.3.1 is compatible with ClickHouse V23.8.16.40.

• By default, text_log is enabled. Logs whose security levels are Warning, Error, and Fatal are recorded. You can query the logs by using system.text_log.

• query_thread_log is disabled by default to resolve the issue that a large amount of disk space is occupied.

• The following issue is fixed: The data distribution of distributed tables is blocked due to unexpected shutdowns.

Comparing changes

Enterprise Edition V24.2.2.16476

• Performance optimization

  • The response interval for resource scaling of serverless clusters is shortened to prevent queries from being interrupted due to delayed scaling.

  • Some parameters are added to ensure that the vertical merge algorithm can be used in some scenarios to improve the merge performance.

  • You can create only table engines that do not store data on local disks when you use the Replicated database engines. This avoids various issues caused by incorrect configurations and operations.

  • The digest_enabled parameter of Keeper is set to 1 to enable real-time data consistency check.

  • The async_replication parameter of Keeper is set to 1 to improve the performance of data replication across multiple nodes when read/write consistency is ensured.

  • The queries per second (QPS) and bandwidth required to access Object Storage Service (OSS) are reduced for the same business load.

• Fixed issues

  • The following issue is fixed: When a session is closed by Keeper, subsequent connections may fail.

  • The following issue is fixed: SQL queries terminate prematurely due to inaccurate memory statistics.

  • The check_not_exists and create_if_not_exists parameters of Keeper are set to 1 to avoid unexpected errors that occur when the ClickHouse server performs operations.

Release details of ApsaraDB for ClickHouse Enterprise Edition V24.2.2.16476

Feature

Description

References

Community-compatible Edition V22.8-1.18.0

• The Kafka engine crash issue is fixed.

• The following issue is fixed: The data distribution of distributed tables is blocked due to unexpected shutdowns.

None

Feature

Description

References

Enterprise Edition V24.2.2.16399

• New features

  • The system.dns_cache table is supported to debug Domain Name System (DNS) issues.

  • The mergeTreeIndex, generate_series, and toMillisecond functions are supported.

  • New syntax

    • A syntax is supported to directly grant specific users query and write permissions on views and materialized views without the need for explicit authorization on the underlying tables.

    • ATTACH PARTITION ALL is supported.

    • Enumerations are allowed to be declared in the structure of an external table.

• Changes

  • Ordinary databases cannot be created. Existing Ordinary databases can still be used.

  • SHOW INDEX | INDEXES | INDICES | KEYS no longer sorts results by the primary key columns.

• Optimization

  • Stability optimization

    • The ApsaraDB for ClickHouse server and ApsaraDB for ClickHouse Keeper dynamically adjust the memory soft limit based on the compute capacity units (CCUs). This removes the limitation of the small maximum memory supported by Keeper and prevents out-of-memory (OOM) issues to some extent.

    • ApsaraDB for ClickHouse Keeper has added a retry mechanism for failed disk-related operations.

    • ApsaraDB for ClickHouse Keeper will abort the startup process if invalid snapshots are detected during service startup. This can prevent data loss.

    • text_log is enabled to help you troubleshoot issues.

  • Performance optimization

    • The size of FileSystemCache can be dynamically adjusted based on the CCUs. In previous versions, the size was fixed to 100 GB.

    • The default compression algorithm changes from LZ4HC(3) to LZ4HC(2), which improves the query efficiency at the expense of less compression.

    • A certain number of logs are cached in the memory to improve the Keeper performance.

  • Serverless optimization

    • The uncompressed cache is disabled to reduce memory usage of the server for elastic scaling.

    • The setting logic of the mark cache size is optimized to reduce the memory usage in some cases for elastic scaling.

    • More aggressive serverless policies are supported. This feature is in invitational preview.

• Fixed issues

  • The following issue is fixed: Queries are unexpectedly terminated due to memory statistics errors.

  • The following issue is fixed: An exception is reported when you execute the first DDL statement after your cluster is restarted.

  • The following issue is fixed: Insertion is interrupted due to the OSS QPS limit.

Release details of ApsaraDB for ClickHouse Enterprise Edition V24.2.2.16399

Feature

Description

References

Community-compatible Edition V23.8

• New features

  • The S3Queue table engine can be used to consume data from Amazon Simple Storage Service (Amazon S3) in streaming mode.

  • Compressed files in the ZIP, TAR, and 7ZIP formats can be directly imported to ApsaraDB for ClickHouse.

  • TRUNCATE DATABASE is supported.

  • The direct vector addition and subtraction operations are supported.

  • Tuple concatenation is supported.

  • The suggestions for correcting the database names or table names are added to error messages.

• Performance optimization

  The performance of querying Parquet, ORC, TSV, CSV, and JSON files has been improved by an average of 40%. Some queries experience performance increases of 4 to 5 times.

• ClickHouse Release 23.8

• S3Queue Table Engine

• 2023 Changelog

• TRUNCATE DATABASE

• ClickHouse: Release 23.8 Webinar

Feature

Description

References

Community-compatible Edition V22.8

• Optimization

  • When you import data from an external table in MaxCompute, exclusive Tunnel and data compression during network transmission are supported.

  • By default, text_log is enabled. Logs whose security levels are Warning, Error, and Fatal are recorded. You can query the logs by using system.text_log.

  • query_thread_log is disabled by default to resolve the issue that a large amount of disk space is occupied.

• Fixed issues

  • The issues of data type conversions and value comparisons between DateTime64 and Datetime or between DateTime64 and Date are fixed.

  • The following issue is fixed: "Invalid number of rows in Chunk".

Import from MaxCompute

Community-compatible Edition V21.8

• New features

  A fallback session lifetime mechanism is introduced to connect ApsaraDB for ClickHouse to ZooKeeper. This mechanism balances loads among ZooKeeper nodes.

• Fixed issues

  The OOM issue caused by index_granularity=0 is fixed.

None

Feature

Description

References

New available region

New features

ApsaraDB for ClickHouse clusters are available in Malaysia (Kuala Lumpur).

• Create an ApsaraDB for ClickHouse cluster

• Fees of compute nodes in ApsaraDB for ClickHouse clusters

• Storage fees of ApsaraDB for ClickHouse clusters

• Fees for backup space in ApsaraDB for ClickHouse Community-compatible Edition clusters

Community-compatible Edition V22.8

Fixed issues

The following issue is fixed: The query result of the Date function is abnormal.

None

Specification changes

The C4 and S4 specifications are discontinued.

None

Feature

Description

References

Data backup disabling

New features

The data backup feature can be disabled for an ApsaraDB for ClickHouse cluster.

Data backup

Feature

Description

References

Scheduled restart of clusters

New features

An ApsaraDB for ClickHouse cluster can be restarted at a scheduled time.

Restart an ApsaraDB for ClickHouse cluster

Tag management

New features

ApsaraDB for ClickHouse allows you to add tags to an ApsaraDB for ClickHouse cluster.

Tag management

ZooKeeper configurations

New features

By default, ApsaraDB for ClickHouse clusters provide ZooKeeper with 4 cores and 8 GB of memory for free.

None

Feature

Description

References

Monitoring and alerting

New features

View monitoring data in the ApsaraDB for ClickHouse console

Upgrade of the major engine version

New features

The major engine version of ApsaraDB for ClickHouse clusters can be upgraded by migrating data between the clusters.

Upgrade the major engine version by migrating data

Feature

Description

References

Release and mounting of a Classic Load Balancer (CLB) instance

New features

A CLB instance can be released and mounted for an ApsaraDB for ClickHouse cluster that is created after December 1, 2021.

Release and mount a CLB instance

Account management

New features

A database account of an ApsaraDB for ClickHouse Community-compatible Edition cluster can be configured by executing SQL statements.

• Manage database accounts of an ApsaraDB for ClickHouse cluster

• CREATE USER

• GRANT

Feature

Description

References

Community-compatible Edition V22.8

• New features

  • The time range of Date32 and DateTime64 is expanded. The valid value ranges from 1900-01-01 to 2299-12-31.

  • Lightweight delete is supported.

  • SQL user-defined functions and executable user-defined functions are supported.

  • Partition-level preprocessing is supported to solve pre-aggregation issues.

• Performance optimization

  • The performance of INSERT operations is doubled.

  • The performance of IN queries for long tables is tripled.

• Date32

• Datetime64

• The Lightweight DELETE Statement

• SQL User Defined Functions

• Executable User Defined Functions

• Projections

Feature

Description

References

Parameter configuration

New features

The config.xml file of ApsaraDB for ClickHouse clusters that are created after December 1, 2021, can be modified, and the change history can be queried.

Configure parameters in the config.xml file

Feature

Description

References

Update of the minor engine version

New features

The minor engine version of ApsaraDB for ClickHouse clusters that are purchased after December 1, 2021, can be updated. The update enhances performance, introduces new features, and fixes bugs.

Update the minor engine version

Setting of a maintenance window

New features

The maintenance window can be set for one or more ApsaraDB for ClickHouse clusters at a time. The clusters are upgraded within the specified maintenance window to ensure the stability of the clusters.

Set a maintenance window

Feature

Description

References

Monitoring and alerting

• New features

  The following metrics are added:

  • Number of Inactive Data Parts

  • Number of TCP connections

  • Number of Init Query runs

  • Number of HTTP connections

  • Number of MaterializeMySQL

  • Number of failed Insert queries

  • Kafka appearance number

  • Number of failed Select queries

  • Number of MaterializeMySQL synchronization failures

  • The number of errors in Kafka's appearance consumption.

• Optimization

  The availability of clusters and the capabilities of monitoring exceptions are improved. This improves O&M stability.

View monitoring data in the ApsaraDB for ClickHouse console

Feature

Description

References

Data backup and restoration

New features

ApsaraDB for ClickHouse clusters of V20.3, V20.8, and V21.8 support data backup and data restoration.

• Data backup

• Data restoration

Feature

Description

References

HTTPS

New features

HTTPS can be used to implement encrypted data transmission.

• Enable HTTPS

• Connect to an ApsaraDB for ClickHouse cluster over HTTPS

MaterializeMySQL engine upgrade

• New features

  The MaterializeMySQL engine supports the dirty data of the ENUM data type.

• Optimization

  • Full data synchronization is optimized. Tables are not locked during a synchronization process. This reduces the negative impact of the operation on your business.

  • DDL synchronization is optimized. Information about errors that occur during a synchronization process is written to the system table.

None

TCP keepalive

New features

By default, the TCP keepalive feature is enabled to ensure connection reliability.

None

Feature

Description

References

MaterializeMySQL engine upgrade

New features

The MaterializeMySQL engine supports the DECIMAL data type.

None

Feature

Description

References

Community-compatible Edition V21.8

• New features

  • More SQL statements and JSON standards are supported.

  • The MaterializedPostgreSQL table engine and the MaterializedPostgreSQL database engine are added to allow synchronous replication of databases and tables.

  • The following functions are added: leftPad(), rightPad(), leftPadUTF8(), and rightPadUTF8().

  • The bin and unbin functions are added.

  • The FIRST keyword is added to the ADD INDEX command. This keyword allows you to add an index to the beginning of an index list.

  • The following data types are supported in the MapAdd and MapSubTract functions: MAP, UINT128, INT128, UINT256, and INT256.

  • The DISTINCT ON (columns) expression is supported.

• Optimization

  The latency of a short query that reads data from multiple table columns is reduced.

None

Feature

Description

References

Preservation of binary log files

The enable_binlog_reserved parameter can be set to enable the preservation feature for binary log files in the MaterializeMySQL engine. This feature prevents data synchronization failures that occur when binary log files are deleted. This helps improve the stability of the MaterializeMySQL engine.

MaterializeMySQL

Feature

Description

References

Cluster restart

ApsaraDB for ClickHouse allows you to restart a cluster. If an ApsaraDB for ClickHouse cluster becomes unavailable or the cluster must be restarted to fix issues, you can restart the cluster.

Restart an ApsaraDB for ClickHouse cluster

Monitoring page optimization

The layout of the monitoring page is optimized to improve user experience.

None

Support for ApsaraDB RDS for MySQL analytic instances

ApsaraDB RDS for MySQL analytic instances are supported by ApsaraDB for ClickHouse. You can synchronize data from an ApsaraDB RDS for MySQL analytic instance to an ApsaraDB for ClickHouse cluster. This helps you analyze business data in a centralized manner.

Create an ApsaraDB for ClickHouse instance

Feature

Description

References

Data migration between ApsaraDB for ClickHouse clusters

The data migration feature is available in the ApsaraDB for ClickHouse console. You can use this feature to migrate data between ApsaraDB for ClickHouse clusters of the same engine version or from a cluster of an earlier version to a cluster of a later version.

Migrate data between ApsaraDB for ClickHouse Community-compatible Edition clusters

Feature

Description

References

Multiple scale-out methods

Simple scale-out is supported. You can select one of the following scale-out methods:

• Migration-based scale-out

• Simple scale-out

Scale-in and scale-out

MaterializeMySQL engine upgrade

• The JSON, TIME, and YEAR data types are supported.

• Bugs that are related to keywords and table creation for incremental data are fixed.

• The following parameters are supported:

  • skip_error_count

  • alter include_tables

MaterializeMySQL

Performance optimization for horizontal scaling

clickhouse-copier is optimized to improve the efficiency of horizontal scaling.

Modify the configurations of an ApsaraDB for ClickHouse Community-compatible Edition cluster

Feature

Description

References

Data backup and restoration

The data backup and data restoration feature is available. This feature prevents data loss and improves data security.

Data backup

Feature

Description

References

Tiered storage of hot and cold data

Cold data and hot data can be automatically migrated. This reduces storage costs by more than 80% and does not degrade performance.

Tiered storage of hot data and cold data

Performance level 2 (PL2) and PL3 Enterprise SSDs (ESSDs)

PL2 and PL3 ESSDs can be used to store data. This improves I/O throughput by several times.

None

Disk encryption

All data on data disks is encrypted based on block storage. If the data is leaked, unauthorized users cannot decrypt the data. This ensures data security.

Disk Encryption

Feature

Description

References

Availability of the China (Hohhot) and China (Hong Kong) regions

None

None

Log delivery optimization

Log data in Simple Log Service can be delivered to ApsaraDB for ClickHouse for real-time analysis.

Synchronize data from Log Service

Feature

Description

References

Availability of the China (Zhangjiakou), China (Chengdu), and Japan (Tokyo) regions

None

None

Account management and permission management on databases and data dictionaries

The features for managing accounts and permissions on databases and data dictionaries are available in the ApsaraDB for ClickHouse console. After you log on to the ApsaraDB for ClickHouse console, you can create accounts and change the passwords of the accounts. You can also grant the permissions on databases and data dictionaries to new users and modify the permissions of existing users.

Manage database accounts of an ApsaraDB for ClickHouse cluster

Change of global parameter values

The values of global parameters can be changed in the ApsaraDB for ClickHouse console. The change takes effect after your cluster is restarted.

Configure parameters in the config.xml file

Feature

Description

References

Balanced data distribution during a scale-out operation

Data is evenly distributed when nodes are added to a cluster. This prevents data skew issues.

None

Query management

Query management is supported. Slow SQL queries can be identified, and running queries can be interrupted.

Monitor slow SQL queries and manage running SQL queries

New specifications

The specifications of 4 CPU cores and 16 GB memory are added to improve the processing capability of single nodes.

Billable items

Data dictionaries

Data dictionaries can be configured in the ApsaraDB for ClickHouse console. If constant data exists or tables that store business data can be split into fact tables and dimension tables, data dictionaries can be used as a substitute for the dimension tables. This allows you to query the data dictionaries to obtain dimension data and improves query efficiency.

Data dictionaries

Resource queues

Resource queues are used to resolve the following issue that may occur in open source ClickHouse: A single query that consumes a large amount of resources can cause cluster breakdown.

Resource queues in ApsaraDB for ClickHouse

Feature

Description

References

Integration with Quick business intelligence (BI)

ApsaraDB for ClickHouse clusters can be added as data sources in Quick BI to build BI systems.

Quick BI

Feature

Description

References

V20.3

ApsaraDB for ClickHouse V20.3.10.75 is released to fix engine bugs and enhance stability.

None

Real-time data migration or synchronization from MySQL databases

Data in MySQL databases can be migrated or synchronized to ApsaraDB for ClickHouse clusters in real time. This allows you to interact with MySQL databases and analyze business data in real time.

Import data from ApsaraDB RDS for MySQL or PolarDB for MySQL

Online log delivery and analysis

The online log delivery and analysis feature is supported. Enterprise users need to only create delivery tasks in the ApsaraDB for ClickHouse console and deliver behavioral log data from Simple Log Service to ApsaraDB for ClickHouse. This allows you to use ApsaraDB for ClickHouse to analyze log data from multiple dimensions.

Synchronize data from Log Service

Feature

Description

References

Integration with Data Management (DMS)

ApsaraDB for ClickHouse is integrated with DMS. This allows you to manage databases and tables, query data, and orchestrate and schedule SQL tasks online.

None

Feature

Description

References

New specifications

The C4, C24, C64, and C104 specifications are added. HTTP ports are enabled.

Billable items