PolarDB:CDC node release notes

Category

Description

New features and enhancements

Upgraded the CDC (log node) to version 5.4.21.

CDC (log node) adds support for a new DN query event variable, Q_OPT_INDEX_FORMAT_HIPPO_ENABLED.

Added support for binlog compression.

Improved data synchronization performance for columnar indexes (CCI).

Accelerated the recovery of the CDC (log node) after a storage node (DN) is resized.

Bug fixes

Fixed an issue that could cause an exception in downstream data synchronization tools, such as Flink CDC, when parsing binlog events of the FORMAT_DESCRIPTION_EVENT type.

Fixed an issue where columnar index (CCI) reported an error when parsing DDL statements that contain the comment character #.

Fixed a bug that caused occasional errors in the binlog consumption pipeline when binlogs were generated at a high rate.

Fixed a bug where enabling the source-side filtering feature in the CDC (log node) occasionally interrupted connections to downstream PolarDB-X instances.

Fixed a bug that occasionally caused JVM crashes due to frequent GC (garbage collection) when the CDC (log node) was running on extra-large instances.

Fixed an issue where the primary CDC (log node) link might be interrupted when an ALTER operation is performed on a table name that contains backticks ``.

Category

Description

New feature

Adds active filtering for binary log dump to exclude delete events generated by archive tables during binary log creation.

Category

Description

New features and enhancements

Added support for virtual tables. You can now configure mapping rules to map multiple qualifying physical tables to a single virtual table.

Added the daemon_tso_heartbeat_suspend_enabled parameter to control whether to suspend the heartbeat check for the TSO service under specific conditions.

Improved performance when syncing data from binlogs to columnar store indexes (CCI).

Fixes

Fixed an issue where an excessively large binlog event could cause an OOM error in the CDC (log node) during binlog pipeline recovery.

Category

Description

New features

You can now modify the partition key and sort key of a clustered columnstore index (CCI). This allows you to adjust the data distribution and sorting strategy based on business changes, optimizing query performance without rebuilding the index.

You can now perform in-place rebuilding of CCI to optimize and reorganize the index without affecting queries or access to historical data versions.

Fixes

A data node (DN) no longer causes data stream interruptions with the single-stream binary log service.

Category

Description

New feature

The CDC node now filters delete events from archive tables when producing binary logs.

Category

Description

New feature

Added support for a one-to-one mapping between a storage node and a binlog stream in multi-stream CDC configurations.

Optimizations

Optimized memory management for the binlog dump operation.

Increased the throughput of CDC nodes when streaming binlogs to downstream systems.

Optimized the SQL statements sent from CDC nodes to the global meta service (GMS) to reduce its load.

Category

Description

New feature

Multi-stream binlog: Now supports a one-to-one mapping between a storage node and a binlog stream.

Category

Description

New features and optimizations

Added support for CHECK constraint.

Optimized the allocation policy for CDC off-heap direct memory to prevent out-of-memory (OOM) issues.

Optimized the CDC forced restart mechanism to trigger even when the worker process is blocked.

Category

Description

Enhancements

Increased the default size of a single binary log file to 1 GB and enabled automatic rotation when the binary log file sequence number exceeds 999,999.

Optimized the download logic for transparently consumed binary log files.

Category

Description

Feature optimizations

Optimized the replication logic for primary and secondary nodes.

Optimized the heartbeat timeout detection logic for the CDC high availability (HA) module to prevent unnecessary process restarts.

Fixes

Fixed an issue where upgrades from earlier CDC versions were interrupted due to a missing RuntimeVersion field in the system table configuration.

The results of the SHOW CREATE TABLE statement now include the WITH PARSER NGRAM attribute.

Category

Description

Fixed issue

Fixed an issue where CDC links were interrupted due to a compatibility difference in how CDC metadata handles REAL and DOUBLE types.

Category

Description

Fixed issue

Fixed a bug where a single SQL statement updating multiple tables would generate consecutive TableMapEvents, causing out-of-order TraceIDs.

Category

Description

Fixes

Fixed a CDC process startup failure caused by a single unavailable compute node.

Fixed dump operation failures on a binary log file caused by an offset mismatch between memory and the file.

Category

Description

Fixes

Fixed an issue where executing the SHOW FULL BINARY LOGS statement returned incomplete information.

Prevented binlog file loss caused by concurrent uploads of the same file during CDC topology reconstruction.

Fixed an issue where the SQL parser failed to parse ALTER statements that contain implicit table groups and did not correctly process marking statements.

Ensured automatic recovery of the replication link by preventing the deletion of local files on the master node during CDC Dumper primary-secondary synchronization.

Category

Description

New feature

CDC nodes can now periodically create sync points.

Bug fix

Fixed a bug where the binary log cursor would not update as expected, blocking consumption.

Category

Description

New features and optimizations

Added support for Change Data Capture (CDC) nodes to periodically create Sync Point.

Optimized full data validation for the upgrade assessment from PolarDB-X 1.0 to PolarDB-X 2.0.

Bug fixes

Fixed global binary log generation failures caused by metadata changes.

Fixed a bug that caused the binary log dump service to fail when a requested binary log file was empty.

Category

Description

New features and optimizations

Primary/secondary DDL replication now supports replicating functions.

Primary/secondary DDL replication now supports replicating sequences.

Primary/secondary DDL replication now supports replicating views.

Primary/secondary DDL replication now supports replicating procedures.

Primary/secondary DDL replication now supports replicating accounts, roles, and permissions.

Primary/secondary DDL replication now supports replicating ALTER TABLEGROUP operations.

Primary/secondary DDL replication now supports replicating ALTER INDEX operations.

Enhanced primary/secondary DDL replication to coordinate consistency across multiple binary log streams.

You can now create a primary/secondary replication link from a specific timestamp.

Added a SQL command to verify data consistency between upstream and downstream.

Introduced a secondary instance role. This role provides the secondary cluster with instance-level read-only capabilities.

Primary/secondary replication can now exactly match the WHERE clause of DML write operations.

You can now bind multi-stream binary log stream names to usernames. This eliminates the need to specify the WITH option when executing related SQL commands with a bound account.

Optimized server_id-based bidirectional replication, allowing you to specify filtering rules from the command line and run automatic checks for server_id anomalies.

Fixes

Fixed an issue where MULTI ALTER ADD/DROP COLUMN statements could incorrectly order metadata columns, interrupting the binary log link.

Fixed an issue where MULTI ALTER ADD/DROP/RENAME INDEX statements could cause index metadata loss, resulting in incorrect DDL statements being written to the binary log.

Fixed an issue where ALTER DDL statements containing the user keyword were filtered due to a misconfiguration of the meta_build_physical_ddl_sql_blacklist_regex parameter.

DROP INDEX DDL statements are now correctly written to the binary log.

Category

Description

New feature

Added support for binary log-related O&M instructions, such as show full binary logs and show full master status.

Fixed issue

Fixed SQL flashback failures in exact match mode.

Category

Description

New feature

Added the SHOW FULL MASTER STATUS command.

Feature optimization

Now supports parameters such as checksum and heartbeat for MySQL binary log dumping, improving compatibility.

Category

Description

New feature

Added support for SQL flashback to roll back an entire transaction.

Fixed issue

Fixed an issue where the recording of TSO heartbeats in the binlog could cause downstream MySQL instances to crash.

Category

Description

Improvements

Optimized the sleep time for gRPC backpressure control, increasing the external binlog dump speed to 130 MB/s.

Bug Fixes

Fixed an issue where sql flashback failed when no DML or DDL operations occurred within the specified time range.

Category

Description

Feature optimization

Improved the performance of the SHOW BINARY LOGS statement to resolve slow queries caused by a large number of binary logs.

Fixes

Fixed a buffer overflow that occurred during JSON data processing.

Fixed a bug that caused the metaDataSource scan thread to exit during a CDC upgrade due to an HA switchover in the metaDB.

Fixed a deadlock that occurred when updating binlog_dumper_info.

Category

Description

New features and optimizations

Added support for the FLUSH LOGS statement.

Optimized the speed of external dumps for transparent consumption.

CDC no longer depends on is_vip to retrieve a DN connection.

CDC no longer depends on the control component to retrieve a MetaDB connection.

Fixes

Fixed a parsing error in table creation statements that included a BIGINT SIGNED column definition.

Fixed gaps in the binary log list caused by the recover tso generation algorithm.

Fixed an issue that prevented Task dump from finding the starting offset on the primary instance DN after read-only instance scaling.

Category

Description

New feature

Added support for persisting transaction objects to reduce memory usage.

Fixes

The recover tso can now be smaller than the TSO of the most recent scaling-tagged transaction during a recovery based on a historical TSO.

gRPC threads now exit correctly even if local files on the Dumper Master are deleted during CDC Dumper primary-secondary synchronization.

Category

Description

Fixes

Fixed a heartbeat window conflict that could cause a link error when consuming binary logs from OSS.

Fixed intermittent data format conversion errors during TableMapEvent processing.

Fixed an inconsistency between the Sql_Mode value in a DDL event and the original Sql_Mode value.

Fixed metadata inconsistency between logical and physical tables for nchar and nvarchar types.

Fixed an issue that prevented a DDL statement containing a line feed from being converted into a single-line comment.

Category

Description

New features and optimizations

Added support for monitoring CPU utilization for worker processes.

Added support for more fine-grained Events Per Second (EPS) monitoring for INSERT, UPDATE, and DELETE operations.

Optimized binary log file uploads to OSS, reaching 700 MB/s.

Added Change Data Capture (CDC) operations and maintenance (O&M) commands, such as start master, stop master, restart master, and reset master.

Fixes

Fixed an issue where transparently consuming offline binary logs could cause prolonged blocking.

Fixed an issue where the dump thread would fail to exit if a binary log file was deleted during a read operation.

Fixed an issue where reset master would fail due to excessive physical DDL metadata.

Fixed an issue where the sql_mode value in a DDL event was inconsistent with its original value.

Fixed an issue where SQL statements containing both add primary key and drop primary key would cause an error in downstream MySQL instances.

Category

Description

Fixes

Fixed an issue that resulted in an incorrect column order in SchemaRepository after an SQL statement was executed.

Fixed an issue where data shaping on an empty string triggered an NPE.

Category

Description

Fixed issue

Fixed a data truncation issue that was caused by metadata exceeding the length of the mediumtext data type.

Category

Description

Fixed issue

Fixed a parsing failure caused by MySQL charsets not being converted to their corresponding Java charsets.

Category

Description

Optimizations

Improved the recovery time for the binary log synchronization link, reducing the RTO to 15s.

Improved CDC metadata recovery speed when scaling to hundreds of thousands of data tables.

Fixes

Fixed a potential offset calculation error when transparently consuming binary logs.

Fixed a compatibility issue with text(0) during metadata consistency checks.

Category

Description

Fixes

Fixed an issue where the CDC link would be interrupted by consecutively adding and dropping a primary key.

Fixed an issue where CDC would disconnect if the consumer program did not send COM_REGISTER_SLAVE before dumping the binary log.

Fixed data link interruptions during multi-stream data routing when the hash value was Integer.MIN_VALUE.

Category

Description

New features and optimizations

Supports multi-level merging. This prevents significant performance degradation when DN nodes are scaled out.

Supports transparent consumption. Downstream consumers can use the dump protocol to consume binary log files from OSS without connecting to OSS directly.

Supports multi-stream binary logs. You can customize the number of streams, and each stream has its own independent log stream.

Supports periodically building full metadata snapshots. This prevents historical DDL tagging records from growing excessively.

Supports downloading binary log files from OSS in parallel to improve instance recovery speed.

Fixes

Fixed compatibility and correctness issues when shaping binary log event data, further improving the stability of the CDC link during DDL changes.

Fixed an issue where a CDC cluster restart could cause a dump timeout for downstream subscriptions.

Fixed incorrect latency calculations during the initial startup of the Dumper process.