PolarDB:Release notes for CDC nodes

Category

Description

New features

You can now modify the partition keys and sort keys of clustered columnstore indexes (CCI). This feature provides greater flexibility for data management, allowing you to adjust the data distribution and sort policies of CCI without having to completely rebuild an index, which in turn optimizes query performance.

Fixed issue

Fixed an issue where a data node (DN) could interrupt the data stream in single-stream binary logging service scenarios.

Category

Description

New feature

The CDC Node now supports filtering out delete events generated from archived tables when producing binary logs.

Category

Description

New feature

Multi-stream binary log: Now supports a one-to-one mapping between a storage node and a Binlog stream.

Feature optimizations

Optimized memory management for dump operations on binary logs.

Improved the performance of CDC nodes when outputting binary logs to downstream systems to increase the output rate.

Optimized the SQL statements sent from CDC nodes to the global meta service (GMS) to reduce the load on the GMS.

Category

Description

New feature

Multi-stream binary log: Added support for a one-to-one mapping between a storage node and a Binlog stream.

Category

Description

New features and optimizations

CHECK constraints are supported.

Optimized the allocation policy for CDC off-heap direct memory to prevent out-of-memory (OOM) issues caused by insufficient memory allocation.

Optimized the forced restart mechanism for CDC to allow a forced restart even when the CDC worker process is blocked.

Category

Description

Feature optimizations

Optimized the default size of a single binary log file to 1 GB. Added support for automatic rotation when the binary log file number exceeds 999,999.

Optimized the download logic for transparently consumed binary log files.

Category

Description

Feature optimizations

Optimized the replication logic for primary and secondary nodes.

Optimized the heartbeat timeout detection logic for the CDC high availability (HA) module to prevent unnecessary process restarts.

Fixes

Fixed an issue where an upgrade from an earlier version of CDC was interrupted because the RuntimeVersion field was missing from the system table configuration.

The following issue is fixed: The WITH PARSER NGRAM attribute is not included in the results of the SHOW CREATE TABLE statement.

Category

Description

Fixed issue

Fixed an issue where CDC links would be interrupted due to compatibility differences when processing REAL and DOUBLE types in CDC metadata.

Category

Description

Fixed issue

Fixed an issue where consecutive TableMapEvents could occur when a single SQL statement updated multiple tables, which resulted in disordered TraceIDs.

Category

Description

Fixes

Fixed an issue where the CDC process failed to start because a single compute node was unavailable.

Fixed an issue where dump operations on a binary log file failed because the offset value in memory was inconsistent with the offset value in the file.

Category

Description

Fixes

Fixed an issue where the information returned from the SHOW FULL BINARY LOGS statement was incomplete.

Fixed an issue that caused a binary log file to be lost when the system concurrently uploaded the same file during CDC topology reconstruction.

Fixed an issue where the SQL parser failed to parse ALTER statements that contain implicit table groups, which prevented marking statements from being processed correctly.

Fixed an issue that prevented the replication link from being automatically recovered. This issue occurred because the system automatically deleted local files on the master node during CDC Dumper primary-secondary synchronization.

Category

Description

New feature

Added support for CDC nodes to periodically build sync points.

Fixed issue

Fixed an issue where consumption was blocked because the binary log cursor was not updated as expected.

Category

Description

New features and optimizations

Added support for the CDC Node to periodically construct Sync Points.

Optimized the full data validation capability in the upgrade assessment from PolarDB-X 1.0 to PolarDB-X 2.0.

Fixes

Fixed an issue where global binary logs failed to be generated due to metadata changes.

Fixed an issue where the binary log dump service could not be provided because the requested target binary log file was empty.

Category

Description

New features and optimizations

Primary/secondary DDL replication now supports replicating functions.

Primary/secondary DDL replication now supports replicating sequences.

Primary/secondary DDL replication now supports replicating views.

Primary/secondary DDL replication now supports replicating procedures.

Primary/secondary DDL replication now supports replicating accounts, roles, and permissions.

Primary/secondary DDL replication now supports replicating alter tablegroup operations.

Primary/secondary DDL replication now supports replicating alter index operations.

Primary/secondary DDL replication now supports consistency-coordinated and aligned replication in multi-stream binary logging scenarios.

Primary/secondary replication now supports creating replication links based on timestamps.

Primary/secondary replication now supports using SQL commands to verify data consistency between upstream and downstream systems.

Added a secondary instance role for primary/secondary replication. This allows the secondary cluster to have instance-level read-only capabilities.

Added support for full mirror matching of WHERE conditions for DML writes in primary/secondary replication.

Added support for binding multi-stream binary log stream names to usernames. When you use a bound account to execute binary log-related SQL statements, you do not need to add the 'with' option.

Optimized server_id-based bidirectional replication for primary/secondary replication. You can now specify filter rules from the command line and perform automatic checks for server_id anomalies.

Fixes

Fixed an issue where the MULTI ALTER ADD/DROP COLUMN statement caused an incorrect metadata column order, which interrupted the binary log link.

Fixed an issue where MULTI ALTER ADD/DROP/RENAME INDEX  could cause a loss of index metadata, resulting in an incorrect DDL statement being recorded in the binary logs.

Fixed a configuration defect in the meta_build_physical_ddl_sql_blacklist_regex parameter that caused DDL statements to be filtered if they contained the user keyword.

Fixed an issue where the drop index DDL statements could not be output to binary logs.

Category

Description

New feature

Adds support for binlog-related O&M instructions, such as SHOW FULL BINARY LOGS and SHOW FULL MASTER STATUS.

Fixed issue

Fixed an issue where SQL flashback exact match failed.

Category

Description

New feature

Added the SHOW FULL MASTER STATUS command.

Feature optimization

Improved MySQL binary log dump compatibility by supporting parameters such as checksum and heartbeat.

Category

Description

New feature

Added support for SQL flashback to roll back an entire transaction.

Fixed issue

Fixed the recording method for TSO heartbeats in the binary log to resolve potential crash issues in downstream MySQL consumers.

Category

Description

Feature optimization

Optimized the sleep time for gRPC backpressure control, increasing the external Binlog dump speed to 130 MB/s.

Fixed issue

Fixed an issue where SQL flashback could not finish because no DML or DDL operations occurred within the specified time range.

Category

Description

Feature optimization

Optimized the query efficiency of the SHOW BINARY LOGS statement when there is many binary logs and resolved the slow query issue.

Fixes

Fixed a buffer overflow issue when shaping JSON data.

Fixed an issue where the metaDataSource scan thread exited due to a high availability (HA) switchover in the metaDB during a CDC upgrade.

Fixed a deadlock issue when updating binlog_dumper_info.

Category

Description

New features and optimizations

Added support for the FLUSH LOGS statement.

Optimized the external dump speed for transparent consumption.

CDC no longer depends on is_vip to retrieve a DN connection.

Removed the control dependency for CDC to retrieve a MetaDB connection.

Fixes

Fixed a parsing syntax error that occurred in table creation statements that included a BIGINT SIGNED column definition.

Fixed an issue where the recover tso generation algorithm could cause gaps in the binary log list.

Fixed an issue where Task dump could not find the starting offset on the primary instance DN after a read-only instance was scaled.

Category

Description

New feature

Added support for the persistence of transaction objects to reduce memory usage.

Fixes

Fixed an issue where, during recovery based on a historical TSO, the `recover tso` could not be smaller than the TSO of the most recent scaling-tagged transaction.

Fixed an issue where the gRPC thread could not exit because local files on the Dumper Master were deleted during CDC Dumper primary-secondary synchronization.

Category

Description

Fixes

Fixed an issue where a heartbeat window conflict could be triggered when consuming binary logs from OSS. This conflict caused a link error.

Fixed an issue where a data format conversion error could occur when shaping a TableMapEvent.

Fixed an issue where the Sql_Mode value in a DDL Event was inconsistent with the original Sql_Mode value.

Fixed an issue where metadata was inconsistent between logical and physical tables for nchar and nvarchar types.

Fixed an issue where converting to a single-line comment failed when a DDL SQL statement contained a line feed.

Category

Description

New features and optimizations

Added support for monitoring and collecting statistics on the CPU utilization of worker processes.

Added support for more fine-grained monitoring of Events Per Second (EPS) for INSERT, UPDATE, and DELETE operations.

Optimized the speed of uploading binary log files to OSS, increasing the rate to 700 MB/s.

Added CDC Operations and Maintenance (O&M) commands, such as start master, stop master, restart master, and reset master.

Fixes

Fixed an issue where a long block occurred during the transparent consumption of offline binary logs.

Fixed an issue where the dump thread could not exit because a binary log file was deleted while being read.

Fixed an issue where `reset master` failed when there was too much physical DDL metadata.

Fixed an issue where the Sql_Mode value in a DDL Event was inconsistent with the original sql_mode value.

Fixed an issue where a downstream MySQL error occurred because the same SQL statement contained both ADD PRIMARY KEY and DROP PRIMARY KEY.

Category

Description

Fixes

Fixed an issue where the column order was not as expected after an SQL statement was executed in the SchemaRepository.

Fixed an issue where an NPE was triggered when shaping an empty string.

Category

Description

Fixed issue

Fixed an issue where data was truncated because the metadata exceeded the length of the mediumtext data type.

Category

Description

Fixed issue

Fixed a parsing failure that occurred because the MySQL charset was not converted to the Java charset.

Category

Description

Feature optimizations

Optimized the recovery time for the binary log synchronization link, reducing the RTO to 15s.

Optimized the recovery speed of CDC metadata information when the number of data tables reaches several hundred thousand.

Fixes

Fixed a potential offset calculation error when transparently consuming binary logs.

Fixed a compatibility issue related to text(0) in metadata consistency checks.

Category

Description

Fixes

Fixed an issue where the CDC link was interrupted due to the continuous addition and deletion of primary keys.

Fixed an issue where CDC actively disconnected because the consumer program did not send COM_REGISTER_SLAVE before dumping binary logs.

Fixed an issue where the data link was interrupted when the hash value was Integer.MIN_VALUE in a multi-stream data routing scenario.

Category

Description

New features and optimizations

Added support for multi-level merging. This ensures that performance does not significantly decrease when DN nodes are linearly increased.

Added support for transparent consumption. Downstream consumer programs can use the dump protocol to directly consume binary log files archived to OSS without needing to connect to OSS themselves.

Added support for multi-stream binary logs. You can customize the number of streams. Each stream has an independent binary log file stream.

Added support for periodically building full metadata snapshots. This resolves the issue of excessive expansion of historical DDL tagging records.

Added support for downloading binary log files from OSS in parallel to improve instance recovery speed.

Fixes

Fixed compatibility and correctness issues when shaping binary log event data. This improves the stability of the CDC link during DDL changes.

Fixed an issue where a dump timeout could occur for downstream subscriptions after a CDC cluster restart.

Fixed an issue where the calculated latency was incorrect when the Dumper process started for the first time.