You can use DTS to retrieve incremental data from user-created MySQL databases in real time. This feature applies to the following scenarios: cache updates, business decoupling, asynchronous data processing, real-time synchronization of heterogeneous data, and real-time data synchronization with complex ETL operations.
The previous change tracking feature requires high costs. To improve user experience and reduce usage costs, Alibaba Cloud has upgraded the change tracking feature to achieve repeated consumption of data across multiple downstream instances.
- You can consume data across multiple downstream instances and filter databases and data tables. For more information, see Consumer groups.
- You can track data changes that are made to user-created MySQL databases.
- You can use the change tracking feature in the classic network and VPCs. You can subscribe
to and consume data in a VPC for lower network latency and higher security.
Note You can select a network type when creating a change tracking channel. For more information, see Track data changes from ApsaraDB RDS for MySQL (new).
- You can consume data in a change tracking channel by using the Kafka client.
Note Only Kafka 0.10.0.x to 1.1.x is supported. For more information, see Use a Kafka client to consume tracked data.
- You can add and remove objects for change tracking. For more information, see Modify objects for change tracking.
- You can modify consumption checkpoints.
Note You need to use your change tracking client to modify consumption checkpoints.
- You can monitor the status of the change tracking channel and receive an alert if the threshold for downstream consumption is reached. You can set the alert threshold based on the sensitivity of your businesses to consumption latency.
The previous change tracking feature requires that one downstream SDK client consumes data and other SDK clients act as backup. If you need to repeatedly consume the incremental data in the same source database, you must purchase multiple change tracking instances. This incurs high costs.
Consumer groups provided by DTS help you reduce costs. You can create consumer groups for downstream consumers who need to subscribe to the same database instance. Consumers in these consumer groups can subscribe to the same database instance and consume data at the same time. This reduces user costs.
Consumer group features
- You can create multiple consumer groups in a change tracking instance to repeatedly consume data.
- A consumer group consumes each message only once, and consumers in the consumer group serve as a backup for each other.
- In a consumer group, only one consumer can consume data at a time, while other consumers act as disaster recovery nodes.
Supported database types and versions
A source MySQL database can be one of the following types: User-Created Database with Public IP Address, User-Created Database in ECS Instance, and User-Created Database Connected Over Express Connect, VPN Gateway, or Smart Access Gateway.
|Source database||Data type|
User-created MySQL database
5.1, 5.5, 5.6, 5.7, or 8.0
ApsaraDB RDS for MySQL
- C and C++
- Go (Golang)
- Proxy (such as HTTP REST)
Supported data types and objects
- Data types
Incremental data in the source database includes schema update that uses Data Definition Language (DDL) and data update that uses Data Manipulation Language (DML).
- Schema update
You can subscribe to the creation, deletion, and modification of all object schemas in an instance. You need to use the change tracking client to filter the required data.
- Data update
You can subscribe to data updates of a selected object, including INSERT, DELETE, and UPDATE operations.
- Schema update
- Objects for change tracking
The objects for change tracking include databases and tables.
Change tracking channel
A change tracking channel pulls incremental data from the source database in real time and stores the incremental data that is generated in the last 24 hours. You can use the Kafka client to subscribe to and consume incremental data from the change tracking channel. You can create, manage, and delete change tracking channels in the DTS console.
The following table describes the statuses of a change tracking channel during creation and running.
|Channel status||Description||Available operations|
|Prechecking||The change tracking channel have been configured and a precheck is being performed.||Delete the change tracking channel.|
|Not Started||The change tracking channel has passed the precheck, but is not started.||
|Performing Initial Change Tracking||The initial change tracking is in progress. This process requires about one minute to complete.||Delete the change tracking channel.|
|Normal||Incremental data is being pulled from the data source.||
|Error||An error occurs when the change tracking channel is pulling incremental data from the data source.||
- A change tracking channel in the Error state will be resumed within seven days. To stop the change tracking, you must release the change tracking channel.
- After a change tracking channel is released, the data in the channel will be released and cannot be recovered. We recommend that you use caution when performing this operation.