You can use DTS to track data changes from user-created MySQL databases and ApsaraDB RDS for MySQL instances in real time. This feature applies to the following scenarios: cache updates, business decoupling, asynchronous data processing, real-time synchronization of heterogeneous data, and real-time synchronization of extract, transform, and load (ETL) operations.

The previous change tracking feature requires high costs. To improve user experience and reduce usage costs, Alibaba Cloud has upgraded the change tracking feature to achieve repeated consumption of data across multiple downstream instances.

Features

  • You can consume data across multiple downstream instances and filter databases and data tables. For more information, see Consumer groups.
  • You can track data changes from user-created MySQL databases.
  • You can use the change tracking feature in the classic network and VPCs. You can track and consume data within a VPC for lower network latency and higher data security.
    Note You can select a network type when you create a change tracking channel. For more information, see Track data changes from ApsaraDB RDS for MySQL (new).
  • You can consume data in a change tracking channel by using a Kafka client.
    Note Only Kafka versions 0.10.0.x to 1.1.x are supported. For more information, see Use a Kafka client to consume tracked data.
  • You can add or remove objects for change tracking. For more information, see Modify objects for change tracking.
  • You can modify consumption checkpoints.
    Note You must use your change tracking client to modify consumption checkpoints.
  • You can monitor the status of the change tracking channel. If the latency threshold for downstream consumption is reached, you will receive an alert. You can set the alert threshold based on the sensitivity of your businesses to consumption latency.

Consumer groups

When you use the previous change tracking feature, one downstream SDK client consumes data and other SDK clients act as backup clients. If you need to repeatedly consume incremental data in the same source database, you must purchase multiple change tracking instances. This incurs high costs.

DTS provides consumer groups to reduce costs. You can create consumer groups for downstream consumers who need to subscribe to the same database instance. Consumers in these consumer groups can subscribe to the same database instance and consume data at the same time. This reduces user costs.

Consumer group features

  • You can create multiple consumer groups in a change tracking instance to repeatedly consume data.
  • A consumer group consumes each message only once. Each consumer in the consumer group serves as a backup for other consumers.
  • In a consumer group, only one consumer can consume data at a time. Other consumers act as disaster recovery nodes.

Supported database types and versions

A source MySQL database can be one of the following types: User-Created Database in ECS Instance, and User-Created Database Connected Over Express Connect, VPN Gateway, or Smart Access Gateway, Database without public IP:Port (Accessed through database gateway) and User-Created Database with Public IP Address .

Source database Data change type

User-created MySQL database

5.1, 5.5, 5.6, 5.7, or 8.0

  • Data update
  • Schema update

ApsaraDB RDS for MySQL

All versions

Supported languages

DTS allows you to consume data by using the clients that are written in the following programming languages:

  • Java
  • C and C++
  • Python
  • Go (Golang)
  • Erlang
  • .NET
  • Ruby
  • Node.js
  • Proxy (such as HTTP REST)

You can consume data by using a Kafka client for a specific language. For more information, visit Kafka clients. DTS provides an example of using the Kafka client for Java to consume data. For more information, see Use a Kafka client to consume tracked data.

Warning If you use the official Kafka client to consume data, you must specify the username in the following format: <Consumer group account>-<Consumer group ID>, for example, dtstest-dtsae******bpv. Otherwise, the connection fails.

Supported data change types and objects

  • Data change types

    Data changes from a source database include schema updates that use the data definition language (DDL) and data updates that use the data manipulation language (DML).

    • Schema updates

      You can track the operations that create, delete, or modify the schemas of objects in an instance. To do so, you must use the change tracking client to filter the required data.

    • Data updates

      You can track data updates of a selected object, including the INSERT, DELETE, and UPDATE operations.

  • Objects for change tracking

    The objects for change tracking include tables and databases.

Change tracking channel

A change tracking channel pulls incremental data from the source database in real time and stores incremental data that was generated in the last 24 hours. You can use a Kafka client to track and consume incremental data in the change tracking channel. You can create, manage, and delete change tracking channels in the DTS console.

The following table describes the statuses of a change tracking channel when you create and run the channel.

Channel status Description Available operation
Prechecking The change tracking channel has been configured and a precheck is being performed. Delete the change tracking channel
Not Started The change tracking channel has passed the precheck, but is not started.
  • Start the change tracking channel
  • Delete the change tracking channel
Performing Initial Change Tracking The initial change tracking is in progress. This process requires about 1 minute to complete. Delete the change tracking channel
Normal Incremental data is being pulled from the data source.
  • View sample code
  • Configure monitoring and alerts
  • Delete the change tracking channel
  • Reconfigure the change tracking channel
Error An error occurs when the change tracking channel is pulling incremental data from the data source.
  • View sample code
  • Configure monitoring and alerts
  • Delete the change tracking channel
  • Reconfigure the change tracking channel
Note
  • A change tracking channel in the Error state will be automatically resumed within seven days. To stop change tracking, you must release the change tracking channel.
  • After a change tracking channel is released, the data in the channel is cleared and cannot be recovered. We recommend that you use caution when you perform this operation.