All Products
Search
Document Center

:Change tracking (new)

Last Updated:Feb 05, 2024

You can use Data Transmission Service (DTS) to track data changes from user-created MySQL databases and ApsaraDB RDS for MySQL instances in real time. This feature applies to the following scenarios: cache updates, business decoupling, synchronization of heterogeneous data, and synchronization of extract, transform, and load (ETL) operations.

The previous change tracking feature requires high costs. To improve user experience and reduce usage costs, Alibaba Cloud has upgraded the change tracking feature to achieve repeated consumption of data across multiple downstream instances.

Features

  • You can consume data across multiple downstream instances and filter databases and data tables. For more information, see Consumer groups.

  • You can track data changes from user-created MySQL databases.

  • You can use the change tracking feature in the classic network and VPCs. You can track and consume data within a VPC for lower network latency and higher data security.

    Note

    You can select a network type when you create a change tracking task. For more information, see Track data changes from an ApsaraDB RDS for MySQL instance.

  • You can consume data in a change tracking task by using a Kafka client.

    Note

    Only Kafka versions 0.10.0.x to 1.1.x are supported. For more information, see Use a Kafka client to consume tracked data.

  • You can add or remove objects for change tracking. For more information, see Modify the objects for change tracking.

  • You can modify consumption checkpoints.

    Note

    You must use your change tracking client to modify consumption checkpoints.

  • You can monitor the status of change tracking tasks. If the latency threshold for downstream consumption is reached, you will receive an alert. You can set the alert threshold based on the sensitivity of your business to consumption latency.

Consumer groups

When you use the previous change tracking feature, one downstream SDK client consumes data and other SDK clients act as backup clients. If you need to repeatedly consume incremental data in the same source database, you must purchase multiple change tracking instances. This incurs high costs.

DTS provides consumer groups to reduce your costs. You can create consumer groups for downstream consumers who need to subscribe to the same database instance. Consumers in these consumer groups can subscribe to the same database instance and consume data at the same time.

Consumer group features

  • You can create multiple consumer groups in a change tracking instance to repeatedly consume data.

  • A consumer group consumes each message only once. Each consumer in the consumer group serves as a backup for other consumers.

  • In a consumer group, only one consumer can consume data at a time. Other consumers act as disaster recovery nodes.

Supported database types and versions

A source MySQL database can be one of the following types: User-Created Database with Public IP Address, User-Created Database in ECS Instance, and User-Created Database Connected Over Express Connect, VPN Gateway, or Smart Access Gateway.

Source database

Data change type

User-created MySQL database

5.1, 5.5, 5.6, 5.7, or 8.0

  • Data update

  • Schema update

ApsaraDB RDS for MySQL

All versions

Supported programming languages

DTS allows clients to consume data by using the following programming languages:

  • Java

  • C/C++

  • Python

  • Go (Golang)

  • Erlang

  • .NET

  • Ruby

  • Node.js

  • Proxy (such as HTTP REST)

You can consume data by using a Kafka client for a specific language. For more information, see Kafka clients. DTS provides an example of using the Kafka client for Java to consume data. For more information, see Use a Kafka client to consume tracked data.

Warning

If you use the official Kafka client to consume data, you must specify the username in the following format: <Consumer group account>-<Consumer group ID>, for example, dtstest-dtsae******bpv. Otherwise, the connection fails.

Supported data change types and objects

  • Data change types

    Data changes include schema updates that use the data definition language (DDL) and data updates that use the data manipulation language (DML).

    • Schema update

      You can track the operations that create, delete, or modify the schemas of objects in an instance. To do this, you must use the change tracking client to filter the required data.

    • Data update

      You can track data updates of a selected object, including the INSERT, DELETE, and UPDATE operations.

  • Objects for change tracking

    The objects for change tracking include tables and databases.

Change tracking tasks

A change tracking task pulls data changes from the source database in real time and stores the data changes that were generated in the last 24 hours. You can use a SDK, Flink, or Kafka client to track and consume data in the change tracking task. You can create, manage, and delete change tracking tasks in the DTS console.

The following table describes the statuses of a change tracking task when you create and run the task.

Task status

Description

Available operation

Prechecking

The configuration of the change tracking task is complete and a precheck is being performed.

Delete the change tracking task

Not Started

The change tracking task has passed the precheck but has not been started.

  • Start the change tracking task

  • Delete the change tracking task

Performing Initial Change Tracking

The initial change tracking is in progress. This process takes about 1 minute.

Delete the change tracking task

Normal

Data changes are being pulled from the data source.

  • View the demo code

  • Configure monitoring and alerts

  • Delete the change tracking task

  • Reconfigure the change tracking task

Error

An error occurs when the change tracking task pulls data changes from the data source.

  • View the demo code

  • Configure monitoring and alerts

  • Delete the change tracking task

  • Reconfigure the change tracking task

Note
  • A change tracking task in the Error state will be resumed within seven days. To stop change tracking, you can delete the change tracking task.

  • After a change tracking task is deleted, the data tracked by the task is cleared and cannot be recovered. We recommend that you use caution when you perform this operation.