How to implement cache updates with change tracking - Data Transmission Service

The change tracking feature of Data Transmission Service (DTS) can be used with MongoDB or Redis to implement a highly reliable and low-latency cache update policy. This topic describes the cache update policy by using an example in which ApsaraDB RDS for MySQL is the source and Tair (Redis-compatible) is the destination.

Prerequisites

You have configured a change tracking task and added a consumer group. For more information, see Create a change tracking task for ApsaraDB RDS for MySQL and Add a consumer group.
A Tair (Redis-compatible) instance is created and public access is enabled. For more information, see Step 1: Create an Instance and Apply for a Public Endpoint.
Note
You must add the client IP address or CIDR block to the whitelist of the Tair instance. For more information, see Configure a whitelist.

Background

A caching layer is often added to an application architecture to improve access speed and handle high read concurrency. Read requests are routed to the cache, which uses in-memory reads to boost performance. Meanwhile, data updates are persisted to a durable database such as ApsaraDB RDS for MySQL to ensure data integrity. This requires a cache update policy to keep the cache synchronized with the database.

Procedure

This section explains how to run the SDK sample code in IntelliJ IDEA to consume tracked data.

Download the change tracking SDK demo package and decompress the package.
Use IntelliJ IDEA to open the decompressed SDK demo package as a Maven project.
In the /subscribe-demo/src/main/java/com/alibaba/dts/demo/ directory, double-click the DTSMySQL2RedisDemo.java file.

Modify the parameters in the main function.

You need to modify the values highlighted in green in the code, such as your broker url in String brokerUrl = "your broker url";.

Change tracking parameters

Parameter	Description	Source
`brokerUrl`	The endpoint and port number of the Data Subscription instance. Note If the ECS instance running the SDK client and the Data Subscription instance are in the same classic network or Virtual Private Cloud (VPC), we recommend that you use the internal endpoint for the subscription to minimize network latency. We do not recommend using a public endpoint because of potential network instability.	In the DTS console, click the ID of the target Data Subscription instance. On the Basic Information page, you can obtain the endpoint and port number in the Network section.
`topic`	The subscription topic for the instance.	In the DTS console, click the ID of the target Data Subscription instance. On the Basic Information page, you can obtain the Topic in the Basic Information section.
`sid`	The ID of the consumer group.	In the DTS console, click the ID of the target Data Subscription instance and then click Consume Data. You can obtain the Consumer Group ID and the Account of the consumer group. Note The password for the consumer group username is specified when you create the consumer group.
`userName`	The username for the consumer group. Warning If you do not use the client provided in this topic, you must set the username in the `<Username>-<Consumer Group ID>` format. Example: `dtstest-dtsae******bpv`. Otherwise, the connection fails.
`password`	The password for the username.
`initCheckpoint`	The consumption checkpoint. This is the UNIX timestamp from which the SDK client begins consuming data. The value must be a UNIX timestamp. Example: 1620962769. Note You can use consumption checkpoint information in the following scenarios: To resume consumption and prevent data loss after an application interruption, pass the last known consumption checkpoint. When starting the client, you can pass a specific consumption checkpoint to consume data from a desired position.	The consumption checkpoint must be within the data range of the Data Subscription instance (as shown in the figure) and must be converted to a UNIX timestamp. Note You can use a search engine to find a UNIX timestamp converter.
`ConsumerContext.ConsumerSubscribeMode subscribeMode`	The usage mode of the SDK client. Valid values: `ConsumerContext.ConsumerSubscribeMode.ASSIGN`: ASSIGN mode. Only one SDK client in a consumer group can consume Data Subscription data. `ConsumerContext.ConsumerSubscribeMode.SUBSCRIBE`: SUBSCRIBE mode. You can start multiple SDK clients in the same consumer group for disaster recovery.	N/A

Destination Tair parameters

Parameter	Description	How to obtain
`redisUrl`	The endpoint of the Tair instance.	Go to the Connection Information page of your Tair instance. In the Connection Information section, find the endpoint for Public Access.
`redisPort`	The port of the Tair instance.	Go to the Connection Information page of your Tair instance. In the Connection Information section, find the port for Public Access.
`redisPassword`	The username and password for the Tair instance. Important The value must be a string in the <user>:<password> format. For example, if your username is `admin` and your password is `Rp829dlwa`, set this parameter to `admin:Rp829dlwa`.	You specified this when you created the Tair instance. Note If you forgot your password, you can reset it. For more information, see Change or reset a password.

In the top menu bar of IntelliJ IDEA, click Run > Run.

Results

Log in to the source database and insert several records into the tracked source table.
The program consumes the incremental data changes and writes the corresponding data to the destination database.
Log in to the destination database and query the data.
The query confirms that the data was written successfully.