Configure cache update policies by using change tracking - Data Transmission Service

The change tracking feature of Data Transmission Service (DTS) can work with ApsaraDB for MongoDB and ApsaraDB for Redis to implement high-reliability, low-latency cache update policies. This topic describes how to configure a cache update policy. In the following example, the source database is an ApsaraDB RDS for MySQL instance and the destination database is an ApsaraDB for Redis instance.

Prerequisites

A change tracking task is configured and a consumer group is added. For more information, see Track data changes from an ApsaraDB RDS for MySQL instance and Create consumer groups.
An ApsaraDB for Redis instance that has Internet access is created. For more information, see Step 1: Create an ApsaraDB for Redis instance and Apply for a public endpoint for an ApsaraDB for Redis instance.
Note
You must add the client IP address or CIDR block to the whitelist of the ApsaraDB for Redis instance. For more information, see Configure whitelists.

Background information

To accelerate access to a business and improve the concurrent read performance, a caching layer is built in the business architecture. The read requests are routed to the caching layer. The memory reading mechanism of the caching layer improves the read performance. To ensure data integrity, the updated business data is persisted in ApsaraDB RDS for MySQL. In this case, you can configure a cache update policy to ensure that the cached data is updated when the business data is updated.

Procedure

The following example uses IntelliJ IDEA and SDK demo code to show how to consume tracked data.

Download the SDK demo package and decompress the package.
Use IntelliJ IDEA to open the decompressed SDK demo package in a Maven project.
In the /subscribe-demo/src/main/java/com/alibaba/dts/demo/ path, double-click the DTSMySQL2RedisDemo.java file.

Modify the parameters in the main function.

In the following figure, the values in green need to be modified. For example, your broker url of the String brokerUrl = "your broker url"; parameter needs to be modified.

Change tracking task parameters

Parameter	Description	Method to obtain the parameter value
`brokerUrl`	The endpoint and port number of the change tracking instance. Note If you track data changes over internal networks, the network latency is minimal. This is applicable if the Elastic Compute Service (ECS) instance on which you deploy the SDK client resides on the classic network or in the same virtual private cloud (VPC) as the change tracking instance. We recommend that you do not use a public endpoint.	On the Change Tracking page of the DTS console, find the change tracking instance that you want to manage and click the instance ID. In the left-side pane, click the Basic Information tab to obtain the endpoint and port number of the change tracking instance in the Network section.
`topic`	The name of the tracked topic of the change tracking instance.	On the Change Tracking page of the DTS console, find the change tracking instance that you want to manage and click the instance ID. In the left-side pane, click the Basic Information tab to obtain the name of the tracked Topic in the Basic Information section.
`sid`	The consumer group ID.	On the Change Tracking page of the DTS console, find the change tracking instance that you want to manage and click the instance ID. In the left-side pane, click the Consume Data tab to view Consumer Group ID/Name and the Account information of the consumer group. Note The password of the consumer group account is specified when you create the consumer group.
`userName`	The username of the consumer group account. Warning If you are not using the SDK client that is described in this topic, you must specify this parameter in the following format: `<Username>-<Consumer group ID>`. Example: `dtstest-dtsae******bpv`. Otherwise, the connection fails.
`password`	The password of the consumer group account.
`initCheckpoint`	The consumer offset, which is the timestamp generated when the SDK client consumes the first data record. The value is a UNIX timestamp. Example: 1620962769. Note The consumer offset can be used in the following scenarios: If the consumption process is interrupted, you can specify the consumer offset to resume data consumption. This prevents data loss. When you start the change tracking client, you can specify the consumer offset to consume data based on your business requirements.	The consumer offset of consumed data must be within the data range of the change tracking instance. The consumer offset must be converted to a UNIX timestamp. Note You can view the data range of the change tracking instance in the Data Range column on the Change Tracking page. You can use a search engine to obtain a UNIX timestamp converter.
`ConsumerContext.ConsumerSubscribeMode subscribeMode`	The mode in which you use the SDK client. Valid values: `ConsumerContext.ConsumerSubscribeMode.ASSIGN`: In ASSIGN mode, you can start only one SDK client in a consumer group to consume tracked data. `ConsumerContext.ConsumerSubscribeMode.SUBSCRIBE`: In SUBSCRIBE mode, you can start multiple SDK clients in a consumer group at a time for high redundancy.	N/A

ApsaraDB for Redis instance parameters

Parameter	Description	Method to obtain the parameter value
`redisUrl`	The endpoint to the ApsaraDB for Redis instance.	On the Instance Information tab of the ApsaraDB for Redis instance, find the Connection Information section and view the endpoint of the ApsaraDB for Redis instance in the Public Access row.
`redisPort`	The port number of the ApsaraDB for Redis instance.	On the Instance Information tab of the ApsaraDB for Redis instance, find the Connection Information section and view the port number of the ApsaraDB for Redis instance in the Public Access row.
`redisPassword`	The username and password of the ApsaraDB for Redis account. Important The database password is in the <user>:<password> format. For example, if the username of the account that you use to log on to the source ApsaraDB for Redis instance is admin and the password is Rp829dlwa, the database password is admin:Rp829dlwa.	The username and password are specified when the ApsaraDB for Redis instance is created. Note If you forget the password, you can reset the password. For more information, see Change or reset the password.

In the top menu bar of IntelliJ IDEA, choose Run > Run to run the program.

Result

Log on to the source database and insert several data entries into the source table of the change tracking task.
The result shows that the program tracks the changes in the data and writes the incremental data to the destination database.
Log on to the destination database and query the data.
The result shows that the incremental data is written to the destination database.