Set a cache update policy by using the change tracking feature - Data Transmission Service

This topic describes how to set a cache update policy for a MongoDB or Redis database when you use the change tracking feature. The cache update policy allows you to achieve high reliability and low latency.

Prerequisites

A change tracking task is configured. For more information, see Track data changes from a PolarDB-X 1.0 instance.

Background information

To accelerate business access and improve concurrent read performance, you can build a caching layer in your business architecture. The read requests are routed to the caching layer. The memory reading mechanism of the caching layer allows you to improve the read performance of your business. To ensure data integrity, the updated business data is persistently stored in ApsaraDB RDS for MySQL. In this case, you can set a cache update policy to ensure that the cached data is updated when the business data is updated.

Before you begin

Create an AccessKey pair, which consists of an AccessKey ID and AccessKey secret. For more information, see Create an AccessKey pair.

Important If you track and consume data as a RAM user, the AliyunDTSFullAccess permission must be granted to the RAM user. For more information, see Use a system policy to authorize a RAM user to manage DTS instances.

Procedure

IntelliJ IDEA (Community Edition 2018.1.4 Windows) is used in this example.

Download a change tracking SDK.
Add the downloaded SDK package to the project dependencies.
1. Open IntelliJ IDEA, choose File > New > Project, and then create a Maven project.
2. In the menu bar, choose File > Project Structure.
3. In the left-side navigation pane, click Modules. On the right side of the page, choose > 1 JARs or directories.
4. Select the downloaded SDK package and add it to the project dependencies.

Perform the following steps to set a cache update policy for a MongoDB or Redis database:

Run the demo code for MongoDB

Double-click the pom.xml file and add the following settings to the file:

<dependencies>
    <dependency>
        <groupId>com.vividsolutions</groupId>
        <artifactId>jts</artifactId>
        <version>1.13</version>
    </dependency>
    <dependency>
        <groupId>org.mongodb</groupId>
        <artifactId>mongo-java-driver</artifactId>
        <version>3.2.0</version>
    </dependency>
</dependencies>

Save the settings and wait for the dependencies to be loaded.
Note We recommend that you enable IntelliJ IDEA to update Maven dependencies.
Create a Java class file named DTSMySQL2Mongo in the Maven project.
Download the demo code for MongoDB.
Note
We recommend that you copy the download URL and open it in Google Chrome to download the demo code.
Copy the downloaded demo code and replace the existing content in the DTSMySQL2Mongo file with the demo code.

Edit the code in the DTSMySQL2Mongo file and set the parameters. Set a cache update policy for a MongoDB database


Parameter	Description
accessKey	Enter the AccessKey ID and AccessKey secret of the Alibaba Cloud account. For more information, see the Before you begin section in this topic.
accessSecret
Subscription_Instance_ID	Enter the ID of the change tracking instance. To obtain the ID of the change tracking instance, perform the following steps:Log on to the DTS console. In the left-side navigation pane, click Change Tracking.
mongUrl	Enter the connection string of the MongoDB database and the name of the authentication database. The format is `<Connection string>:<Service port number>/<Authentication database name>`, for example, `ds-bp********-pub.mongodb.rds.aliyuncs.com:3717/admin`.
mongUserName	Enter the database account of the MongoDB database. Note The database account must have the read and write permissions on the destination database. For example, if the tracked data is written to a MongoDB database named testdata, the database account must have the read and write permissions on the testdata database.
mongUserPassword	Enter the password of the database account.

In the top menu bar of IntelliJ IDEA, choose Run > Run to run the program.

Run the demo code for Redis

Double-click the pom.xml file and add the following settings to the file:

<dependencies>
    <dependency>
        <groupId>com.vividsolutions</groupId>
        <artifactId>jts</artifactId>
        <version>1.13</version>
    </dependency>
    <dependency>
        <groupId>redis.clients</groupId>
        <artifactId>jedis</artifactId>
        <version>2.7.2</version>
    </dependency>
</dependencies>

Save the settings and wait for the dependencies to be loaded.
Note We recommend that you enable IntelliJ IDEA to update Maven dependencies.
Create a Java class file named DTSMySQL2Redis in the Maven project.
Download the demo code for Redis.
Note
We recommend that you copy the download URL and open it in Google Chrome to download the demo code.
Copy the downloaded demo code and replace the existing content in the DTSMySQL2Redis file with the demo code.

Edit the code in the DTSMySQL2Redis file and set the parameters. Set a cache update policy for a Redis database


Parameter	Description
accessKey	Enter the AccessKey ID and AccessKey secret of the Alibaba Cloud account. For more information, see the Before you begin section in this topic.
accessSecret
Subscription_Instance_ID	Enter the ID of the change tracking instance. To obtain the ID of the change tracking instance, perform the following steps:Log on to the DTS console. In the left-side navigation pane, click Change Tracking.
redisPassword	Enter the database password of the Redis database. Note If a password is set for the Redis database, you must change the code in line 73 from `pool = new JedisPool(config, redisUrl, redisPort, 3000);` to `pool = new JedisPool(config, redisUrl, redisPort, 3000,redisPassword)`.
redisPort	Enter the service port number of the Redis database. The default port number is 6379.
redisUrl	Enter the endpoint (domain name or IP address) of the Redis database.

In the top menu bar of IntelliJ IDEA, choose Run > Run to run the program.

When the program runs, it retrieves incremental data from the source database and writes the data to the MongoDB or Redis database.

Result

Log on to the source database and insert several data records into the source table. Then, the program tracks the inserted data records and writes the data records to the destination database.
- Execution result of the demo code for MongoDB
- Execution result of the demo code for Redis
Log on to the destination database and query the inserted data records. Then, the data records are written to the database.
- Query result for the MongoDB database
- Query result for the Redis database