Create a change tracking task for a PolarDB for PostgreSQL (Compatible with Oracle) cluster - Data Transmission Service

Data Transmission Service (DTS) change tracking captures INSERT, DELETE, and UPDATE operations from a PolarDB for PostgreSQL (Compatible with Oracle) cluster in real time. Downstream clients consume the tracked data through consumer groups using the SDK, Flink DTS Connector, or a Kafka client.

Prerequisites

Before you begin, make sure that:

A source PolarDB for PostgreSQL (Compatible with Oracle) cluster is created. For more information, see Create a cluster.
The wal_level parameter is set to logical for the source cluster. This setting enables logical encoding in write-ahead logging (WAL). For more information, see Specify cluster parameters.
If the ApsaraDB RDS for MySQL instance uses the classic network type, an internal endpoint is configured for the instance.

Limitations

Source database limits

Source tables must have PRIMARY KEY or UNIQUE constraints, and all fields must be unique. Otherwise, tracked data changes may contain duplicates.
If you select tables as the objects to track, a single change tracking task supports up to 500 tables. Exceeding this limit causes a request error. To track more than 500 tables, split them across multiple change tracking tasks or track the entire database instead.
WAL requirements:
- The wal_level parameter must be set to logical. This enables logical encoding in WAL. For more information, see Specify cluster parameters.
- WAL logs must be retained for more than 24 hours. If DTS cannot obtain WAL logs, the task may fail, and data inconsistency or loss may occur. If WAL log retention does not meet this requirement, the service reliability and performance stated in the Service Level Agreement (SLA) of DTS may not be guaranteed.
If the source instance is a read-only or temporary instance, make sure that it records transaction logs.

Other limits

Schema updates in the source instance cannot be tracked.
A single change tracking task can track data changes from only one database. To track changes from multiple databases, create a separate change tracking task for each database.
DTS uses the ROUND(COLUMN,PRECISION) function to retrieve values from FLOAT or DOUBLE columns. If you do not specify a precision, DTS uses the following defaults: Make sure that the precision settings meet your business requirements.
- FLOAT: 38 digits
- DOUBLE: 308 digits

Create a change tracking task

Step 1: Open the change tracking page

Log on to the Data Management (DMS) console.
In the top navigation bar, click DTS.
In the left-side navigation pane, choose DTS (DTS) > Change Tracking.

If you use Simple Mode, move the pointer over the icon in the upper-left corner, and then choose All functions > DTS > Change Tracking. For more information, see Customize the layout and style of the DMS console. You can also use the new DTS console.

Step 2: Select a region

To the right of Change Tracking Tasks, select the region where you want to create the change tracking task.

If you use the new DTS console, select the region from the drop-down list to the right of Workbench on the Change Tracking Tasks page.

Step 3: Configure the source database and consumer network type

Click Create Task. On the page that appears, configure the source database instance and the consumer network type.

Warning

After you specify the source database instance, read the Limits displayed in the upper part of the page. Failure to follow these limits may cause the task to fail or prevent tracked data from being consumed.

Source database

Parameter	Description
Task Name	DTS automatically assigns a name. Specify a descriptive name that makes the task easy to identify. The task name does not need to be unique.
Select an existing database connection	The instance to which the source database belongs. If you use an existing instance, DTS automatically fills in the source database parameters. Otherwise, configure the parameters manually.
Database Type	Select PolarDB (Compatible with Oracle).
Access Method	Select Public IP Address.
Instance Region	The region where the PolarDB for PostgreSQL (Compatible with Oracle) cluster resides.
Data Sources	Select Single Data Source. Enter the endpoint of the source cluster or the primary node in the Domain Name or IP Address field, and set the Port Number.
Domain Name or IP Address	The endpoint of the source cluster or the primary node.
Port Number	The service port number of the source cluster. Default value: 1521.
Database Name	The name of the source database.
Database Account	The privileged account of the PolarDB for PostgreSQL (Compatible with Oracle) database.
Database Password	The password of the database account.

Consumer network type

Parameter	Description
Network Type	Fixed to VPC. Select a VPC and a vSwitch. For more information, see VPCs.

The consumer network type cannot be changed after the change tracking task is configured. If your change tracking client is deployed in a VPC, select the same VPC and vSwitch connected to the client to minimize network latency.

Step 4: Test connectivity

Click Test Connectivity and Proceed in the lower part of the page.

If the source database is an Alibaba Cloud database instance (such as ApsaraDB RDS for MySQL or ApsaraDB for MongoDB), DTS automatically adds the CIDR blocks of DTS servers in the corresponding region to the whitelist of the instance. If the source database is hosted on an ECS instance, DTS automatically adds the CIDR blocks to the security group rules of the ECS instance. For self-managed databases in a data center or on a third-party cloud, manually add the CIDR blocks of DTS servers to the security settings of the database. For more information, see Add the CIDR blocks of DTS servers.

Warning

Adding public CIDR blocks of DTS servers to the whitelist of a database instance or to the security group rules of an ECS instance may introduce security risks. Before you proceed, take preventive measures, including but not limited to: strengthening your username and password, restricting exposed ports, authenticating API calls, regularly auditing the whitelist or security group rules, and removing unauthorized CIDR blocks. You can also connect the database to DTS through Express Connect, VPN Gateway, or Smart Access Gateway.

Step 5: Configure tracking objects and advanced settings

Basic settings

Parameter	Description
Data Change Types	Data Update and Schema Update are selected by default and cannot be changed. DTS tracks data changes (INSERT, DELETE, and UPDATE operations) and schema changes on the selected objects.
Source Objects	Select one or more objects from the Source Objects section and click the icon to add them to the Selected Objects section.

You can select tables or databases as the objects for change tracking.

If you select a database, DTS tracks incremental data of all objects in the database, including new objects added later.

If you select a table, DTS tracks only incremental data of that specific table. To track additional tables, add them to the object list. For more information, see Modify the objects for change tracking.

Advanced settings

Parameter	Description
Monitoring and Alerting	Specify whether to enable alerting for this task. When enabled, the specified alert contacts receive notifications if the task fails or latency exceeds the threshold. Options: No (default) or Yes (requires you to set the alert threshold and contacts).
Retry Time for Failed Connections	The retry duration when a connection fails. DTS retries automatically within this period. Valid values: 10 to 1440 minutes. Default: 120 minutes. Set this to more than 30 minutes for best results. If DTS reconnects within the specified period, the task resumes. Otherwise, the task fails.
Configure ETL	Specify whether to enable the extract, transform, and load (ETL) feature. Options: Yes (enter your policy in the code editor) or No. For more information, see What is ETL?.

For Retry Time for Failed Connections:

If an instance serves as the source or destination of multiple change tracking tasks, the shortest retry duration takes precedence.

You are charged for the DTS instance during retries. Set the retry duration based on your business requirements. Release the DTS instance as soon as possible after the source and destination instances are released.

Step 6: Run the precheck

Click Next: Save Task Settings and Precheck in the lower part of the page.

You can move the pointer over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters to view the API parameter settings used to configure the instance.

DTS runs a precheck before starting the change tracking task. The task starts only after it passes the precheck.

If any item fails the precheck, click View Details next to the failed item, troubleshoot the issue based on the error message, and then click Precheck Again.
If an alert item cannot be ignored, click View Details next to the failed item and troubleshoot the issues. Then, run a precheck again.
If an alert item can be ignored, click Confirm Alert Details. In the View Details dialog box, click Ignore, and then click OK. Then click Precheck Again. Ignoring alert items may cause data inconsistency and expose your business to potential risks.

Step 7: Purchase the instance

Wait until Success Rate reaches 100%, and then click Next: Purchase Instance.

On the Purchase page, configure the billing method of the change tracking instance.

Parameter	Description
Billing method	Subscription: Pay upfront when you create the instance. More cost-effective for long-term use, with lower prices for longer durations. Pay-as-you-go: Billed hourly. Best for short-term use. Release the instance when it is no longer needed to reduce costs.
Resource Group Settings	The resource group for the instance. Default: default resource group. For more information, see What is Resource Management?.
Subscription Duration	Available only for the Subscription billing method. Options: 1 to 9 months, or 1, 2, 3, or 5 years.

Step 8: Start the task

Read and select the Data Transmission Service (Pay-as-you-go) Service Terms.
Click Buy and Start to start the change tracking task.

You can view the task progress in the task list.

Next steps

After the change tracking task starts running, create consumer groups to consume the tracked data from downstream clients.

Create and manage consumer groups. For more information, see Create consumer groups.
Consume tracked data using one of the following methods: