Data Transmission Service (DTS) tracks incremental data changes from an RDS PostgreSQL instance and makes them available for downstream consumption via consumer clients. This guide walks you through creating a change tracking task for an RDS PostgreSQL source.
Prerequisites
Before you begin, ensure that you have:
An RDS PostgreSQL source instance. For more information, see Create an ApsaraDB RDS for PostgreSQL instance.
The
wal_levelparameter of the RDS PostgreSQL instance set tological. For more information, see Set instance parameters.A Virtual Private Cloud (VPC) and a vSwitch to run the change tracking instance. For more information, see Create and manage a VPC.
For supported PostgreSQL versions, see Overview of change tracking solutions.
Limitations
Source database
| Limitation | Details |
|---|---|
| Table constraints | Source tables must have PRIMARY KEY or UNIQUE constraints, with all fields unique. Without these, tracked data changes may contain duplicates. |
| Maximum tables per task | 500 tables. If a task tracks more than 500 tables, a request error occurs. Split large table sets across multiple tasks, or track at the database level instead. |
| Write-Ahead Logging (WAL) | WAL must be enabled. Set the wal_level parameter to logical. WAL logs must be retained for more than 24 hours. If logs are purged earlier, DTS may fail to retrieve them, which can cause task failure or, in extreme cases, data inconsistency or loss. This scenario is not covered by the DTS Service-Level Agreement (SLA). |
| Read-only or temporary instances | The instance must have logs that record transactions. |
| Logical Replication Slot Failover | RDS PostgreSQL must support and have Logical Replication Slot Failover enabled. For more information, see Logical Replication Slot Failover. |
| Single record size (tracking) | If a single updated record exceeds 256 MB, the change tracking instance may fail and cannot be recovered. If this occurs, reconfigure the change tracking instance. |
Other limitations
| Limitation | Details |
|---|---|
| Data Definition Language (DDL) tracking | To track schema updates (CREATE, DROP, ALTER), create triggers and functions in the source database. For more information, see Implement incremental DDL migration for PostgreSQL using triggers and functions. |
| Databases per task | A single change tracking task tracks data changes from one database only. To track multiple databases, create a separate task for each. |
| Single row size (consumption) | If a single tracked row exceeds 16 MB, the data cannot be consumed and may cause an out-of-memory (OOM) error on the consumer client. |
| FLOAT and DOUBLE precision | DTS retrieves FLOAT and DOUBLE column values using ROUND(COLUMN,PRECISION). Default precision: 38 digits for FLOAT, 308 digits for DOUBLE. Verify that these defaults meet your requirements. |
| Time zone | Time data types that include a time zone are converted to UTC+8. |
| Instance recovery | If a change tracking instance fails, DTS helpdesk attempts recovery within 8 hours. During recovery, operations such as restarting the instance and adjusting parameters may be performed. Only DTS instance parameters are modified; database parameters are not modified. |
Replication slot behavior
DTS creates a replication slot with the prefix dts_sync_ in the source database to replicate data. Understand the lifecycle of these slots before you start:
DTS automatically purges stale replication slots every 90 minutes to prevent excessive disk usage.
When a change tracking task is released or fails, DTS automatically removes its replication slot.
If a primary/secondary switchover occurs on the RDS PostgreSQL instance, log on to the secondary database and manually remove the replication slot.
DTS adds a heartbeat table named
dts_postgres_heartbeatto the source database to measure tracking latency accurately.
Do not modify the endpoint or zone of the RDS PostgreSQL instance while a change tracking task is running. Doing so causes the task to fail.
Create a change tracking task
Step 1: Go to the Change Tracking Tasks page
Use the DTS console or the DMS console to open the Change Tracking Tasks page.
DTS console
Log on to the DTS console.DTS console
In the left-side navigation pane, click Change Tracking.
In the upper-left corner, select the region where the change tracking instance resides.
DMS console
The exact navigation path may vary depending on your DMS console mode and layout. For more information, see Simple mode and Customize the layout and style of the DMS console.
Log on to the DMS console.DMS console
In the top navigation bar, move your pointer over Data + AI > DTS (DTS) > Change Tracking.
From the drop-down list to the right of Change Tracking Tasks, select the region where the change tracking instance resides.
Step 2: Configure the source database
Click Create Task.
On the task configuration page, read the Limits displayed at the top before proceeding.
WarningSkipping the limits review may cause the task to fail or prevent tracked data from being consumed.
In the Source Database and Consumer Network Type sections, configure the following parameters.
For SSL encryption on a self-managed PostgreSQL database, uploading the CA Certificate is required. Upload the Client Certificate and Private Key of Client Certificate only if you want to use client-side certificate authentication. For details on configuring SSL for RDS PostgreSQL, see SSL encryption.
Category Parameter Description (None) Task Name A name for the change tracking task. DTS assigns a default name automatically. Use a descriptive name to identify the task easily. Names do not need to be unique. Source Database Select Existing Connection If the source instance is already registered with DTS, select it from the drop-down list. DTS fills in the database parameters automatically. Otherwise, configure the parameters below. In the DMS console, select the instance from Select a DMS database instance. Database Type Select PostgreSQL. Access Method Select Alibaba Cloud Instance. Instance Region Select the region where the RDS PostgreSQL instance resides. Replicate Data Across Alibaba Cloud Accounts Select No for same-account access. To track data across Alibaba Cloud accounts, select Yesalert notification settings and specify the Alibaba Cloud Account and RAM Role Name. Configure Resource Access Management (RAM) authorization for the account used to set up the DTS task. For more information, see Configure RAM authorization for cross-account DTS tasks. Instance ID Select the RDS PostgreSQL instance ID. Database Name Enter the name of the database to track. Database Account Enter the privileged account of the RDS PostgreSQL instance. The account must be the owner of the selected database. For more information, see Create an account and Create a database. Database Password The password for the database account. Encryption Select Non-encrypted (default) or SSL-encrypted. For SSL encryption, upload the CA Certificate, and optionally the Client Certificate, Private Key of Client Certificate, and Private Key Password of Client Certificate. Consumer Network Type Network Type Fixed to VPC. Select a VPC and a vSwitch. This setting cannot be changed after the task is created. If your change tracking client runs in a VPC, select the same VPC and vSwitch as your client to minimize latency. For more information, see VPCs. Click Test Connectivity and Proceed.
Make sure DTS server IP addresses are added to the security group or whitelist of your source database. For details, see Add DTS server IP addresses to a whitelist. If the source database uses an access method other than Alibaba Cloud Instance, click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.
Step 3: Select objects to track
On the Configure Objects page, configure the following settings.
Parameter Description Data Change Types Selected by default and cannot be changed. Choose from: Data Update (INSERT, DELETE, UPDATE operations) and Schema Update (CREATE, DROP, ALTER on all object schemas). To track schema updates, create triggers and functions first. For more information, see Implement incremental DDL migration for PostgreSQL using triggers and functions. Source Objects Select one or more objects from Source Objects and click the arrow icon to move them to Selected Objects. Select a database to track all objects in it, including future ones. Select a table to track that table only. To track additional tables later, add them separately. For more information, see Modify the objects for change tracking. Click Next: Advanced Settings and configure the following settings.
When multiple tasks share the same source instance, the shortest Retry Time for Failed Connections value takes precedence across all tasks. DTS charges fees during retry periods. Set the retry window based on your business requirements, and release the DTS instance promptly if the source instance is no longer needed.
Parameter Description Dedicated Cluster for Task Scheduling By default, DTS schedules the task to a shared cluster. To run the task on dedicated hardware, purchase a dedicated cluster. For more information, see What is a DTS dedicated cluster. Retry Time for Failed Connections How long DTS retries when the connection to the source fails. Valid values: 10–1440 minutes. Default: 720 minutes. Set this to at least 30 minutes. If DTS reconnects within this period, the task resumes automatically; otherwise, the task fails. Retry Time for Other Issues How long DTS retries when Data Manipulation Language (DML) or DDL operations fail after the task starts. Valid values: 1–1440 minutes. Default: 10 minutes. We recommend that you set this to greater than 10 minutes. This value must be less than Retry Time for Failed Connections. Environment Tag (Optional) Attach a tag to identify the environment, for example, production or test. Monitoring and Alerting Select Yes to receive alerts when the task fails or latency exceeds a threshold. Configure alert thresholds and notification settings. For more information, see Configure monitoring and alerting when you create a DTS task.
Step 4: Run the precheck
Click Next: Save Task Settings and Precheck. To preview the API parameters for this task configuration, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.
Wait for the precheck to complete.
If an item fails, click View Details next to it, fix the reported issue, and run the precheck again.
If an item shows an alert, either fix it or click Confirm Alert Details > Ignore > OK, then click Precheck Again. Ignoring alert items may cause data inconsistency.
Step 5: Purchase and start the task
After Success Rate reaches 100%, click Next: Purchase Instance.
On the Purchase page, configure the billing settings.
Parameter Description Billing method Subscription: Pay upfront for a fixed term (1–9 months, or 1, 2, 3, or 5 years). More cost-effective for long-term use. Pay-as-you-go: Billed hourly. Suitable for short-term use. Release the instance when you no longer need it to avoid ongoing charges. Resource Group Settings The resource group for the instance. Defaults to default resource group. For more information, see What is Resource Management? Subscription Duration Available for subscription billing only. Select the subscription term and instance count. Read and accept the Data Transmission Service (Pay-as-you-go) Service Terms.
Click Buy and Start. The change tracking task appears in the task list.
What's next
After the change tracking task is running, create consumer groups and connect your downstream clients.
Create and manage consumer groups. For more information, see Create consumer groups.
Consume the tracked data using one of the following methods: