Data Transmission Service:Synchronize data from a self-managed Oracle database to an ApsaraMQ for Kafka instance

Source database limitations

Objects to synchronize:

• Tables must have PRIMARY KEY or UNIQUE constraints with all fields unique. Tables without these constraints may produce duplicate records in the destination.

• For Oracle 12c and later, table names cannot exceed 30 bytes.

• If you select tables as objects to synchronize and plan to edit them in the destination (for example, renaming tables or columns), a single task supports up to 1,000 tables. Tasks with more than 1,000 tables fail with a request error. Split the tables across multiple tasks, or synchronize the entire database instead.

Oracle RAC:

• If the source is an Oracle RAC database connected over Express Connect, specify a virtual IP address (VIP) when configuring the task.

• For Oracle RAC databases, use a VIP instead of a Single Client Access Name (SCAN) IP address. After you specify a VIP, node failover for the Oracle RAC database is not supported.

Log retention:

If DTS cannot read redo logs or archive logs, the task fails, or even data inconsistency or data loss may occur. After full data synchronization completes, you can reduce the retention period to more than 24 hours. DTS service-level agreements (SLAs) do not apply if the retention period is shorter than these minimums.

Other source limitations:

• A primary/secondary switchover on the source database while a task is running causes the task to fail.

• Empty strings of the VARCHAR2 type are treated as null values in Oracle. If the corresponding column in the destination has a NOT NULL constraint, the task fails.

• Do not update LONGTEXT fields during synchronization. Doing so causes the task to fail.

• Do not execute DDL statements during schema synchronization or full data synchronization. Doing so causes the task to fail.

Single-record size limit

The maximum size of a single record that DTS can write to Kafka is 10 MB. If a source row exceeds this limit, the task is interrupted.

To avoid this, do not synchronize entire tables that contain large fields. Synchronize only specific columns instead. When configuring the task, exclude columns with large values. If a table with large fields is already included in the task, remove the table, add it again, and then apply filter conditions to exclude the large columns.

Other limitations

• Renamed tables: If you rename a table in the source database and the new name is not in the selected objects, DTS does not synchronize the renamed table's data to Kafka. To resume synchronization, reselect the objects. See Add an object to a data synchronization task.

• Oracle Data Pump: Do not use Oracle Data Pump to write data to the source database during incremental data synchronization. Data loss may occur.

• Performance impact: DTS uses read and write resources from both the source and destination databases during full data synchronization, which may increase server load. Schedule synchronization tasks during off-peak hours to minimize impact.

• Tablespace size: Concurrent INSERT operations during full data synchronization cause table fragmentation. The destination tablespace will be larger than the source after full data synchronization completes.

• Synchronization latency: Latency is calculated based on the timestamp of the latest synchronized record in the destination and the current time in the source. If no DML operation runs on the source for an extended period, the displayed latency may be inaccurate. Run a DML operation on the source to refresh the latency. If you synchronize an entire database, create a heartbeat table that is updated every second to keep the latency accurate.

• Exclusive writes: Write data to the destination only through DTS. Writing through other tools may cause data inconsistency or data loss — for example, when using DMS to perform online DDL operations.

• Kafka scaling: If the destination Kafka cluster is scaled up or down, restart the Kafka cluster to resume synchronization.

Step 1: Go to the Data Synchronization Tasks page

1. Log on to the Data Management (DMS) console.

2. In the top navigation bar, click Data + AI.

3. In the left-side navigation pane, choose DTS (DTS) > Data Synchronization.

Steps may vary depending on the DMS console mode and layout. See Simple mode and Customize the layout and style of the DMS console. You can also go directly to the Data Synchronization Tasks page of the new DTS console.

Step 2: Select the region

On the right side of Data Synchronization Tasks, select the region where the data synchronization instance resides.

In the new DTS console, select the region from the top navigation bar.

Step 3: Configure source and destination databases

Click Create Task. In the Create Data Synchronization Task wizard, configure the following parameters.

Source Database

Destination Database

Step 4: Test connectivity

Click Test Connectivity and Proceed.

DTS automatically adds its CIDR blocks to the whitelists of Alibaba Cloud database instances (such as ApsaraDB RDS for MySQL or ApsaraDB for MongoDB). For ECS-hosted databases, DTS automatically adds its CIDR blocks to the ECS security group rules. If the database is deployed across multiple ECS instances, manually add the DTS CIDR blocks to each ECS instance's security group rules. For on-premises databases or databases from third-party providers, manually add the DTS CIDR blocks to the database whitelist. See CIDR blocks of DTS servers.

Step 5: Configure objects and synchronization settings

Configure the following parameters:

Parameter	Description
Synchronization Types	By default, Incremental Data Synchronization is selected. Also select Schema Synchronization and Full Data Synchronization. After the precheck, DTS synchronizes historical data from the source to the destination as the baseline for incremental synchronization.
Processing Mode of Conflicting Tables	Precheck and Report Errors: DTS checks for tables with identical names in the source and destination. The task cannot start if conflicts are found. To resolve naming conflicts without deleting or renaming destination tables, use the object name mapping feature. See Map object names. Ignore Errors and Proceed: Skips the naming conflict check. If the source and destination share the same schema, DTS retains existing destination records during full data synchronization and overwrites them during incremental data synchronization. If schemas differ, initialization may partially fail or the task may fail entirely. Use with caution.
Data Format in Kafka	The format for data written to Kafka. DTS Avro: data follows the DTS Avro schema. See the schema definition on GitHub. SharePlex JSON: data uses the SharePlex JSON format. See Shareplex Json.
Policy for Shipping Data to Kafka Partitions	The partitioning policy for synchronized data. See Specify the policy for migrating data to Kafka partitions.
Capitalization of Object Names in Destination Instance	The capitalization policy for database, table, and column names in the destination. Default: DTS default policy. See Specify the capitalization of object names in the destination instance.
Source Objects	Select columns, tables, or databases from the Source Objects section and click the icon to move them to Selected Objects. Selecting tables or columns excludes views, triggers, and stored procedures.
Selected Objects	To rename a single object, right-click it. See Map the name of a single object. To rename multiple objects at once, click Batch Edit. See Map multiple object names at a time. To filter specific SQL operations for a table, right-click the table and select the operations. See SQL operations that can be synchronized. To filter rows, right-click a table and set filter conditions. See Set filter conditions. Renaming an object may cause dependent objects to fail synchronization.

Step 6: Configure advanced settings

Click Next: Advanced Settings and configure the following parameters:

Step 7: Save settings and run the precheck

To preview OpenAPI parameters before saving, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.

Click Next: Save Task Settings and Precheck.

DTS runs a precheck before the task can start. If the precheck fails:

• Click View Details next to each failed item, resolve the issue, and run the precheck again.

• If an alert is triggered for an item that can be ignored, click Confirm Alert Details, then click Ignore in the dialog box, click OK, and then click Precheck Again. Ignoring an alert may cause data inconsistency.

Step 8: Purchase a synchronization instance

Wait until the Success Rate reaches 100%, then click Next: Purchase Instance.

On the purchase page, configure the following parameters:

Step 9: Start the task

Read and select Data Transmission Service (Pay-as-you-go) Service Terms, then click Buy and Start. In the dialog box, click OK.

The task appears in the task list. Monitor its progress from there.