Data Transmission Service:Data shipping

When to use data shipping

Use data shipping in these situations:

• Unsupported source database: The source is a database that DTS doesn't natively support, such as a database provided by another Chinese vendor.

• Non-standard data types: The data is log data or another special type that requires custom encoding before ingestion.

• Restricted direct access: The source database can't expose its credentials or network endpoint directly to DTS for security reasons.

If your source database is already supported by DTS, use a standard data synchronization task instead. See the list of databases supported by DTS.

How it works

Data shipping follows a two-phase setup:

1. Create a data shipping instance in the DTS console. This provisions the channel that receives data from the SDK.

2. Configure and start the SDK. After the instance is created, retrieve the connection parameters from the instance details page, configure the SDK, and start pushing data.

DTS synchronizes the data from the shipping channel to the destination ADB for PostgreSQL instance.

Prerequisites

Before you begin, ensure that you have:

• An ADB for PostgreSQL instance to receive the data. Only ADB for PostgreSQL is supported as a destination. See Create an instance

• A database and schema created in the destination instance. In this guide, the schema is named dts_deliver_test. See the Import data section in "Use SQL to import vector data"

• An AccessKey ID and AccessKey secret for the account that owns the data shipping instance. See Create an AccessKey pair

• (Conditional) If you select Express Connect, VPN Gateway, or Smart Access Gateway as the access method, configure DTS access over VPN Gateway first. See Connect a data center to DTS by using VPN Gateway

• Development capability to implement the data shipping SDK in your codebase

Important notes

• Encoding capabilities are required to use the data shipping SDK to ship data from a data source to DTS.

• The schema name in the destination ADB for PostgreSQL instance must exactly match the database name you enter in the Drop data object configuration step and the dbName parameter in the SDK. A mismatch causes the destination database to fail to receive data.

• After a data shipping instance is created, the number of shards cannot be changed.

• Start the data shipping SDK immediately after the instance is created. If the SDK is not started promptly, incremental data cannot be collected and the instance fails.

Billing

See Billing overview.

Create a data shipping instance

1. Go to the Data Synchronization Tasks page.

   1. Log on to the Data Management (DMS) console.

   2. In the top navigation bar, click Data + AI.

   3. In the left-side navigation pane, choose DTS (DTS) > Data Synchronization.

   Operations may vary based on the mode and layout of the DMS console. See Simple mode and Customize the layout and style of the DMS console. You can also go directly to the Data Synchronization Tasks page in the new DTS console.

2. Select the region where the data synchronization instance resides.

   In the new DTS console, select the region in the top navigation bar.

3. Click Create Task. In the Create Task wizard, configure the source and destination databases.

   Section	Parameter	Description
   N/A	Task Name	A name for the DTS task. DTS generates a name automatically. Specify a descriptive name to make the task easy to identify. The name does not need to be unique.
   Source Database	Select a DMS database instance	Do not select an existing instance. Configure the source parameters manually for the data shipping task.
   	Database Type	Select Data Shipping.
   	Access Method	The access method of the source. In this example, Public IP Address is selected. If you select Express Connect, VPN Gateway, or Smart Access Gateway, also configure the VPC and vSwitch parameters.
   	Instance Region	The region where the source database resides. If the region is not listed, select the geographically closest region.
   Destination Database	Select a DMS database instance	Select an existing database instance, or leave blank and configure the parameters below manually. If you select an existing instance, DTS populates the parameters automatically.
   	Database Type	Select AnalyticDB for PostgreSQL.
   	Access Method	Select Alibaba Cloud Instance.
   	Instance Region	The region where the destination ADB for PostgreSQL instance resides.
   	Instance ID	The ID of the destination ADB for PostgreSQL instance.
   	Database Name	The name of the database in the destination instance that receives data.
   	Database Account	The account for the destination instance. The account must have read and write permissions on the destination database. See Create and manage a database account.
   	Database Password	The password for the database account.

4. Click Test Connectivity and Proceed. DTS automatically adds its server CIDR blocks to the whitelist of Alibaba Cloud database instances and to the security group rules of ECS-hosted databases. For self-managed databases in a data center or hosted by a third-party provider, manually add the DTS server CIDR blocks to the database whitelist. See Add the CIDR blocks of DTS servers.

   Warning

   Adding DTS server CIDR blocks to your database whitelist or security group rules introduces security exposure. Before proceeding, take preventive measures such as strengthening credentials, restricting exposed ports, authenticating API calls, and auditing the whitelist regularly. Alternatively, connect the database to DTS through Express Connect, VPN Gateway, or Smart Access Gateway.

5. Configure the objects to ship and advanced settings.

   Parameter	Description
   Processing Mode of Conflicting Tables	Precheck and Report Errors: checks whether the destination contains tables with the same names as the source. If identical names exist, the precheck fails and the task cannot start. To resolve naming conflicts without deleting or renaming destination tables, use object name mapping. See Map object names. Ignore Errors and Proceed: skips the precheck for identical table names. If the source and destination schemas match and a record has the same primary key or unique key value, full synchronization retains the existing destination record while incremental synchronization overwrites it. If the schemas differ, initialization may fail or only some columns are synchronized. Proceed with caution.
   Capitalization of Object Names in Destination Instance	Controls the capitalization of database, table, and column names in the destination. Defaults to DTS default policy. See Specify the capitalization of object names in the destination instance.
   Drop data object configuration	Configure the source databases and tables to ship: 1. Click Add Library. In the New Database dialog box, enter the source database name. This name must match the schema name of the destination ADB for PostgreSQL instance. In this example, enter dts_deliver_test. To add another database, click Add next to the existing entry. 2. Click OK. 3. Click the icon next to the database to expand the list. 4. Click Add Table next to Table. In the Add Table dialog box, enter the table names. The table name must match the tableName parameter in the SDK. In this example, enter tab1, tab2, and tab3. 5. Click OK. 6. (Optional) Configure table and column name mappings: click Edit next to a table, modify Table Name for table mappings, then clear Synchronize All Tables and modify Column Name and Map column name for column mappings. The Column Name value corresponds to the name in the createField method of FakeSource.java. The Map column name value is the column name in the destination instance. Click to add more columns. Click OK when done.

6. Click Next: Advanced Settings and configure the advanced parameters.

   Parameter	Description
   Dedicated Cluster for Task Scheduling	By default, DTS schedules the task to a shared cluster. To use a dedicated cluster, purchase one separately. See What is a DTS dedicated cluster?
   Set Alerts	Whether to enable alerting. No: alerting disabled. Yes: configure the alert threshold and notification settings. See Configure monitoring and alerting when you create a DTS task.
   Retry Time for Failed Connections	How long DTS retries after a connection failure, in minutes. Valid values: 10–1440. Default: 720. Set this to more than 30 minutes. If DTS reconnects within this window, the task resumes. Otherwise, the task fails. Note If multiple tasks share the same source or destination, the shortest configured retry time applies to all. You are charged for the instance during retry periods.
   Retry Time for Other Issues	How long DTS retries after DDL or DML operation failures, in minutes. Valid values: 1–1440. Default: 10. Set this to more than 10 minutes. This value must be less than Retry Time for Failed Connections.
   Enable Throttling for Incremental Data Synchronization	Whether to throttle incremental synchronization. If enabled, configure RPS of Incremental Data Migration and BPS of Incremental Data Migration to reduce load on the destination.
   Environment Tag	A tag to identify the DTS instance. Select based on your environment.
   Configure ETL	Whether to enable extract, transform, and load (ETL). Yes: enter data processing statements in the code editor. See Configure ETL in a data migration or data synchronization task. No: ETL disabled. See What is ETL?

7. Save the task and run a precheck.

   • To preview the API parameters for this configuration, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.

   • Click Next: Save Task Settings and Precheck.

   DTS runs a precheck before starting the task. If the precheck fails, click View Details next to each failed item, fix the issues, and click Precheck Again. If an alert is triggered for an item that can be safely ignored, click Confirm Alert Details, then click Ignore in the dialog box, click OK, and click Precheck Again. Ignoring an alert may result in data inconsistency.

8. Wait until Success Rate reaches 100%, then click Next: Purchase Instance.

9. On the Purchase Instance page, configure billing and instance settings.

   Parameter	Description
   Billing Method	Subscription: pay upfront for a fixed term. More cost-effective for long-term use. Pay-as-you-go: billed hourly. Suitable for short-term use. Release the instance when no longer needed to avoid unnecessary charges.
   Number of Shards	The number of partitions in the destination topic. This cannot be changed after the instance is created.
   Resource Group Settings	The resource group for the instance. Default: default resource group. See What is Resource Management?
   Instance Class	The synchronization specification, which determines performance. See Instance classes of data synchronization instances.
   Subscription Duration	Available only for the Subscription billing method. Valid values: 1–9 months, or 1, 2, 3, or 5 years.

10. Read and select Data Transmission Service (Pay-as-you-go) Service Terms.

11. Click Buy and Start, then click OK in the confirmation dialog box.

The task appears in the task list. Start the data shipping SDK immediately after the instance is created.

Configure and start the SDK

After creating the instance, retrieve the connection parameters and configure the SDK.

<dependency>
  <groupId>com.aliyun.dts.deliver</groupId>
  <artifactId>dts-deliver-client</artifactId>
  <version>1.0.0</version>
</dependency>