Data Transmission Service:Synchronize data from AnalyticDB for PostgreSQL to Alibaba Cloud Message Queue for Kafka

Step 1: Go to the Data Synchronization page

Use one of the following methods:

DTS console

1. Log on to the DTS console.

2. In the left-side navigation pane, click Data Synchronization.

3. In the upper-left corner, select the region where the synchronization task resides.

DMS console

The actual operations may vary based on the mode and layout of the DMS console. For more information, see Simple mode and Customize the layout and style of the DMS console.

1. Log on to the DMS console.

2. In the top navigation bar, move the pointer over Data + AI and choose DTS (DTS) > Data Synchronization.

3. From the drop-down list to the right of Data Synchronization Tasks, select the region where the synchronization instance resides.

Step 2: Configure source and destination databases

1. Click Create Task.

2. Configure the source database parameters.

   Parameter	Description
   Task Name	The name of the DTS task. DTS generates a name automatically. Specify a descriptive name to make it easy to identify the task. The name does not need to be unique.
   Select Existing Connection	If the source instance is already registered with DTS, select it from the drop-down list. DTS automatically fills in the connection parameters. Otherwise, configure the parameters manually. Note In the DMS console, select the instance from the Select a DMS database instance drop-down list.
   Database Type	Select AnalyticDB for PostgreSQL.
   Access Method	Select Alibaba Cloud Instance.
   Instance Region	Select the region where the source instance resides.
   Replicate Data Across Alibaba Cloud Accounts	Select No for synchronization within the same Alibaba Cloud account.
   Instance ID	Select the instance ID of the source AnalyticDB for PostgreSQL instance.
   Database Name	Enter the name of the database that contains the data to be synchronized.
   Database Account	Enter the database account. The account must have read permissions on the objects to be synchronized.
   Database Password	Enter the password for the database account.

3. Configure the destination database parameters.

   Parameter	Description
   Select Existing Connection	If the destination instance is already registered with DTS, select it from the drop-down list. DTS automatically fills in the connection parameters. Otherwise, configure the parameters manually. Note In the DMS console, select the instance from the Select a DMS database instance drop-down list.
   Database Type	Select Kafka.
   Access Method	Select Alibaba Cloud Instance.
   Instance Region	Select the region where the destination Kafka instance resides.
   Kafka Instance ID	Select the ID of the destination Kafka instance.
   Encryption	Select Non-encrypted or SCRAM-SHA-256 based on your security requirements.
   Topic	Select the topic to receive the synchronized data.
   Use Kafka Schema Registry	Kafka Schema Registry is a metadata service that provides a RESTful interface for storing and retrieving Avro schemas. Select No to skip, or Yes and enter the URL or IP address of your Schema Registry.

4. Click Test Connectivity and Proceed.

   - DTS server CIDR blocks must be added to the security settings of both source and destination databases. For more information, see Add DTS server IP addresses to a whitelist. - If the source or destination database is a self-managed database and its Access Method is not set to Alibaba Cloud Instance, click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

Step 3: Configure synchronization objects

1. In the Configure Objects step, set the following parameters. Configure object mapping in the destination Kafka instance (optional)

   1. In the Selected Objects section, hover over the destination topic name at the table level.

   2. Click Edit next to the topic name.

   3. In the Edit Table dialog box, configure the mapping.

      Note

      At the database level, the Edit Schema dialog box appears with fewer parameters. At the table level, the Edit Table dialog box appears.

      If you synchronize at a granularity other than the entire database, Destination Database Name (Name of target Topic) and Number of Partitions cannot be modified in the Edit Schema dialog box.

      Parameter	Description
      Name of target Topic	The destination topic to which the source table is synchronized. Defaults to the Topic selected in the destination database configuration. Important: If the destination is an ApsaraMQ for Kafka instance, the topic must already exist. If it does not exist, the task fails. If the destination is a self-managed Kafka cluster and schema synchronization is included, DTS attempts to create the topic. Changing the Name of target Topic directs writes to the new topic.
      Filter Conditions	For more information, see Set filter conditions.
      Number of Partitions	The number of partitions for the destination topic.
      Column	Edit the column names written to the destination topic.

   - To select the SQL operations to synchronize at the database or table level, right-click the object in Selected Objects and select the desired operations. - If you use object name mapping, other objects that depend on the mapped object may fail to synchronize.

   Parameter	Description
   Synchronization Types	By default, Incremental Data Synchronization is selected. Also select Schema Synchronization and Full Data Synchronization. After the precheck, DTS synchronizes historical data from the source to the destination, which serves as the basis for incremental data synchronization. Note Schema Synchronization is not supported when the destination Kafka instance's Access Method is Alibaba Cloud Instance.
   Processing Mode of Conflicting Tables	Precheck and Report Errors (default): checks whether the destination contains tables with the same names as those in the source. If identical names exist, the precheck fails and the task cannot start. Note Use the object name mapping feature to rename tables in the destination if conflicts exist. For more information, see Map object names. Ignore Errors and Proceed: skips the name conflict check. Warning If you select this option, data inconsistency may occur. During full data synchronization, existing records in the destination are retained when a primary key or unique key conflict is detected. During incremental data synchronization, existing records are overwritten. If the source and destination have different schemas, some columns may not be synchronized, or the task may fail.
   Data Format in Kafka	Select the data format for synchronized messages. DTS Avro: parse data according to the DTS Avro schema definition. See also the DTS Avro deserialization example. Canal JSON: see Canal JSON for parameter descriptions and examples.
   Kafka Data Compression Format	Select the compression format for Kafka messages. LZ4 (default): lower compression ratio, higher speed. GZIP: higher compression ratio, lower speed. Note GZIP consumes more CPU resources. Snappy: medium compression ratio and speed.
   Policy for Shipping Data to Kafka Partitions	Select a partition policy based on your requirements.
   Message acknowledgement mechanism	Select a message acknowledgment mechanism based on your requirements.
   Topic That Stores DDL Information	Select the topic to store DDL information. If no topic is selected, DDL information is stored in the data topic by default.
   Capitalization of Object Names in Destination Instance	The capitalization of database names, table names, and column names in the destination. The default is DTS default policy. For more information, see Specify the capitalization of object names in the destination instance.
   Source Objects	Select objects from Source Objects and click the right arrow icon to move them to Selected Objects. Objects can be selected at the table level.
   Selected Objects	No additional configuration is required for a basic setup. To map source tables to specific Kafka topics, partitions, or columns, follow the steps below.

2. Click Next: Advanced Settings and configure the advanced parameters.

   Parameter	Description
   Dedicated Cluster for Task Scheduling	By default, DTS schedules the task to the shared cluster. To improve stability, purchase a dedicated cluster. For more information, see What is a DTS dedicated cluster.
   Enable Encryption For Data Transmission	Enabling encrypted transmission may affect synchronization performance. Select based on your security requirements. Default: No.
   Retry Time for Failed Connections	The retry window for failed connections. Valid values: 10 to 1440 minutes. Default: 720 minutes. Set this to more than 30 minutes. If DTS reconnects within this window, the task resumes. Otherwise, the task fails. Note If multiple tasks share the same source or destination database, the shortest retry window takes precedence. DTS instance fees continue to accrue during retries. Set this value based on your business requirements and release the instance promptly when the source or destination is released.
   Retry Time for Other Issues	The retry window for DDL or DML operation failures. Valid values: 1 to 1440 minutes. Default: 10 minutes. Set this to more than 10 minutes. Important This value must be smaller than Retry Time for Failed Connections.
   Enable Throttling for Full Data Synchronization	Limits the read rate during full data synchronization to reduce load on the source and destination. Configure Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s) as needed. This parameter is available only when Full Data Synchronization is selected.
   Enable Throttling for Incremental Data Synchronization	Limits the write rate during incremental data synchronization. Configure RPS of Incremental Data Synchronization and Data synchronization speed for incremental synchronization (MB/s) as needed.
   Environment Tag	(Optional) Tag to identify the environment of the instance.
   Scene Label	(Optional) Tag to identify the usage scenario of the instance. Scenario tags do not affect task operation.
   Configure ETL	Specifies whether to enable the extract, transform, and load (ETL) feature. Select Yes to enter data processing statements in the code editor. For more information, see Configure ETL in a data migration or data synchronization task. Select No to skip ETL configuration. For more information about ETL, see What is ETL?
   Monitoring and Alerting	Configures alerts for task failures or synchronization latency exceeding the threshold. Select Yes and configure the alert threshold and notification settings. For more information, see Configure monitoring and alerting when you create a DTS task.

Step 4: Run the precheck

1. To preview the API parameters for this task configuration, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.

2. Click Next: Save Task Settings and Precheck.

   - DTS performs a precheck before the task can start. The task starts only after it passes the precheck. - If the precheck fails, click View Details next to each failed item, address the issues, and rerun the precheck. - If an alert is triggered during the precheck: - For alerts that cannot be ignored: click View Details, fix the issue, and rerun the precheck. - For alerts that can be ignored: click Confirm Alert Details, click Ignore in the details panel, confirm, and then click Precheck Again. Ignoring alerts may result in data inconsistency.

Step 5: Purchase the instance

1. Wait until the Success Rate reaches 100%, then click Next: Purchase Instance.

2. On the buy page, configure the billing and instance parameters.

   Parameter	Description
   Billing Method	Subscription: pay upfront for a set period. More cost-effective for long-term use. Pay-as-you-go: billed hourly. Suitable for short-term use. Release the instance when no longer needed to stop charges.
   Resource Group Settings	The resource group for the instance. Default: default resource group. For more information, see What is Resource Management?
   Instance Class	DTS provides instance classes with varying synchronization speeds. Select a class based on your requirements. For more information, see Instance classes of data synchronization instances.
   Subscription Duration	Available only for the Subscription billing method. Options: 1–9 months, 1 year, 2 years, 3 years, or 5 years.

3. Read and select Data Transmission Service (Pay-as-you-go) Service Terms.

4. Click Buy and Start, then click OK in the dialog box.

The task appears in the task list. Monitor its progress there.