Data Transmission Service:Synchronize data from RDS for MySQL to DataHub

Billing

Limitations

• Initial full data synchronization is not supported. DTS does not synchronize existing data from the source instance to DataHub.

• Only table-level data synchronization is supported.

• Do not perform DDL operations on source tables during synchronization. Otherwise, the task fails.

Supported SQL operations

Procedure

1. Purchase a data synchronization instance. Purchase process.

   Note

   When you make the purchase, select MySQL as the source instance, DataHub as the target instance, and One-way Synchronization as the synchronization topology.

2. Log on to the DTS console.

   Note

   If you are automatically redirected to the Data Management (DMS) console, you can click the icon in the lower-right corner and then click to return to the classic DTS console.

3. In the left-side navigation pane, click Data Synchronization.

4. At the top of the Synchronization Tasks page, select the region where your destination instance is located.

5. Find the data synchronization task that you purchased and click Configure Task.

6. Configure the source and destination instances.

   

   Section	Parameter	Description
   N/A	Synchronization Task Name	DTS auto-generates a task name. Specify a descriptive name for easy identification. Uniqueness is not required.
   Source Instance Details	Instance Type	Select the source database type. This example uses RDS Instance. Note If the source is a self-managed MySQL database, complete the preparatory steps in Preparation overview.
   	Instance Region	The region selected during purchase. Cannot be changed.
   	Instance ID	Select the ID of the source RDS instance.
   	Database Account	Enter the database account of the source RDS instance. Note If the source RDS instance runs MySQL 5.5 or MySQL 5.6, you do not need to configure Database Account and Database Password.
   	Database Password	Enter the password for the database account.
   	Connection Method	Select Non-encrypted or SSL-encrypted. If you select SSL-encrypted, enable SSL encryption for the RDS instance first. Use a cloud-native certificate to quickly enable SSL encryption. Important Currently, the Encryption parameter is available only for regions in the Chinese mainland and the China (Hong Kong) region.
   Destination Instance Details	Instance Type	Fixed to DataHub. Cannot be changed.
   	Instance Region	The region selected during purchase. Cannot be changed.
   	Project	Select the Project of the DataHub instance.

7. In the lower-right corner of the page, click Set Whitelist and Next.

   If the source or destination database is an Alibaba Cloud database instance, such as an ApsaraDB RDS for MySQL or ApsaraDB for MongoDB instance, DTS automatically adds the CIDR blocks of DTS servers to the IP address whitelist of the instance. If the source or destination database is a self-managed database hosted on an Elastic Compute Service (ECS) instance, DTS automatically adds the CIDR blocks of DTS servers to the security group rules of the ECS instance, and you must make sure that the ECS instance can access the database. If the self-managed database is hosted on multiple ECS instances, you must manually add the CIDR blocks of DTS servers to the security group rules of each ECS instance. If the source or destination database is a self-managed database that is deployed in a data center or provided by a third-party cloud service provider, you must manually add the CIDR blocks of DTS servers to the IP address whitelist of the database to allow DTS to access the database. For more information, see Whitelist DTS server IP addresses.

   Warning

   Adding the public IP address blocks of the DTS service, either automatically or manually, may pose security risks. Using this product, you acknowledge that you understand and accept the potential security risks and that you must implement basic security measures. These measures include, but are not limited to, strengthening password security, limiting the ports open to each CIDR block, using authentication for internal API calls, and regularly checking and restricting unnecessary CIDR blocks. Alternatively, you can connect through a private network using a leased line, VPN Gateway, or Smart Access Gateway.

8. Configure the synchronization policy and objects.

   

   Parameter	Description
   Initial Synchronization	Select Initial Schema Synchronization. Note After you select Initial Schema Synchronization, DTS synchronizes the schema of the selected objects (such as table schemas) to the destination DataHub instance during initialization.
   Select synchronization objects	In the Source Objects box, click the objects that you want to synchronize, and then click the icon to move them to the Selected Objects box. Note You can select tables as synchronization objects. Object names remain unchanged by default. To rename objects in the destination, use the object name mapping feature. Set the name of a synchronization object in the destination instance.
   Enable new naming rules for additional columns	DTS adds additional columns to the destination topic. If an additional column name conflicts with an existing column name, the task fails. Set Enable new naming rules for additional columns to Yes or No. Warning Evaluate whether additional column names conflict with existing column names before selecting a rule. Otherwise, the task may fail or data may be lost. Names and definitions of additional columns.
   Rename databases and tables	Change the names of synchronized objects in the destination instance. For more information, see Map databases, tables, and columns.
   Replicate temporary tables during DMS_ONLINE_DDL process	If you use Data Management (DMS) to perform online DDL changes on the source database, you can choose whether to synchronize the temporary tables generated by the DDL changes. Yes: Synchronizes the temporary tables generated by online DDL changes. Note If a large amount of temporary table data is generated by online DDL changes, the data synchronization task may be delayed. No: Does not synchronize the temporary tables generated by online DDL changes. Only the original DDL operations from the source database are synchronized. Note This option causes tables in the destination database to be locked.
   Retry time for connection failures	If DTS cannot connect to the source or destination instance, it retries for 720 minutes (12 hours) by default. You can also specify a custom retry duration. If DTS reconnects to the source or destination instance within the specified duration, the synchronization task automatically resumes. Otherwise, the task fails. Note You are billed for task run time during connection retries. Customize the retry duration based on your business needs, or release the DTS instance as soon as the source and destination instances are released.

9. Optional: In the Selected Objects box, hover over the name of a topic that you want to synchronize, click the Edit icon that appears, and then set the shard key (the key used for partitioning) in the dialog box that appears.

   

10. After completing the preceding configurations, click Precheck and Start in the lower-right corner of the page.

    Note

    • A precheck runs before the synchronization task starts, and you can only start the task after it passes.

    • If the precheck fails, click the icon next to the failed item to view the details.

      • You can fix the issues based on the cause and run the precheck again.

      • If you do not need to fix the items that triggered warnings, you can click Ignore or Ignore Warnings and Rerun Precheck to skip the warnings and run the precheck again.

11. After the Precheck dialog box displays Precheck Passed, close the Precheck dialog box. The synchronization task starts automatically.

12. Wait for the task to finish initialization and enter the Synchronizing state.

    You can view the status of the data synchronization task on the Data Synchronization page.

DataHub topic schema

The following table describes the additional columns.

dts_before_flag and dts_after_flag

The values of dts_before_flag and dts_after_flag vary by operation type:

• INSERT

  All column values are the newly inserted values. dts_before_flag = N, dts_after_flag = Y.

  

• UPDATE

  DTS splits an UPDATE into two log entries with the same dts_record_id, dts_operation_flag, and dts_utc_timestamp.

  The first entry records the pre-update value (dts_before_flag = Y, dts_after_flag = N). The second entry records the post-update value (dts_before_flag = N, dts_after_flag = Y).

  For example, for a record where dts_id is 10006, dts_db_name is dtstestdata, and dts_table_name is customer, the value of dts_address is updated from Qujing to Hangzhou.

• DELETE

  All column values are the pre-deletion values. dts_before_flag = Y, dts_after_flag = N.

  

Next steps

After the synchronization task is configured, you can analyze the synchronized data in DataHub using Realtime Compute for Apache Flink.