Data Transmission Service:Synchronize data from RDS MySQL to Tablestore

Precautions

• During initial full data synchronization, DTS consumes read and write resources on the source and destination databases, which may increase the database load. We recommend that you evaluate the performance impact and synchronize data during off-peak hours.
• Data Definition Language (DDL) operations are not synchronized. If you perform a DDL operation during synchronization, you must first remove the synchronization object, delete the table in the Tablestore instance, and then add the synchronization object.
• The number of tables to be synchronized cannot exceed the Tablestore instance limit of 64. If you need more tables, request a limit increase for your destination Tablestore instance.
• The names of tables or columns to be synchronized must comply with the Tablestore naming conventions:
  • A table or column name must consist of uppercase or lowercase letters, digits, and underscores (_). It must start with a letter or an underscore (_).
  • A table or column name must be 1 to 255 characters in length.

Billing

Initial synchronization types

Before you begin

1. Create a RAM user and grant the AliyunOTSFullAccess permission to the user to manage Tablestore.
   1. Log on to the RAM console.
   2. Create a RAM user.
   3. On the left-side navigation pane, choose Identities > Users.
   4. Find the target RAM user and click Add Permissions in the Actions column.
   5. In the dialog box that appears, enter AliyunOTSFullAccess in the search box, and then click the policy name in the search results.
      In the Select Permissions section, click the System Policy tab and set the authorization scope to All resources of the Alibaba Cloud account.
   6. Click OK.
   7. Click Complete.
2. Create an AccessKey pair for the RAM user. For more information, see Create an AccessKey pair.
3. Optional: If you use a RAM user to configure the data synchronization task, you must also grant the AliyunDTSDefaultRole permission to the user. For more information, see Grant DTS permissions to access cloud resources.
   Note If you use your Alibaba Cloud account to configure the data synchronization task, you can skip this step.

Procedure

1. Purchase a data synchronization task. For more information, see Purchase process.
   Note On the purchase page, set the source instance to MySQL, the destination instance to Tablestore, and the synchronization topology to One-way Synchronization.

2. Log on to the Data Transmission Service (DTS) console.

3. In the left-side navigation pane, click Data Synchronization.

4. In the upper part of the Data Synchronization Tasks page, select the region in which the destination instance resides.

5. Find the data synchronization task and click Configure Task in the Actions column.

6. Configure the source and destination databases.

   Section	Parameter	Description
   N/A	Synchronization Task Name	DTS automatically generates a task name. We recommend that you use a descriptive name for easy identification. The name does not have to be unique.
   Source Instance	Instance Type	Select RDS Instance.
   	Instance Region	The region of the source instance selected when you purchased the data synchronization instance. This value cannot be changed.
   	Instance ID	Select the ID of the source ApsaraDB RDS for MySQL instance.
   	Database Account	Enter the database account for the source ApsaraDB RDS for MySQL instance. The account must have REPLICATION CLIENT, REPLICATION SLAVE, and SHOW VIEW permissions, as well as SELECT permission on all objects to be synchronized. Note If the database engine of the source ApsaraDB RDS for MySQL instance is MySQL 5.5 or MySQL 5.6, you do not need to configure Database Account and Database Password.
   	Database Password	Enter the password for the database account.
   	Connection Method	Select Non-encrypted or SSL-encrypted based on your requirements. If you select SSL-encrypted, you must enable SSL encryption for the ApsaraDB RDS for MySQL instance first. For more information, see Configure SSL encryption. Important Currently, the Encryption parameter is available only for regions in the Chinese mainland and the China (Hong Kong) region.
   Destination Instance	Instance Type	Select Tablestore.
   	Instance Region	The region of the destination instance selected when you purchased the data synchronization instance. This value cannot be changed.
   	Instance ID	Select the ID of the destination Tablestore instance.
   	AccessKey ID	Enter the AccessKey ID. For information about how to create an AccessKey pair, see Create an AccessKey pair.
   	AccessKeySecret	Enter the AccessKey secret that corresponds to the AccessKey ID.

7. In the lower-right corner of the page, click Set Whitelist and Next.

   If the source or destination database is an Alibaba Cloud database instance, such as an ApsaraDB RDS for MySQL or ApsaraDB for MongoDB instance, DTS automatically adds the CIDR blocks of DTS servers to the IP address whitelist of the instance. If the source or destination database is a self-managed database hosted on an Elastic Compute Service (ECS) instance, DTS automatically adds the CIDR blocks of DTS servers to the security group rules of the ECS instance, and you must make sure that the ECS instance can access the database. If the self-managed database is hosted on multiple ECS instances, you must manually add the CIDR blocks of DTS servers to the security group rules of each ECS instance. If the source or destination database is a self-managed database that is deployed in a data center or provided by a third-party cloud service provider, you must manually add the CIDR blocks of DTS servers to the IP address whitelist of the database to allow DTS to access the database. For more information, see Whitelist DTS server IP addresses.

   Warning

   Adding the public IP address blocks of the DTS service, either automatically or manually, may pose security risks. Using this product, you acknowledge that you understand and accept the potential security risks and that you must implement basic security measures. These measures include, but are not limited to, strengthening password security, limiting the ports open to each CIDR block, using authentication for internal API calls, and regularly checking and restricting unnecessary CIDR blocks. Alternatively, you can connect through a private network using a leased line, VPN Gateway, or Smart Access Gateway.

8. Configure the synchronization policy and objects.
   1. Configure the synchronization policy.

      Parameter	Description
      Initial Synchronization	Select Initial schema synchronization, Initial full data synchronization, and Incremental data synchronization. For more information, see Initial synchronization types.
      Handling of Tables with the Same Name	Precheck and Report Errors: Checks for tables in the destination database with the same names as tables in the source database. If no such tables exist, the precheck passes. If conflicting tables exist, an error is reported during the precheck phase, and the data synchronization task is not started. Note If you cannot easily delete or rename the conflicting table in the destination database, you can map object names to avoid conflicts. Ignore Errors and Proceed: Skips the check for tables with the same name in the destination database. Warning If you select Ignore Errors and Proceed, data inconsistencies may occur, which poses risks to your services. For example: If the table schemas are identical, records in the destination database with matching primary keys are retained during initial synchronization but are overwritten during incremental synchronization. If the table schemas are different, initial synchronization may fail or synchronize only a subset of columns.
      Merge Tables	Select Yes: In OLTP scenarios, sharding is often used to improve table response times. You can use the table merging feature of DTS to synchronize multiple source tables with identical schemas (sharded tables) into a single table in the Tablestore instance. Note DTS adds a column named __dts_data_source of the varchar type to the destination table to store the data source. DTS writes values in the <DTS instance ID>:<Source database name>.<Source table name> format to identify the source table. Example: dts********:dtstestdata.customer1. This feature is applied at the instance level. To merge some tables but not others, you must create separate data synchronization tasks. Select No: This is the default option.
      Operation Types	Select the operation types to synchronize based on your business needs. By default, all operation types are selected.
      Error Handling Strategy	Select a policy for handling data write errors: Skip Block
      Data Write Mode	Update Row: Uses PutRowChange for row-level updates. Overwrite Row: Uses UpdateRowChange for row-level overwrites.
      Batch Write Operation	The API operation used for batch writes. BulkImportRequest: Writes data offline. BatchWriteRowRequest: Writes data in batches. We recommend BulkImportRequest for higher read/write efficiency and lower costs on the Tablestore instance.
      More Settings
      └ Queue Size	The queue length for the data write process of the Tablestore instance.
      └ Threads	The number of callback handling threads for the data write process of the Tablestore instance.
      └ Concurrency	The concurrency limit for requests to the Tablestore instance.
      └ Buckets	The number of concurrent buckets for sequential incremental writes. A higher value improves concurrent write performance. Note This value must be less than or equal to the value of Concurrency.

   2. Configure synchronization objects.

      Parameter

      Description

      Select Synchronization Objects

      In the Source Objects box, select the tables to migrate (up to 64), and then click the to move them to the Selected Objects box. By default, synchronized object names remain unchanged. To use different names in the destination instance, use the object name mapping feature. For more information, see Map object names to avoid conflicts. Warning If you set Merge Tables to Yes in the previous step and select multiple source tables, you must use the object name mapping feature to map them to the same table name in the Tablestore instance. Otherwise, data inconsistencies may occur.

   3. Click Next.
9. Optional: In the Selected Objects box, hover over a table to be synchronized, click Edit that appears next to the table name, and then set the data type for each column in the Tablestore instance.
   In the column configuration area of the Edit Table dialog box, click the TargetType drop-down list for a target column to select Integer, Binary, String, Double, or Boolean.
10. Configure the primary key columns for the tables to be synchronized to the Tablestore instance.
    Click Define Primary Key for the target table. In the dialog box that appears, set Primary Key Name, Primary Key Type, and Primary Key Order. After configuration, the Definition Status column changes to Defined.

    Note For more information about primary keys in Tablestore, see Primary keys.

11. In the lower-right corner of the page, click Next: Precheck and Start Task.

    Note

    • Before you can start the data synchronization task, DTS performs a precheck. You can start the data synchronization task only after the task passes the precheck.

    • If the task fails to pass the precheck, you can click the icon next to each failed item to view details. After you troubleshoot the issues based on the causes, run a precheck again.

12. Close the Precheck dialog box after the following message is displayed: Precheck Passed. Then, the data synchronization task starts.

13. Wait until the initial synchronization is complete and the data synchronization task enters the Synchronizing state.

    You can view the status of the data synchronization task on the Data Synchronization Tasks page.