Data Transmission Service:Migrate RDS for MySQL to a self-managed Kafka cluster

Precautions

• During a full data migration, DTS consumes read and write resources on the source and destination databases, increasing their load. If your databases have poor performance, low specifications, or high workloads (for example, if the source database has many slow SQL queries or tables without primary keys, or if deadlocks occur in the destination database), the increased load can strain your databases or even cause service interruptions. Perform the data migration during off-peak hours, such as when the CPU utilization of both databases is below 30%.

• If the source database does not have a primary key or unique constraint, and no fields have unique values, duplicate data may occur in the destination database.

• Only data tables can be migrated.

Billing

Procedure

1. Log on to the DTS console.

   Note

   If you are automatically redirected to the Data Management (DMS) console, you can click the icon in the lower-right corner and then click to return to the classic DTS console.

2. In the left-side navigation pane, click Data Migration.

3. At the top of the Migration Tasks page, select the region of the destination cluster.

4. In the upper-right corner of the page, click Create Data Migration Task.

5. Configure the source and destination databases.

   Section	Parameter	Description
   N/A	Task name	DTS automatically generates a task name, but we recommend that you specify a descriptive one for easier identification. The name does not need to be unique.
   Source database	Instance type	Select RDS.
   	Instance region	Select the region where the source RDS instance is located.
   	Instance ID	Select the ID of the source RDS instance.
   	Database account	Enter the database account for the source RDS instance. The account must have the REPLICATION CLIENT, REPLICATION SLAVE, SHOW VIEW, and SELECT permissions on all migration objects.
   	Database password	Enter the password for the database account.
   	Connection method	Select Non-encrypted or SSL-encrypted as needed. If you select SSL-encrypted, you must first enable SSL encryption for the RDS instance. For more information, see Set SSL encryption. Important Currently, the Encryption can be set only in the Chinese mainland and China (Hong Kong) regions.
   Destination database	Instance type	Select User-Created Database with Public IP Address.
   	Instance region	This parameter does not need to be configured.
   	Database type	Select Kafka.
   	Hostname or IP address	Enter the endpoint of the self-managed Kafka cluster. In this topic, a public endpoint is used as an example.
   	Port	Enter the service port of the Kafka cluster. The default port is 9092.
   	Database account	Enter the username for the Kafka cluster. You can leave this blank if authentication is not enabled for the Kafka cluster.
   	Database password	Enter the password for the Kafka cluster user. You can leave this blank if authentication is not enabled for the Kafka cluster.
   	Topic	Click Get Topic List and select the destination topic from the drop-down list.
   	Kafka version	Select the version of the destination Kafka cluster.
   	Connection method	Based on your business and security requirements, select Non-encrypted or SCRAM-SHA-256.

6. After you complete the configuration, click Set Whitelist and Next in the lower-right corner of the page.

   If the source or destination is an Alibaba Cloud database instance, such as ApsaraDB RDS for MySQL or ApsaraDB for MongoDB, DTS automatically adds the DTS server IP addresses to the instance's whitelist. For a self-managed database on an ECS instance, DTS adds the DTS server IP addresses to the security rules. You must still ensure that the database itself does not restrict access from the DTS server IP addresses. For a database cluster on multiple ECS instances, you must manually add the DTS server IP addresses to the security rules of each instance. For an on-premises database or a database from another cloud provider, you must also manually add the DTS server IP addresses to allow access. For a list of DTS server IP addresses, see IP addresses of DTS servers.

   Warning

   Adding the public CIDR blocks of DTS servers, whether automatically or manually, may introduce security risks. By using this product, you acknowledge and accept these potential risks. You are responsible for implementing basic security measures, including but not limited to using strong passwords, restricting open ports, using authentication for internal API calls, regularly reviewing and restricting unnecessary network segments, or connecting through private networks such as Express Connect, VPN Gateway, or Smart Access Gateway.

7. Configure migration types, policies, and objects.

   Setting	Description
   Migration type	Select Schema Migration, Full Data Migration, and Incremental Data Migration. Important To ensure data consistency, if you perform a full data migration without selecting Incremental Data Migration, do not write new data to the source database.
   Data format in Kafka	Data migrated to the Kafka cluster is stored in Avro or Canal JSON format. For more information, see Data storage formats in message queues.
   Kafka partitioning strategy	Select a migration strategy based on your business requirements. For more information, see Kafka partition synchronization policies.
   Migration objects	In the Available Objects box, click the table that you want to migrate, and then click the icon to move it to the Selected Objects box. Note DTS automatically maps the table name to the topic name that you selected in Step 5. To change the destination topic for the migration, see Map objects.
   Object name mapping	To change the names of migration objects in the destination instance, use the object name mapping feature. For more information, see Map objects.
   Connection retry duration	Set the duration for DTS to retry connecting to the source or destination database if a connection fails. The default retry duration is 12 hours. You can also specify a custom duration. If DTS reconnects to the databases within the specified time, the migration task automatically resumes. Otherwise, the task fails. Note You are charged for the DTS task during the connection retry period. We recommend that you customize the retry duration based on your business needs or release the DTS instance as soon as possible after the source and destination database instances are released.
   Copy temporary tables for DMS online DDL	Specify whether to replicate temporary tables that are generated when you use Data Management (DMS) to perform online DDL operations on the source database. Yes: Replicates the data from temporary tables generated by online DDL operations. Note If online DDL operations generate a large amount of data in the temporary tables, migration latency may increase. No: Does not replicate data from temporary tables. Only the original DDL data from the source database is migrated. Note This option may cause table locks on the destination database.

8. After you complete the configuration, click Precheck and Start in the lower-right corner of the page.

   Note

   • Before the migration task starts, DTS runs a precheck. The task can start only after it passes the precheck.

   • If the precheck fails, click the icon next to the failed item to view details.

     • Fix the issues as prompted and run the precheck again.

     • If you do not need to fix the warning items, you can select Ignore and then click Ignore Warnings and Rerun Precheck to run the precheck again.

9. After the task passes the precheck, click Next.

10. In the Confirm Settings dialog box that appears, select a Instance Class and select the Data Transmission Service (pay-as-you-go) Service Terms checkbox.

11. Click Buy and Start to begin the migration.

    • Schema migration + Full data migration

      Allow the task to complete automatically. Stopping it manually may result in incomplete data.

    • Schema migration + Full data migration + Incremental data migration

      The migration task does not stop automatically. You must stop it manually.

      Important

      Choose an appropriate time to stop the task manually, such as during off-peak hours or when you are ready to switch your business to the destination cluster.

      1. Wait until the migration task enters the Incremental Data Migration phase and the status shows Undelayed. Then, stop writing data to the source database for several minutes. During this time, the status of Incremental Data Migration may show a latency.

      2. Wait for the Incremental Data Migration status to show Undelayed again. Then, manually stop the migration task. Select the check box of the target migration task, and click Pause in the batch operation bar at the bottom of the page.