ApsaraDB for ClickHouse:Migrate data between ApsaraDB for ClickHouse clusters

Prerequisites

• Both the source and destination clusters are ApsaraDB for ClickHouse clusters.

• The usernames and passwords of database accounts are created for the source ApsaraDB for ClickHouse cluster and the destination ApsaraDB for ClickHouse cluster.

• The source ApsaraDB for ClickHouse cluster and the destination ApsaraDB for ClickHouse cluster are deployed in the same region and use the same virtual private cloud (VPC).

• If you enable tiered storage of hot data and cold data for the source cluster, you must also enable tiered storage of hot data and cold data for the destination cluster. If you disable tiered storage of hot data and cold data for the source cluster, you must also disable tiered storage of hot data and cold data for the destination cluster.

• Each local table in the source ApsaraDB for ClickHouse cluster corresponds to a unique distributed table.

• The storage space of the destination ApsaraDB for ClickHouse cluster is greater than or equal to 1.2 times the storage space of the source ApsaraDB for ClickHouse cluster.

• The IP address of your source ApsaraDB for ClickHouse cluster is added to the IP address whitelist of your destination ApsaraDB for ClickHouse cluster. The IP address of your destination ApsaraDB for ClickHouse cluster is added to the IP address whitelist of your source ApsaraDB for ClickHouse cluster.

  Note

  You can execute the SELECT * FROM system.clusters; statement to query the IP address of an ApsaraDB for ClickHouse cluster. For more information about how to configure a whitelist, see Configure a whitelist.

Precautions

• Data migration between two ApsaraDB for ClickHouse clusters requires that the versions of the two clusters be the same or the version of the destination cluster be later than that of the source cluster.

• Before you migrate data, make sure that no management operations, such as scale-out, upgrade, or downgrade operations, are being performed on the source ApsaraDB for ClickHouse cluster and the destination ApsaraDB for ClickHouse cluster.

• Full data migration and incremental data migration are supported between ApsaraDB for ClickHouse clusters.

• You can migrate the following objects from the source ApsaraDB for ClickHouse cluster: the cluster, databases, tables, data dictionaries, materialized views, user permissions, and cluster configurations.

  For non-MergeTree tables such as external tables and log tables, only the schemas are migrated. Therefore, if the source ApsaraDB for ClickHouse cluster contains non-MergeTree tables and you migrate data from the source ApsaraDB for ClickHouse cluster to the destination ApsaraDB for ClickHouse cluster, the non-MergeTree tables in the destination ApsaraDB for ClickHouse cluster have only a table schema and no specific business data. You can use the remote function to migrate business data. For more information, see Use the remote function to migrate data.

Impacts of data migration

• During the migration, you can read data from or write data to the source ApsaraDB for ClickHouse cluster of V20.8 or later. To ensure that the migration task is completed in a timely manner, we recommend that you stop writing data to the source ApsaraDB for ClickHouse cluster when 95% of the data has been migrated. For the source ApsaraDB for ClickHouse cluster of V20.3 or earlier, you can read data but cannot write data until the migration is complete.

• You cannot read data from or write data to the destination ApsaraDB for ClickHouse cluster until the migration is complete.

Procedure

1. Log on to the ApsaraDB for ClickHouse console.

2. On the Clusters page, click the Default Instances tab, find the cluster that you want to manage, and then click the ID of the cluster.

3. In the left-side navigation pane of the page that appears, click Migrate Instance.

4. On the page that appears, click Create Migration Task.

   

5. In the Select the source and destination instances step, select a source cluster ID from the Instance ID drop-down list and set the Database Account and Database Password parameters in the Source Instance Information section. Then, set the Database Account and Database Password parameters in the Destination Instance Information section.

6. Click Test Connectivity and Proceed to check whether the two clusters are connected.

7. After the source cluster and destination cluster are connected, proceed to the Migration Content step. Click Next: Pre-detect and Start Synchronization.

8. After the migration task is configured, the system pre-checks the migration configuration and starts the migration task in the backend.

   • If no issues are detected during the precheck, click Completed.

   • If an issue is detected, follow the instructions to resolve the issue and migrate the data again. The following table describes the precheck items.

     Item	Description
     Cluster Status Detection	Before you migrate data, make sure that no management operations, such as scale-out, upgrade, or downgrade operations, are being performed on the source ApsaraDB for ClickHouse cluster and the destination ApsaraDB for ClickHouse cluster. If management operations are being performed on the source cluster and the destination cluster, the system cannot start a migration task.
     Storage Space Detection	Before the system starts a migration task, the system checks the storage space of the source cluster and the destination cluster. Make sure that the storage space of the destination cluster is greater than or equal to 1.2 times the storage space of the source cluster.
     Local Table and Distributed Table Detection	If no distributed table is created for a local table or multiple distributed tables are created for the same local table, the precheck fails. You must delete redundant distributed tables to ensure that the distributed table of each local table is unique.

9. After the migration task is complete, you can view the migration task on the Migrate Instance page.

   

   Note

   If Completed is displayed in the Migration Status column, data is migrated from the source cluster to the destination cluster.

References

For more information about how to migrate data from a self-managed ClickHouse cluster to an ApsaraDB for ClickHouse cluster, see Migrate data from a self-managed ClickHouse cluster to an ApsaraDB for ClickHouse cluster.