ApsaraDB for ClickHouse provides the data migration feature that supports full data migration and incremental data migration. You can start a data migration task on a destination cluster. This feature allows you to migrate data between clusters of the same kernel version or from a cluster to another cluster of a later version. This topic describes how to migrate data between clusters in the ApsaraDB for ClickHouse console.
Prerequisites
Both the source and destination clusters are ApsaraDB for ClickHouse clusters.
Usernames and passwords of database accounts are created for the source and destination clusters.
The replicas of the source and the destination clusters must be consistent.
The source and destination clusters must be deployed in the same region and use the same virtual private cloud (VPC).
If you enable tiered storage of hot data and cold data for the source cluster, you must also enable tiered storage of hot data and cold data for the destination cluster. If you disable tiered storage of hot data and cold data for the source cluster, you must also disable tiered storage of hot data and cold data for the destination cluster.
Each local table in the source cluster must correspond to a unique distributed table.
The storage space of the destination cluster must be greater than or equal to 1.2 times the storage space of the source cluster.
The IP address of your source ApsaraDB for ClickHouse cluster is added to the IP address whitelist of your destination ApsaraDB for ClickHouse cluster.
The IP address of your destination ApsaraDB for ClickHouse cluster is added to the IP address whitelist of your source ApsaraDB for ClickHouse cluster.
You can execute the select * from system.clusters;
statement to query the IP address of an ApsaraDB for ClickHouse cluster. For more information about how to configure a whitelist, see Configure a whitelist.
Precautions
You cannot migrate data from a cluster of a later version to a cluster of an earlier version.
Before you migrate data, make sure that no management operations, such as scale-outs, upgrades, or downgrades, are being performed on the source and destination clusters.
Full data migration and incremental data migration are supported between ApsaraDB for ClickHouse clusters.
For non-MergeTree tables such as external tables and log tables, only the schemas are migrated. The data is not migrated. Therefore, if the source cluster contains non-MergeTree tables and you migrate data from the source cluster to the destination cluster, the non-MergeTree tables in the destination cluster have only a table schema and no specific business data.
Impacts of data migration
During the migration, you can read data from or write data to the source cluster of V20.8 or later. To ensure that the migration task is completed in a timely manner, we recommend that you stop writing data to the source cluster when 95% of the data has been migrated. For the source cluster of V20.3 or earlier, you can read data but cannot write data until the migration is complete.
In addition, you cannot read data from or write data to the destination cluster until the migration is complete.
Supported versions
The following table describes the version mappings between source clusters and destination clusters.
Source cluster version | Destination cluster version |
19.15 | 19.15, 20.3, 20.8, 21.8, and 22.8 |
20.3 | 20.3, 20.8, 21.8, and 22.8 |
20.8 | 20.8, 21.8, and 22.8 |
21.8 | 21.8 and 22.8 |
22.8 | 22.8 |
Procedure
Log on to the ApsaraDB for ClickHouse console.
On the Clusters page, click the Default Clusters tab, and click the ID of the destination cluster.
In the left-side navigation pane of the page that appears, click Migrate Instance.
On the Cluster page that appears, click Create Migration Task.
NoteIf no data is migrated to the selected cluster, the message "No data found." is displayed.
In the Select the source and destination clusters step, select a source cluster ID from the Instance ID drop-down list, and configure the Database Account and Database Password parameters of the source and destination clusters.
Click Test Connectivity and Proceed to check whether the two clusters are connected.
If the connection is established, proceed to the next step.
If the connection fails to be established: If the message Source cluster account or password error. appears, modify the Database Account or Database Password parameters.
After the source and destination clusters are connected, proceed to the Migration Content step. Click Next: Pre-detect and Start Synchronization.
NoteYou can migrate the following objects from the source cluster: the cluster, databases, tables, data dictionaries, materialized views, user permissions, and cluster configurations.
After the migration task is configured, the system pre-checks the migration configuration and starts the migration task in the backend.
If no issues are detected during the precheck, click Completed.
If an issue is detected, follow the on-screen instructions to resolve the issue, and then migrate the data again. The following table describes the precheck items.
Item
Description
Cluster Status Detection
Before you migrate data, make sure that no management operations, such as scale-outs, upgrades, or downgrades, are being performed on the source and destination clusters. If management operations are being performed on the source and destination clusters, the system cannot start a migration task.
Storage Space Detection
Before the system starts a migration task, the system checks the storage capacity of the source and destination clusters. Make sure that the storage space of the destination cluster is greater than or equal to 1.2 times the storage space of the source cluster.
Local Table and Distributed Table Detection
If no distributed table is created for a local table or multiple distributed tables are created for the same local table, the precheck fails. You must delete redundant distributed tables to ensure that the distributed table of each local table is unique.
After the migration task is complete, you can view the migration task on the Migrate Cluster page.
NoteIf Migration Status is Completed, data is migrated from the source cluster to the destination cluster.
References
For information about how to migrate data from a self-managed ClickHouse cluster to an ApsaraDB for ClickHouse cluster, see Migrate data from a self-managed ClickHouse cluster to an ApsaraDB for ClickHouse cluster.