ApsaraDB for ClickHouse:Upgrade via data migration

Prerequisites

Before you begin, make sure you have:

• Two Community-compatible Edition clusters, both in Running status

  To migrate between Community-compatible Edition and Enterprise Edition, see Migrate a ClickHouse Community-compatible Edition cluster to an Enterprise Edition cluster.

• A database account and password configured on both clusters

• The same tiered storage configuration for hot and cold data on both clusters

• Both clusters in the same region and Virtual Private Cloud (VPC), with each cluster's IP address added to the other's whitelist

  Run SELECT * FROM system.clusters; to find a cluster's IP address. For whitelist configuration, see Set a whitelist.

• A destination cluster whose major engine version is later than the source cluster's

• Available disk storage on the destination cluster (excluding cold storage) that is at least 1.2x the source cluster's used disk storage (excluding cold storage)

• Each local table in the source cluster paired with exactly one distributed table

For supported versions, see Release notes for ApsaraDB for ClickHouse Community-compatible Edition.

What can and cannot be migrated

Supported migration content:

• Databases, tables (MergeTree engine), materialized views, data dictionaries (SQL-created only), user permissions, and cluster configurations

• Table schemas of non-MergeTree tables (external tables and Log tables) — schemas only, without business data

  If the source cluster contains non-MergeTree tables, the destination cluster will only have their schemas after migration, without any business data. To migrate the business data, you can use the remote function. For more information, see Use the remote function to migrate data.

Not supported:

• Data dictionaries created using XML

• Kafka and RabbitMQ engine tables (business data)

Preparation checklist

Before starting the migration:

• Confirm no management operations (scale-out, upgrade, or downgrade) are running on either cluster.

• Check for XML-created data dictionaries by running: SELECT * FROM system.dictionaries WHERE (database = '') OR isNull(database); — if this returns results, remove those dictionaries before proceeding.

• If a data dictionary accesses an external service, confirm the service is reachable and its whitelist allows access from the destination cluster.

• If a data dictionary uses an internal ClickHouse table as its source with the HOST parameter set to an IP address, the IP changes after migration — manually recreate that data dictionary with the new IP after migration.

• For Kafka and RabbitMQ tables: clear them from the source cluster and recreate them on the destination cluster, or use different consumer groups to avoid data splitting.

• Keep total cold data in the source cluster under 1 TB. Larger cold data volumes significantly extend migration time and may cause the task to fail.

• After the upgrade, update the endpoint in your client settings to point to the destination cluster.

Potential impacts

Source cluster during migration:

• Read and write operations continue normally.

• Data Definition Language (DDL) operations (adding, deleting, or modifying databases and tables) are blocked.

• When the estimated remaining migration time drops to 10 minutes or less, the source cluster automatically pauses writes to keep data consistent:

  • If this happens within the preset write-stop time window, writes stop immediately.

  • If this happens outside the write-stop window and within 5 days of task creation, modify the write-stop time window to continue.

  • If this happens outside the write-stop window and beyond 5 days of task creation, the migration fails. Cancel the task, clear migrated data from the destination cluster, and start over.

• Writes resume automatically when all data is migrated, or when the write-stop time window ends before migration completes.

Destination cluster after migration:

The destination cluster runs frequent merge operations for a period after migration, increasing I/O usage and query latency. Plan for this elevated latency before switching traffic. To estimate how long the merge operations will take, see Calculate the merge duration after migration.

Migration overview

All steps are performed on the destination cluster, not the source cluster.

1. Create a migration task on the destination cluster.

2. Assess whether the migration can complete (required only if source write speed exceeds 20 MB/s).

3. Monitor the migration task.

4. (Optional) Cancel the migration task if needed.

5. (Optional) Modify the write-stop time window.

Step 1: create a migration task

1. Log on to the ApsaraDB for ClickHouse console.

2. On the Clusters page, select the Clusters of Community-compatible Edition tab, then click the ID of the destination cluster.

3. In the left navigation pane, click Data Migration and Synchronization > Migration from ClickHouse.

4. Click Create Migration Task.

5. Configure the source and destination instances, then click Test Connectivity and Proceed.

   If the connection test fails, reconfigure the instances as directed by the error message.

   

6. Review the migration content details, then click Next: Pre-detect and Start Synchronization.

7. The system runs a precheck with the following checks:

   Check	Requirement
   Instance status	No management tasks (scale-out, configuration changes) running on either cluster
   Storage space	Destination disk storage >= 1.2x source disk storage (excluding cold storage)
   Local table and distributed table	Each local table in the source cluster has exactly one distributed table

   • If the precheck succeeds:

     1. Review the impact details on the page.

     2. Set the Time of Stopping Data Writing.

        Set the write-stop window to at least 30 minutes to improve the migration success rate. The end date must be no later than 5 days from today. Schedule the window during off-peak hours to minimize business impact.

     3. Click Completed to create and start the task.

   • If the precheck fails: Resolve the issues as indicated, then retry the migration.

Step 2: assess whether the migration can complete

Skip this step if the source cluster's write speed is less than 20 MB/s.

If the source cluster's write speed exceeds 20 MB/s, check that the destination cluster can keep up:

1. Open View cluster monitoring information and check the Disk throughput metric for the destination cluster.

2. Compare the two write speeds:

   • Destination write speed >= source write speed: The migration has a high success rate. Proceed to Step 3.

   • Destination write speed < source write speed: The migration may fail. Cancel the migration task and perform a manual migration instead.

Step 3: monitor the migration task

1. On the Clusters page, select the Clusters of Community-compatible Edition tab, then click the ID of the destination cluster.

2. In the left navigation pane, click Migration from ClickHouse. The migration list shows the Migration Status, Running Information, and Data Write-Stop Window for each task.

   When the estimated remaining time in the Running Information column drops to 10 minutes or less and the status is Migrating, the write-stop logic triggers. See Potential impacts for what happens in each scenario.

Step 4: (Optional) cancel the migration task

1. On the Clusters page, select the Clusters of Community-compatible Edition tab, then click the ID of the destination cluster.

2. In the left navigation pane, click Migration from ClickHouse.

3. In the Actions column for the target task, click Stop Migration.

4. In the Stop Migration dialog box, click OK.

The task list may not update immediately after cancellation. Refresh the page to check the latest status. After cancellation, the Migration Status changes to Completed. Before starting a new migration, clear the migrated data from the destination cluster to avoid data duplication.

Step 5: (Optional) modify the write-stop time window

1. On the Clusters page, select the Clusters of Community-compatible Edition tab, then click the ID of the destination cluster.

2. In the left navigation pane, click Migration from ClickHouse.

3. In the Actions column for the target task, click Modify Data Write-Stop Time Window.

4. In the Modify Data Write-Stop Time Window dialog box, select a new Write-stop Time. The same rules that apply when creating the task also apply here.

5. Click OK.

What's next

After confirming that all business data has been successfully migrated to the destination cluster, delete the source cluster.

For deletion instructions, see Delete a cluster.