ApsaraDB for ClickHouse:Scale an ApsaraDB for ClickHouse cluster

Constraints

• ZooKeeper specification changes are supported only for clusters created after December 1, 2021. For pricing, see Pricing for ZooKeeper specifications of the Community-Compatible Edition.

• Upgrading storage capacity or storage type does not restart the cluster (applies only to clusters created after December 1, 2021). Changing cluster specifications or ZooKeeper specifications restarts the cluster.

• Server Specifications or ZooKeeper Specifications cannot be changed at the same time as an Upgrade of Storage Capacity or Storage Type.

• Changing ZooKeeper specifications during peak hours may cause inconsistencies between table metadata and actual data. Perform this change during off-peak hours or when write operations are stopped.

Impact by cluster type

Procedure

1. Log on to the ApsaraDB for ClickHouse console.

2. In the upper-left corner, select the region where your cluster resides.

3. On the Clusters page, click the Clusters of Community-compatible Edition tab.

4. Find the target cluster. In the Actions column, click Change.

5. In the Change dialog box, select Scale Up or Scale Down, and click OK.

6. On the Upgrade/Downgrade page, select the desired configurations. By default, a cluster has a ZooKeeper service with 4 cores and 8 GB of memory. To check for resource bottlenecks, go to the Monitoring and Alerting page and view ZooKeeper metrics on the Cluster Monitoring panel. Upgrade the ZooKeeper specifications if the defaults are insufficient.

7. Click Buy Now and complete the payment.

8. On the The order is complete page, click Console.

9. Check the cluster status in the Status column of the Clusters of Community-compatible Edition list.

   • Storage Capacity changes take effect immediately. The cluster status remains Running.

   • Server Specifications or ZooKeeper Specifications changes take 10 to 15 minutes. The status changes from Changing Specification to Running when the operation completes.

Post-scaling behavior

After changing cluster or ZooKeeper specifications, the cluster restarts. Restart duration depends on the number of databases, tables, and the volume of cold data. High-frequency merge operations may run for a period after the change, increasing I/O usage and potentially increasing request latency. For information about estimating merge duration, see Calculate the merge duration after migration.

Scale-out methods

Scale-in methods

Migration scope

During scale-out with data migration or standard scale-in, the following data is migrated to the new cluster configuration.

Supported:

• Databases, dictionaries, and materialized views

• Table schemas for all tables except Kafka and RabbitMQ engine tables

• Data from MergeTree engine family tables (incremental migration)

Not supported:

• Kafka and RabbitMQ engine tables and their data

• Data from non-MergeTree tables (such as external tables and Log family engine tables)

Constraints

• DDL operations are prohibited during the entire scale-out or scale-in process.

• CPU and memory usage increases during the operation. Estimated overhead: less than 5 cores and 20 GB of memory per node.

• After scaling, high-frequency merge operations continue for a period, increasing I/O usage and potentially increasing request latency. For information about estimating merge duration, see Calculate the merge duration after migration.

• After scaling out, internal cluster node IP addresses change. If your application connects to specific node IP addresses, retrieve the VPC CIDR block again. For details, see Obtain the VPC CIDR block of a cluster.

Step 1: Handle Kafka and RabbitMQ engine tables

Skip this step if your cluster has no Kafka or RabbitMQ engine tables.

1. Log on to the cluster. For connection instructions, see Connect to a ClickHouse cluster using DMS.

2. Query for Kafka and RabbitMQ engine tables:

      SELECT * FROM `system`.`tables` WHERE engine IN ('RabbitMQ', 'Kafka');

3. Back up the CREATE TABLE statement for each table:

      SHOW CREATE TABLE <table_name>;

4. Delete the Kafka and RabbitMQ engine tables. > Important: When deleting a Kafka table, also delete the materialized views that reference it. Otherwise, the scale-out or scale-in operation fails.

Step 2: Back up data from non-MergeTree tables

Skip this step if your cluster has no non-MergeTree tables with data that needs to be preserved.

1. Identify non-MergeTree tables whose data requires migration:

      SELECT
          `database` AS database_name,
          `name` AS table_name,
          `engine`
      FROM `system`.`tables`
      WHERE (`engine` NOT LIKE '%MergeTree%')
        AND (`engine` != 'Distributed')
        AND (`engine` != 'MaterializedView')
        AND (`engine` NOT IN ('Kafka', 'RabbitMQ'))
        AND (`database` NOT IN ('system', 'INFORMATION_SCHEMA', 'information_schema'))
        AND (`database` NOT IN (
            SELECT `name`
            FROM `system`.`databases`
            WHERE `engine` IN ('MySQL', 'MaterializedMySQL', 'MaterializeMySQL',
                               'Lazy', 'PostgreSQL', 'MaterializedPostgreSQL', 'SQLite')
        ))

2. Back up the data from the identified tables. For instructions, see Back up data to OSS.

Step 3: Scale out or scale in from the console

1. Log on to the ApsaraDB for ClickHouse console.

2. In the upper-left corner, select the region where your cluster resides.

3. On the Clusters page, select the Clusters of Community-compatible Edition tab.

4. Find the target cluster. In the Actions column, click Change.

5. In the Change dialog box, select Scale Out or Scale In, and click OK.

6. In the check window, review the check status. The Scale Out window selects Migration Expansion by default. To use Simple Expansion, click Previous, select Simple Expansion in the Scale-out dialog box, and click Next. Common check failure reasons:

   • If the check passes, click Next.

   • If the check fails, fix the reported issues and click Retry Check. Click Next after the check passes.

   Failure	Resolution
   Missing unique distributed table	A local table has no corresponding distributed table. Create one.
   Corresponding distributed table is not unique	A local table has more than one distributed table. Keep only one.
   Kafka/RabbitMQ engine tables exist	Delete these tables first (see Step 1).
   Non-replicated *MergeTree tables in a master-replica instance	Data is inconsistent between replicas. Resolve the inconsistency.
   Distributed and local table columns are inconsistent	Align the column definitions.
   Table is missing on some nodes	Create matching tables on all shards. For materialized view inner tables, rename the inner table and rebuild the materialized view. For details, see The inner table of a materialized view is inconsistent across shards.

7. On the Upgrade/Downgrade page, configure the Server Nodes count and the write suspension window. Write suspension window requirements:

   • Set the write suspension time to at least 30 minutes.

   • The operation must complete within 5 days. The end date of Stopping Data Writing for the source cluster must be no later than the current date plus 5 days.

   • Set the write suspension window during off-peak hours to reduce business impact.

8. Click Buy Now and complete the payment.

9. On the The order is complete page, click Console.

10. Check the cluster status in the Status column of the Clusters of Community-compatible Edition list. The operation completes when the status changes from Scaling to Running. A scale-out or scale-in operation takes at least 30 minutes. The exact duration depends on data volume.

Step 4: Recreate Kafka and RabbitMQ engine tables

Skip this step if you did not delete any Kafka or RabbitMQ engine tables in Step 1.

Log on to the cluster and run the CREATE TABLE statements you backed up in Step 1. For connection instructions, see Connect to a ClickHouse cluster using DMS.

Step 5: Restore data to non-MergeTree tables

Skip this step if you did not back up any non-MergeTree table data in Step 2.

Log on to the cluster and import the data you backed up in Step 2. For instructions, see Import data from OSS.

Post-scaling behavior

• Merge operations: High-frequency merge operations continue for a period after the operation, increasing I/O usage and potentially increasing request latency. For information about estimating merge duration, see Calculate the merge duration after migration.

• IP addresses: After scaling out, internal cluster node IP addresses change. Update your application configuration if it connects to specific node IPs. For details, see Obtain the VPC CIDR block of a cluster.

• Verify data: After the operation completes and the cluster status returns to Running, verify that your data and tables are intact.