When you upgrade the traffic specification of an ApsaraMQ for Kafka instance, the cluster may scale out by adding brokers. After a scale-out, existing topic partitions remain on the original brokers and are subject to pre-upgrade traffic limits. Rebalance topic traffic to redistribute partitions across all brokers, including the new ones, so that existing topics can use the additional capacity.
Topics created after the scale-out are automatically distributed across all brokers and are not subject to pre-upgrade traffic limits.
On serverless ApsaraMQ for Kafka instances, topic rebalancing is triggered automatically. No manual action is required.
Prerequisites
Before you begin, make sure that you have:
An ApsaraMQ for Kafka instance in the Running (Pending Rebalancing) state
An instance enters the Running (Pending Rebalancing) state after a cluster scale-out. For more information about upgrading traffic specifications and scale-out trigger conditions, see Upgrade instance configurations.
Rebalance topic traffic in the console
Log on to the ApsaraMQ for Kafka console.
In the Resource Distribution section of the Overview page, select the region where your instance resides.
On the Instances page, click the name of the instance that you want to manage.
On the Instance Details page, click Rebalance Topic Traffic in the upper-right corner of the Overview section.
In the Rebalance Topic Traffic for Instance panel, select a Traffic Rebalancing Method. For details about each method, see Rebalancing methods.
After you select a method, all topics on the instance enter the Pending Rebalancing state. You can check the topic status in the Status column on the Topics page.
Result
After topic traffic is rebalanced, all topics return to the Running state. Verify the topic status in the Status column on the Topics page.
Rebalancing methods
Select a rebalancing method based on your workload requirements. The following table summarizes the available methods:
| Method | Duration | Partition count change | Message ordering impact |
|---|---|---|---|
| Migrate Data from Partitions of All Topics (Recommended) (recommended) | Seconds (cloud storage) to hours (local storage) | No | No |
| Add Partitions to All Topics | Seconds | Yes | Yes |
| Do Not Rebalance (Not Recommended) | Immediate | No | No |
Migrate data from partitions of all topics (recommended)
Redistributes existing partitions across all brokers without changing the partition count. This is the recommended method for all scale-out scenarios.
The behavior depends on the storage engine:
| Storage engine | Mechanism | Impact | Duration |
|---|---|---|---|
| Cloud storage | Modifies the partition-to-broker mapping without migrating data. | No temporary internal traffic is generated. | Seconds. Each topic takes approximately 30 seconds. |
| Local storage | Uses the kafka-reassign-partitions tool to migrate partition data to new brokers. | Temporary internal traffic is generated during migration. | Minutes to hours, depending on data volume. For large datasets, migration may take several hours or longer. |
You can specify local storage as the storage engine only when you create topics for Professional Edition instances. For local storage migrations, run the rebalance during off-peak hours to minimize the impact of temporary internal traffic on your workloads.
Add partitions to all topics
Adds new partitions on the new brokers for all existing topics. This is the fastest method but changes the partition count.
Duration: Seconds.
Impact:
New messages in the added partitions may arrive out of order.
The total partition count increases. If your client does not automatically detect new partitions, restart the client or update the client code. This applies to extract, transform, and load (ETL) scenarios and scenarios where messages are sent to or consumed from specific partitions.
Best for: Workloads where partition ordering is not required, the target partition is not specified for message production, and the consumption mode is subscription.
Do Not Rebalance
No action is taken. Existing topics remain on the pre-scale-out brokers. Topics created after the scale-out are evenly distributed across all brokers.
Duration: Takes effect immediately.
Impact:
Existing topics remain subject to the traffic specification purchased before the scale-out.
If the existing topic traffic volume is large, broker traffic may become unbalanced.
Best for: Scenarios where existing topic traffic is small and unlikely to increase, or where most traffic is directed to new topics created after the scale-out.
Skipping rebalancing means the original topics cannot use the upgraded capacity.
Usage notes
While an instance is in the Running (Pending Rebalancing) state:
You can continue to send and receive messages.
You cannot create resources such as topics and consumer groups until the rebalancing completes.
Rebalancing is optional. If you do not need the original topics to use the upgraded capacity, select Do Not Rebalance. For details, see Do Not Rebalance.