Rebalance topic traffic - ApsaraMQ for Kafka - Alibaba Cloud Documentation Center

When you upgrade the traffic specification of an ApsaraMQ for Kafka instance, the corresponding cluster may be scaled out. After the cluster is scaled out, you must rebalance the topic traffic to evenly distribute the traffic across brokers in the scaled-out cluster. Otherwise, the original topic traffic is still distributed across the brokers that are in the cluster before the scale-out. The original topics are subject to the maximum bandwidth purchased before the scale-out. New topics are not subject to the original maximum bandwidth.

Prerequisites

Your ApsaraMQ for Kafka instance is in the Running (Pending Rebalancing) state.

Note

For more information about how to upgrade the traffic specification of a Message Queue for Apache Kafka instance and when a cluster scale-out is triggered, see Upgrade instance specifications.

Precautions

When your ApsaraMQ for Kafka instance is in the Running (Pending Rebalancing) state, you can use this instance to send and consume messages but cannot create resources such as topics and consumer groups in this instance.Group You must wait until the topic traffic is rebalanced or choose not to rebalance the topic traffic before you can create a resource.

Traffic rebalancing methods

Traffic rebalancing method	How it works	Impact	Suitable scenarios	Duration
Add Partitions to All Topics	After the cluster is scaled out, the system adds partitions to the new brokers for all topics on the original brokers.	New messages in partitions are out of order. The number of partitions changes. If your client cannot automatically detect new partitions, you may need to restart the client or modify the client code. This may occur in scenarios in which extract, transfer, and load (ETL) is performed, or in scenarios in which you send messages to or consume messages from specified partitions.	The order of partitions is not required. The partition to which messages are sent is not specified. The selected consumption mode is subscription.	Within seconds.
Migrate Data from Partitions of All Topics (Recommended)	Local storage: The kafka-reassign-partitions tool is used to migrate topic data among partitions. Cloud storage: The mapping is modified, but the topic data is not migrated among partitions. Note You can select Local Storage for Storage Engine only when you create topics for a Professional Edition instance.	Local storage: Temporary internal traffic is generated. Cloud storage: No temporary internal traffic is generated.	All cluster scale-out scenarios.	Local storage: within minutes or hours. The actual duration depends on the amount of data that you want to migrate from the local storage. Migrating a large amount of data may take several hours or longer. We recommend that you migrate data during off-peak hours. Cloud storage: within seconds. It takes about 30 seconds to migrate a topic.
Do Not Rebalance (Not Recommended)	No operation is performed. The original topics are still distributed on the brokers of the cluster before the scale-out, and new topics are evenly distributed on all brokers after the scale-out.	The original topics are subject to the traffic specification purchased before the scale-out. If the original topic traffic is heavy, the traffic may be unbalanced among brokers.	The original topic traffic is light, and is not greatly increased after the cluster is scaled out. Topics are created after the cluster is scaled out. Most of the traffic is directed to the new topics.	The configuration immediately takes effect.

Procedure

Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, click the name of the region where your instance is deployed.
On the Instances page, click the name of the instance that you want to manage.
In the Overview section of the Instance Details page, click Rebalance Topic Traffic in the upper-right corner.
In the Rebalance Topic Traffic for Instance panel, set the Traffic Rebalancing Method parameter. For more information about the supported traffic rebalancing methods, see Traffic rebalancing methods.
After you select a traffic rebalancing method, all topics in the instance enter the Pending Rebalancing state. You can check the topic status in the Status column on the Topics page.

Result

After the topic traffic is rebalanced, the topics enter the Running state, as shown in the Status column on the Topics page.