When you upgrade the traffic specifications of a Message Queue for Apache Kafka instance, the corresponding cluster may be scaled out. After the cluster is scaled out, you must rebalance the topic traffic to evenly distribute the traffic across brokers in the scaled-out cluster. Otherwise, the original topic traffic is still distributed across the brokers that are in the cluster before the scale-out. The original topics are subject to the maximum bandwidth purchased before the scale-out. New topics are not subject to this maximum bandwidth.

Prerequisites

Your Message Queue for Apache Kafka instance is in the Running (Pending Rebalancing) state.
Note For more information about how to upgrade the traffic specifications of an instance and when a cluster scale-out is triggered, see Upgrade instance specifications.

Usage notes

When your Message Queue for Apache Kafka instance is in the Running (Pending Rebalancing) state, you can use this instance to send and consume messages but cannot create resources such as topics and consumer groups in this instance. You must wait until the topic traffic is rebalanced or choose not to rebalance the topic traffic before you can create a resource.

Traffic rebalancing methods

Traffic rebalancing method Principle Impact Scenario Duration
Add Partitions to All Topics After the cluster is scaled out, the system adds partitions to the new brokers for all topics on the original brokers.
  • New messages in partitions are out of order.
  • The number of partitions changes. If your client cannot automatically detect new partitions, you may need to restart the client or modify the client code. This may occur in scenarios where stream computing is performed or you send messages to or consume messages from specified partitions.
  • The partition order is not required.
  • The partition to which messages are sent is not specified.
  • The selected consumption mode is subscribe.
Within seconds.
Migrate Data from Partitions of All Topics (Recommended)
  • Local storage: The kafka-reassign-partitions tool is used to migrate topic data in partitions.
  • Cloud storage: The mapping is modified, and the topic data in partitions is not migrated.
  • Local storage: Temporary internal traffic is generated.
  • Cloud storage: No temporary internal traffic is generated.
All cluster scale-out scenarios are supported.
  • Local storage: within minutes or hours. The actual duration depends on the volume of the data that you want to migrate from the local storage. If the data volume is large, the migration may require several hours or longer. We recommend that you migrate data during off-peak hours.
  • Cloud storage: within seconds. It takes about 30 seconds to migrate a topic.
Do Not Rebalance (Not Recommended) You do not need to perform operations. The original topics are still distributed on the brokers of the cluster before the scale-out, and new topics are evenly distributed across all brokers after the scale-out.
  • The original topics are subject to the maximum bandwidth purchased before the scale-out.
  • If the original topic traffic is heavy, the traffic between brokers may be unbalanced.
  • The original topic traffic is light, and is not greatly increased after the cluster is scaled out.
  • Topics are created after the cluster is scaled out. Most of the traffic is directed to the new topics.
None. The rebalancing immediately takes effect.

Procedure

  1. Log on to the Message Queue for Apache Kafka console.
  2. In the Resource Distribution section of the Overview page, select the region where your instance resides.
  3. On the Instances page, click the name of the instance that you want to manage.
  4. In the Overview section of the Instance Details page, click Rebalance Topic Traffic in the upper-right corner.
  5. In the Rebalance Topic Traffic for Instance panel, set the Traffic Rebalancing Method parameter. For more information about the supported traffic rebalancing methods, see Traffic rebalancing methods.
    After you select a traffic rebalancing method, all topics in the instance enter the Pending Rebalancing state. You can check the topic status in the Status column on the Topics page.

Result

After the topic traffic is rebalanced, the topics enter the Running state, as shown in the Status column on the Topics page.