Limitations of Message Queue for Apache Kafka - ApsaraMQ for Kafka

Message Queue for Apache Kafka has specific limits. When you use Message Queue for Apache Kafka, do not exceed these limits to prevent exceptions in your application.

Important

The Service Level Agreement (SLA) does not cover instability caused by exceeding these limits.

Usage limits

The following table lists the usage limits for Message Queue for Apache Kafka.

Item	Limit	Description
Total number of topics (total partitions)	Limited	In Message Queue for Apache Kafka, storage and coordination occur at the partition level. Too many topics (or total partitions) can lead to storage fragmentation, which degrades cluster performance and stability.
Minimum number of partitions per topic	Subscription and pay-as-you-go instances: For topics that use cloud storage: 2 For topics that use local storage: 1 Serverless instances: For topics that use cloud-native storage: 1	If traffic is high, a single partition can become a hotspot and cause data skew. Configure an appropriate number of partitions.
Decreasing the number of partitions for a topic	Not supported	This is an inherent design limitation of Apache Kafka.
Exposing ZooKeeper	Not supported	Since Apache Kafka 0.9.0, clients no longer need to access ZooKeeper. The ZooKeeper used by Message Queue for Apache Kafka is partially shared and is not exposed for security reasons. Your applications do not need to access ZooKeeper.
Logging on to the machine where Message Queue for Apache Kafka is deployed	Not supported	None.
Supported versions	2.2.x to 3.3.x	Non-serverless instances support versions 2.2.x to 2.6.x. Serverless instances support version 3.3.x. To upgrade the instance version, see Upgrade an instance version.
Partition-to-topic ratio	1:1	The number of available topics is directly related to the total number of partitions. For example, if you purchase an instance of the alikafka.hw.2xlarge specification with 50 purchased partitions and 1,000 free partitions, the total number of partitions is 50 + 1,000 = 1,050. In this case, you can use up to 1,050 topics. Note This applies only to non-serverless instances.
Changing the region of an instance	Not supported	Once an instance is deployed, its region is fixed and cannot be changed. To change the region, release the instance and purchase a new one.
Changing the network properties of an instance	Supported	You can change the network properties as needed. For more information, see Upgrade instance configurations.
Message size	10 MB	Messages cannot exceed 10 MB. Larger messages are rejected.
Monitoring and alerting	Supported	Monitoring data is delayed by 1 minute.
Endpoint	Depends on the purchased edition	Non-serverless instances: Standard Edition: Supports default endpoint and SSL endpoint. Professional Edition: Supports default endpoint, SSL endpoint, and SASL endpoint. Serverless instances: Support default endpoint, SSL endpoint, and SASL endpoint.
Single-partition topics that use cloud storage	May become unavailable during downtime or upgrades.	Create topics with more than one partition. If you must use a single partition, select local storage. Note This limitation applies only to non-serverless instances. On serverless instances, single-partition cloud storage is highly available. Only Professional Edition instances support selecting local storage when you create a topic. This feature is not available for Standard Edition instances.
Maximum number of messages per batch	32,767	If individual messages are very small, set `batch.size` to a value no greater than 16,384. Note This limit applies only to non-serverless instances.

Note

Message Queue for Apache Kafka no longer supports purchasing non-serverless instances based on topic editions. For existing instances purchased this way, the topic-to-partition ratio is 1:16. For Professional Edition instances, the number of topics is double the number of purchased topics.

Quota limits

The following table lists the quota limits for Message Queue for Apache Kafka. Exceeding these limits may cause stability issues. The "Other limits" section lists scenarios that may negatively affect the server. Use these features carefully to avoid overloading the server and impacting stability.

Unless otherwise specified, the limits apply per cluster. To request a quota increase, submit a ticket.

In the table, "//" indicates integer division, which rounds down.

Item	Limit				Description
Item	Non-serverless instances	Serverless (Basic)	Serverless (Standard and Professional)	Hard limit	Description
Connections per node	Starts at 1,000. Increases by 1,000 for every additional 100 MB/s of actual production traffic. Maximum: 10,000. Formula: `C = min(10000, 1000 + (F // 100) * 1000)`	Starts at 2,000. Increases by 1,000 for every additional 300 MB/s of reserved production capacity. Maximum: 10,000. Formula: `C = min(10000, 2000 + (F // 300) * 1000)`		Yes	The number of TCP connections on a single broker. To increase the connection limit, submit a ticket. When the limit is exceeded, new connections are established slowly, and the least recently used (LRU) connections are terminated.
SSL connections per node	Starts at 200. Increases by 100 for every additional 100 MB/s of actual production traffic. Maximum: 1,000. Formula: `C = min(1000, 200 + (F // 100) * 100)`	Starts at 200. Increases by 100 for every additional 300 MB/s of reserved production capacity. Maximum: 1,000. Formula: `C = min(1000, 200 + (F // 300) * 100)`		Yes	The number of SSL TCP connections on a single broker. When the limit is exceeded, new connections are established slowly, and the least recently used (LRU) connections are terminated.
Connection establishment rate per node	50/second	150/second	150/second	Yes	The number of connection attempts per second between a client and a server. This includes failed attempts for reasons such as authentication failure. When the limit is exceeded, new connections are established slowly.
SSL connection establishment rate per node	10/second			Yes	The number of SSL connection attempts per second between a client and a server. This includes failed attempts for reasons such as authentication failure. When the limit is exceeded, new connections are established slowly.
Batch size	A batch size with a 50th percentile (TP50) below 4 KB is considered fragmented.			No	The size of a message batch in a PRODUCE request. Fragmented batches (frequent, small batches) increase server CPU load and affect cluster stability. Use client versions 2.4 or later to improve batching. For more information, see Improve send performance (reduce fragmented requests).
Produce request rate per cluster	Starts at 10,000 requests/second. Increases by 2,000 for every additional 20 MB/s of actual production traffic. Formula: `R = 10000 + (F // 20) * 2000`	Starts at 10,000 requests/second. Increases by 5,000 for every additional 300 MB/s of reserved production capacity. Formula: `R = 10000 + (F // 300) * 5000`	Starts at 10,000 requests/second. Increases by 2,000 for every additional 60 MB/s of reserved production capacity. Formula: `R = 10000 + (F // 60) * 2000`	No	The number of PRODUCE requests sent by clients per second. A high request rate can increase the server's CPU load and affect cluster stability. To increase the request limit, submit a ticket.
Fetch request rate per cluster	Starts at 5,000 requests/second. Increases by 1,000 for every additional 20 MB/s of actual consumption traffic. Formula: `R = 5000 + (F // 20) * 1000`	Starts at 5,000 requests/second. Increases by 2,500 for every additional 100 MB/s of reserved subscription capacity. Formula: `R = 5000 + (F // 100) * 2500`	Starts at 5,000 requests/second. For every 20 MB/s increase in reserved subscription throughput, the number of requests per second increases by 1,000. Formula: `R = 5000 + (F // 20) * 1000`	No	The number of FETCH requests from clients per second. A high request rate can increase the server's CPU load and affect cluster stability. To increase the request limit, submit a ticket.
Offset commit rate per node	Starts at 100 requests/second. Increases by 100 for every additional 100 MB/s of actual production traffic. Maximum: 1,000 requests/second. Formula: `R = min(1000, 100 + (F // 100) * 100)`	Starts at 100 requests/second. Increases by 100 for every additional 100 MB/s of reserved production capacity. Maximum: 1,000 requests/second. Formula: `R = min(1000, 100 + (F // 100) * 100)`		No	The number of OFFSET_COMMIT requests sent by clients per second. A high request rate can increase the server's CPU load and affect cluster stability. To increase the request limit, submit a ticket.
Metadata request rate per cluster	Starts at 100 requests/second. Increases by 100 for every additional 100 MB/s of actual production traffic. Maximum: 1,000 requests/second. Formula: `R = min(1000, 100 + (F // 100) * 100)`	Starts at 100 requests/second. Increases by 100 for every additional 100 MB/s of reserved production capacity. Maximum: 1,000 requests/second. Formula: `R = min(1000, 100 + (F // 100) * 100)`		No	Client metadata requests received by the server, such as `METADATA, INIT_PRODUCER_ID, CREATE_ACL, and JOIN_GROUP`. Warning An excessive request rate can affect cluster stability.
Maximum number of partitions	For the maximum number of partitions for different instance specifications, see Instance specifications and partition limits.			Yes	The total number of partitions across all topics. If the limit is exceeded, you cannot create new topics or add partitions. To increase the partition limit, submit a ticket.
Partition creation/deletion rate per cluster	900 partitions per 10 seconds			No	This includes all partition operations performed via the console, API, or Kafka Admin tools.
Number of consumer groups per cluster	2,000 per cluster The recommended subscription ratio of topics to groups is 1:1, and it should not exceed 3:1.			No	The number of consumer groups used by clients. To increase the group limit, submit a ticket. Warning An excessive number of consumer groups can increase the coordinator load on the server and complicate metadata management, which affects performance and failure recovery time.
Message format version	The message format version for producing and consuming messages must be V1 or later.			Yes	We recommend using client versions 2.4 or later. Warning Using an older Kafka message format can prevent messages from being produced or consumed and can lead to issues such as increased server CPU usage, decreased throughput, and compatibility or security problems.
Invalid timestamp interception	When `message.timestamp.type=CreateTime`, the broker rejects a message if the difference between the broker's timestamp upon receipt and the timestamp in the message exceeds the configured `message.timestamp.difference.max.ms` value. This configuration prevents incorrect timestamp settings. If the timestamp is set too far in the past, the log segment may be deleted immediately. If it is set too far in the future, the log segment may never be deleted.			Yes	The client receives a clear error: INVALID_TIMESTAMP (32, "The timestamp of the message is out of acceptable range.").
Compacted topic size limit	Compacted topics are not enabled by default.			Yes	Compacted topics are disabled by default to prevent abnormal write patterns from exhausting cluster storage and causing a crash. If you require this feature, submit a ticket.
Other limits	Enabling compression algorithms like GZIP consumes more server resources, which can increase latency and reduce throughput. Initializing a large number of transactions at a high frequency to get a `Producer Id` can cause out-of-memory (OOM) errors and server overload, which affects stability. For this reason, we have adjusted the `transactional.id.expiration.ms` kernel parameter to 60 minutes. If you have specific requirements, submit a ticket. If the CPU utilization of an instance exceeds 85%, it can affect cluster stability and lead to issues such as downtime or long-tail latency fluctuations in message production and consumption. Kafka performance depends on the cluster. If your production behavior or partition assignment is skewed, you may not be able to fully use the cluster's capacity. The open source transactional messaging feature has many known and unfixed issues, such as KAFKA-12671. Use this feature with caution. For more information, see KAFKA ISSUES. Message duplication can occur in Kafka in various scenarios, such as during a rebalance. To prevent business impact from duplicate messages, implement idempotent consumption logic.			No	None