ApsaraMQ for Kafka:Estimate cluster resources for ApsaraMQ for Confluent

Define workload parameters

Calculate broker count

A single Kafka Broker supports up to 100 MB/s of throughput. Plan for 50% I/O bandwidth headroom to handle traffic spikes.

Production clusters require at least 4 brokers:

Broker count = Max(4, peak_inflow x (fan_out + 2 x replication_factor - 1) x 2 / 400)

Test clusters require at least 3 brokers:

Broker count = Max(3, peak_inflow x (fan_out + 2 x replication_factor - 1) x 2 / 300)

Partition limits also constrain broker count:

• Each broker: no more than 2,000 partition replicas

• Entire cluster: no more than 200,000 partition replicas

If your total partition replica count is high, size the broker count based on partition limits rather than throughput alone.

Broker count = Max(4, 200 x (2 + 2 x 3 - 1) x 2 / 400)
             = Max(4, 200 x 7 x 2 / 400)
             = Max(4, 7)
             = 7 brokers

Choose CUs per broker

CU requirements depend on cluster configuration, client behavior, partition count, and the number of producers and consumers. Use these guidelines as a starting point:

Calculate disk size per broker

Disk per broker = Max(1 TB, average_inflow x retention_period x replication_factor / broker_count)

Convert all values to consistent units before calculating. If average_inflow is in MB/s, convert retention_period to seconds (1 day = 86,400 seconds).

Disk per broker = Max(1 TB, 50 MB/s x 604,800 s x 3 / 7)
               = Max(1 TB, ~12,960,000 MB)
               = Max(1 TB, 12.4 TB)
               = 12.4 TB per broker

Connect

SchemaRegistry

ControlCenter

KsqlDB

KafkaRestProxy

Throughput and latency by CU count

With 4 brokers at 8 or more CUs each, the cluster reaches a baseline throughput of 400 MB/s. Additional CUs primarily improve latency-constrained throughput rather than peak throughput.

Scale out with additional brokers

Each additional broker adds approximately 100 MB/s of throughput. Allocate at least 8 CUs per broker when scaling out to prevent compute from becoming a bottleneck.

Production environment

Total cluster throughput: 400 MB/s (excluding replication traffic).

Test environment

Total cluster throughput: 300 MB/s (excluding replication traffic).