ApsaraMQ for Kafka:Terms

Apache Kafka

A distributed, open-source platform for real-time data stream processing. Publish, subscribe to, store, and process data streams in real time. For more information, see Apache Kafka.

ApsaraMQ for Kafka

A fully managed Apache Kafka service provided by Alibaba Cloud. ApsaraMQ for Kafka handles deployment and maintenance while delivering high scalability, reliability, and throughput at lower cost. For more information, see What is ApsaraMQ for Kafka?

Publish-subscribe model

An asynchronous communication model in which producers and consumers are decoupled. A producer sends messages to a topic without knowing which consumers read them. A consumer reads messages from a topic without knowing which producer sent them. ApsaraMQ for Kafka uses this model. For more information, see Publish-subscribe model of ApsaraMQ Kafka.

Message

The basic unit of data in ApsaraMQ for Kafka. Each message is internally a byte array. Common examples include page views, server logs, and system resource metrics such as CPU and memory usage.

Message retention duration

The maximum duration that messages are retained when disk capacity is sufficient.

• Valid values: 24 to 480 hours. Default value: 72 hours.

• When disk usage reaches 85%, the system deletes the oldest messages first to maintain service availability.

Maximum message size

The maximum size of a single message that producers can send and consumers can receive.

• Standard Edition and Professional Edition instances support messages up to 10 MB.

• Before you change this value, make sure the new value matches the maximum message size configured on both your producers and consumers.

Broker

A server node in an ApsaraMQ for Kafka cluster. Broker quantity and configurations scale automatically based on your instance traffic specifications. No manual broker management is required.

ZooKeeper

A distributed, open-source coordination service. In ApsaraMQ for Kafka, ZooKeeper manages clusters, stores configurations, and handles leader election. ZooKeeper is an internal component and requires no user interaction.

Cluster

A collection of brokers that work together to store and serve messages.

Instance

An independent resource entity in ApsaraMQ for Kafka. Each instance corresponds to one cluster.

VPC-connected instance

An instance that exposes only a virtual private cloud (VPC) endpoint. A VPC-connected instance is accessible only from the VPC in which it is deployed.

Internet- and VPC-connected instance

An instance that exposes both a public endpoint and a VPC endpoint. This instance type is accessible from the VPC in which it is deployed and over the internet.

Endpoint

The address that producers and consumers use to connect to ApsaraMQ for Kafka, in one of the following formats:

• <broker-ip>:<port> -- broker-level IP address and port

• <instance-domain>:<port> -- instance-level domain name and port

For more information, see Comparison among endpoints.

Topic

A message topic that is used to classify messages.

Partition

A subdivision of a topic. Each partition is an ordered, independently readable message queue. Partitions serve two purposes:

• Parallelism: Multiple consumers can read from different partitions at the same time.

• Scalability: Data storage and I/O are distributed across brokers in the cluster.

The number of partitions determines the maximum throughput and consumer concurrency for a topic.

Topic traffic rebalance

An operation that redistributes topic traffic evenly after an ApsaraMQ for Kafka cluster scales out. Run a rebalance to make sure new brokers share the load. For more information, see Rebalance topic traffic.

Replicas

Copies of each partition that provide data redundancy and high availability. Each partition has a configurable number of replicas, known as the replication factor.

• Default replication factor: 3.

• Higher replication factors increase fault tolerance but also increase storage and network overhead.

Partition replicas

The total replica count across all partitions in a topic, calculated as:

Partition replicas = Number of partitions x Replication factor

For example, a topic with 10 partitions and a replication factor of 3 has 30 partition replicas. A higher partition replica count improves fault tolerance and read performance but increases network and storage overhead.

Cloud storage

A storage engine built on Alibaba Cloud disks. Compared to local storage, cloud storage delivers better auto scaling, reliability, availability, and cost-effectiveness. Use cloud storage for most workloads. For more information, see Comparison between storage engines.

Local storage

A storage engine that uses the In-Sync Replicas (ISR) algorithm from native Apache Kafka. Use local storage when your workload requires compaction, idempotence, transactions, or partitionally ordered messages. For more information, see Comparison between storage engines.

Cleanup policy

The log cleanup policy for a topic. Cleanup policies apply only to topics that use local storage on Professional Edition instances. ApsaraMQ for Kafka supports two policies:

• Delete (default): Retains messages up to the configured retention period. When disk usage exceeds 85%, the system deletes the oldest messages first to maintain availability.

• Compact: Retains only the latest value for each message key, following Apache Kafka log compaction. Use this policy for state-recovery scenarios, such as restoring a system or reloading a cache after a restart. Kafka Connect and Confluent Schema Registry require log-compacted topics.

Producer

An application that sends messages to ApsaraMQ for Kafka.

Consumer

An application that reads messages from ApsaraMQ for Kafka.

Group

A set of consumers that share the same group ID. Within a consumer group, each message from a subscribed topic is delivered to exactly one consumer. This enables:

• Load balancing: Message processing is distributed across group members.

• Parallel consumption: Multiple consumers process messages from a topic concurrently.

Subscription

The relationship between a consumer group and the topics it consumes. Only actively connected consumer groups can be queried. Disconnected groups are not visible.

Offset

A sequence number assigned to each message as it arrives at a partition. Offsets identify a message's position within the partition log and are used to track consumption progress.

Earliest offset

The offset of the first available message in a partition. For more information, see View partition status.

Maximum offset

The offset of the most recent message in a partition. For more information, see View partition status.

Consumer offset

The offset of the latest message that a consumer group has consumed in a partition. This value tracks the group's consumption progress. For more information, see View consumer information.

Accumulated messages

The number of unconsumed messages in a partition, calculated as:

Accumulated messages = Maximum offset - Consumer offset

A growing count indicates consumers are falling behind -- either blocked or consuming slower than producers are sending. To address accumulation, analyze consumer health and scale up consumption. You can clear all accumulated messages to consume from the latest offset, or reset the consumer offset to a specific point in time. For more information, see Reset consumer offsets.

Latest consumption time

The time when the most recently consumed message was originally published to the broker. When no messages are accumulated, this value is close to the current send time.

Normal message

The default message type. Messages with the same key are stored in the same partition in send order. However, if a broker restarts or fails, a small number of messages may be delivered out of order. For more information, see Comparison between storage engines.

Partitionally ordered message

A message type that guarantees strict ordering within a partition. Messages with the same key are stored in the same partition in send order, and this ordering holds even during broker failures. The trade-off: affected partitions stop accepting writes until the failed brokers recover. For more information, see Comparison between storage engines.

Major version upgrade

An upgrade that changes the major version number of an ApsaraMQ for Kafka instance. For example, upgrading from version 0.10.x to version 2.x. For more information, see Upgrade instance versions.

Minor version update

An update that keeps the major version number unchanged. For example, updating from version 0.10 to version 0.10.2, or from version 0.10.2 to the 0.10.2 kernel-optimized version. For more information, see Upgrade instance versions.

RAM

Resource Access Management (RAM) is an Alibaba Cloud service for managing user identities and resource access permissions. Use RAM to control who can access ApsaraMQ for Kafka resources through the console or API operations. For more information, see Overview.

ACL

Access control lists (ACLs) manage permissions for Simple Authentication and Security Layer (SASL) users and clients to send and receive messages through SDKs. ACLs apply specifically to scenarios in which you want to implement access control for users who use SDKs to send and receive messages. This feature is consistent with open-source Apache Kafka ACLs. For more information, see Overview.

Connector

A component that synchronizes data between ApsaraMQ for Kafka and other Alibaba Cloud services. For more information, see Overview.

Tag

A key-value pair attached to an ApsaraMQ for Kafka resource. Use tags to classify, search, and aggregate resources by custom criteria. For more information, see Manage tags.