Before you use ApsaraMQ for Kafka to send and subscribe to messages, you must create resources in the ApsaraMQ for Kafka console. Otherwise, you cannot pass authentication or use the management and maintenance functions. Resources here refers to topics and consumer groups.

Prerequisites

ApsaraMQ for Kafka has been purchased and deployed based on the network type:

Background information

After the ApsaraMQ for Kafka instance is deployed, you must create topics and consumer groups.
  • A topic is the first-level identifier for classifying messages in ApsaraMQ for Kafka. For example, you can create a topic named Topic_Trade for transaction messages. The first step to using ApsaraMQ for Kafka is to create a topic for your application.
  • A consumer group identifies a type of consumer. The consumers in a group subscribe to and consume a type of message with the same consumption logic. The relationship between consumer groups and topics is N:N. One consumer group can subscribe to multiple topics, and one topic can be subscribed to by multiple consumer groups.

Step 1: Create a topic

Perform the following steps to create a topic:

  1. Log on to the ApsaraMQ for Kafka console.
  2. In the top navigation bar, select a region.
    Important You must create the topic in the region where the application is located. This is the region where the Elastic Compute Service (ECS) instance for the application is deployed. A topic cannot be used across regions. For example, if a topic is created in the China (Beijing) region, the message producer and consumer must run on ECS instances in the China (Beijing) region.
  3. In the left-side navigation pane, click Topics.
  4. On the Topics page, select an instance and click Create Topic.
  5. In the Create Topic dialog box, set topic properties and then click Create.
    • For Standard Edition instances:

      In the Create Topic dialog box, enter the topic name, select an instance, enter the description and tags, select the number of partitions, and then click Create.

    • For Professional Edition instances:

      In the Create Topic dialog box, enter the topic name, select an instance, enter the description and tags, and select the number of partitions. Click Advanced Settings, select Storage Engine and Message Type, and then click Create.

      ParameterDescription
      Storage EngineApsaraMQ for Kafka supports the following storage engines:
      • Cloud Storage: accesses Alibaba Cloud disks at the underlying layer and stores three replicas in distributed mode. This storage engine features low latency, high performance, data persistence, and high reliability.
      • Local Storage: stores three replicas in distributed mode as an Apache Kafka in-sync replica (ISR) set.
        Important Only ApsaraMQ for Kafka Professional Edition instances running Apache Kafka 2.2.0 support local storage. For more information about how to upgrade an instance edition, see Upgrade the versions of instances.

      For more information about storage engines, see Storage engine comparison.

      cleanup.policy

      If you select local storage, you must configure a log cleanup policy. ApsaraMQ for Kafka supports the following log cleanup policies:

      • delete: the default message cleanup policy. If the remaining disk space is sufficient, messages are retained for the maximum retention period. If disk usage exceeds 85%, earlier messages are deleted to ensure service availability.
      • compact: the Apache Kafka log compaction policy. In log compaction, if multiple messages have the same key, only the message with the latest key value is retained. This policy is applicable to scenarios where the system recovers from a system failure or the cache is reloaded after a system restart. For example, when you use Kafka Connect or Confluent Schema Registry, you must store the system status information or configuration information in a log-compacted topic.
        Important Log-compacted topics are generally used only in some ecosystem components, such as Kafka Connect or Confluent Schema Registry. Do not enable log compaction on the topics for sending and subscribing to messages in other components. For more information, see ApsaraMQ for Kafka demos.
      Message TypeApsaraMQ for Kafka supports the following message types:
      • Normal message: Messages with the same key are stored in the same partition in the order they are sent. If an instance in the cluster fails, the messages may be out of order.
      • Partitionally ordered message: Messages with the same key are stored in the same partition in the order they are sent. If an instance in the cluster fails, the messages are still stored in the same partition in the order they are sent. However, some messages in the partition cannot be sent until the partition is restored.
    After the topic is created, it appears in the list on the Topics page.

Step 2: Create a consumer group

Perform the following steps to create a consumer group:

  1. In the left-side navigation pane of the ApsaraMQ for Kafka console, click Consumer Groups.
  2. On the Consumer Groups page, click an instance and then click Create Consumer Group.
  3. In the Create Consumer Group dialog box, enter the consumer group name, select an instance, enter tags, and then click Create.
    After the consumer group is created, it appears in the list on the Consumer Groups page.

What to do next

Use SDKs to send and subscribe to messages based on the network type: