Topics that are automatically created are applicable to testing scenarios or temporary migration scenarios. If you enable auto Topic creation in the production environment for a long period of time, your resources may be randomly created due to client misuse. In the production environment, we recommend that you try to Message Queue for Apache Kafkaconsole or API to manage topics.

Prerequisites

Before you back up data, the following prerequisites are met:
  1. Ensure Message Queue for Apache Kafkathe major version of the instance is 2.xand the minor version is the latest. How to View and upgrade Message Queue for Apache Kafkafor more information about the major version or minor version of the instance, see Upgrade the instance version.
  2. Submit a ticket for Message Queue for Apache Kafkaturn on automatic Topic creation on the instance.

Background information

When a Topic is automatically created Message Queue for Apache Kafkaafter auto create Topic is enabled for an instance, the client Message Queue for Apache Kafkawhen the instance sends a request to obtain the metadata of a Topic that does not exist, for example, sending a message to a Topic that does not exist, Message Queue for Apache Kafkathe Topic is automatically created by the instance.

Call Producer or Consumer APIs to create a Topic

After you enable Topic creation for an instance, you can call the Producer API or Consumer API to create a Topic automatically.

Notice
  • The name of the automatically created Topic must be Message Queue for Apache Kafkaotherwise, the Topic will not be created and the Producer or Consumer will receive messages similar to no Metadata found error.
  • For the number of automatically created topics, follow these rules: Message Queue for Apache Kafkathe instance type. Otherwise, the Topic will not be created and the Producer or Consumer will receive messages similar to no Metadata found error.
  • The total number of partitions for the automatically created Topic is subject to Message Queue for Apache Kafkathe instance type. Otherwise, the Topic will not be created and the Producer or Consumer will receive messages similar to no Metadata found error.
  • After the create Topic feature is enabled, pay attention to Message Queue for Apache Kafkaconsole, to purchase new resources and delete useless resources.
  • by default, the storage engine of the automatically created Topic is cloud storage, the number of partitions is 12, and the description is auto created by metadata by default. Creating a Topic automatically by using the Producer or Consumer API is essentially sending requests to obtain the Metadata of the Topic. If Message Queue for Apache Kafkaif the server finds that the requested Topic does not exist, it automatically creates this Topic.
Call the Producer or Consumer API to create a Topic.
  • The sample code for calling the Producer API to create a Topic is as follows:
    // The Java API is used as an example. APIs in other programming languages are similar.
    ProducerRecord<String, String> kafkaMessage =  new ProducerRecord<String, String>("newTopicName", value);
    // If the Topic does not exist, the default storage engine is cloud storage, and the default number of partitions is 12.
    Future metadataFuture = producer.send(kafkaMessage);
  • The sample code for calling the Consumer API to create a Topic is as follows:
    // The Java API is used as an example. APIs in other programming languages are similar.
    consumer.subscribe(Collections.singletonList("newTopicName"));
    // If the Topic does not exist, the default storage engine is cloud storage, and the default number of partitions is 12.
    consumer.poll(Duration.ofSeconds(1));
Message Queue for Apache KafkaOf the console Topicsthe page shows the automatically created Topic.

AdminClient.createTopics() to create a Topic.

After you enable auto-create Topic for an instance, you can call the AdminClient.createTopics() method to create a Topic.

Notice
  • The name of the Topic created Message Queue for Apache Kafkaotherwise, the Topic cannot be created.
  • For the number of created topics, follow these rules: Message Queue for Apache Kafkathe specification of the instance. Otherwise, the Topic cannot be created and the system throws the following error: Exception in thread "main" java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidTopicException: The max topic count of your instance is xxx, topic runs out..
  • The total number of partitions for the Topic follows: Message Queue for Apache Kafkathe instance type. If you do not specify a Topic, the Topic cannot be created and the following error message is displayed: Exception in thread "main" java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidTopicException: The max partition count of your instance is xxx, partition runs out..
  • The number of partitions in the single Topic must be less than or equal to 360. Otherwise, the Topic cannot be created and the following error occurs: Exception in thread "main" java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidTopicException: The max partition count of a single topic is 360..
  • You must submit a Topic creation request after the previous Topic creation request is processed. Otherwise, the Topic cannot be created and the following error occurs: org.apache.kafka.common.errors.InvalidTopicException: Try to create topic: topicA. But some other topics are being created, please try again later.
  • assignments are not supported.
  • Other parameters than Topic name, storage engine, cleanup.policy, and number of partitions (such as min.isr) are not supported in creation. Message Queue for Apache KafkaAutomatically configures the optimal parameters for you.
  • The description of the created Topic is auto created by admin-client by default. If you have configured "cleanup.policy":"compact", the storage engine is Local storage. Otherwise, it is cloud storage.
  • With the ACL-enabled Message Queue for Apache Kafkathe instance does not support calling. AdminClient.createTopics() create a Topic.
Call AdminClient.createTopics() method to create a Topic.
The following code provides an example on how to use the self-signed mode:
// The Java API is used as an example. APIs in other programming languages are similar.
CreateTopicsOptions createTopicsOptions = new CreateTopicsOptions();
// More than 20 seconds are recommended.
createTopicsOptions.timeoutMs(20000);
// If you want to verify a Topic only, use the following line of code. The Topic will not be actually created.
// createTopicsOptions.validateOnly(true);
Collection newTopics = new ArrayList<>();

// Create a Topic for the cloud storage engine. The Topic must have 12 partitions and a default value of 1 for the replicationFactor.
NewTopic cloudTopic = new NewTopic("cloudTopic", 12, (short) 1);
newTopics.add(cloudTopic);

// Create a Topic for a Local storage engine with 12 partitions and a default replicationFactor of 3.
NewTopic compactLocalTopic = new NewTopic("compactLocalTopic", 12, (short) 3);
Map<String, String> map = new HashMap<>();

// Set cleanup.policy to compact.
map.put(TopicConfig.CLEANUP_POLICY_CONFIG, TopicConfig.CLEANUP_POLICY_COMPACT);
compactLocalTopic.configs(map);
newTopics.add(compactLocalTopic);

// Submit the request.
CreateTopicsResult result = adminClient.createTopics(newTopics, createTopicsOptions);

// Wait until the creation is complete. After the creation is complete, no error is returned. If the creation fails or times out, an error is returned.
result.all().get();
Message Queue for Apache KafkaOf the console Topicsthe created Topic is displayed in the Topic list.