When you migrate a self-managed Kafka deployment to ApsaraMQ for Kafka or run integration tests, recreating hundreds of topics manually is impractical. Automatic topic creation lets ApsaraMQ for Kafka create a topic on the fly when a producer sends a message to a topic that does not exist, so you can focus on validating your migration or tests rather than provisioning topics one by one.
Two approaches are available for topic creation:
| Approach | When to use | Trade-offs |
|---|---|---|
| Manual creation (recommended for production) | Production workloads | Full control over partition count, storage engine, and naming. Supports fine-grained access control and resource auditing. |
| Automatic creation | Cloud migration, testing | Faster setup, but auto-created topics use default configurations. No fine-grained access control or auditing. |
To create topics manually, use the ApsaraMQ for Kafka console or the CreateTopic API.
Supported instances
Your ApsaraMQ for Kafka instance must meet both requirements:
| Item | Requirement |
|---|---|
| Status | Running |
| Version | Major version V2.2.0 or later, minor version latest |
To check these values, open the Instance Details page in the ApsaraMQ for Kafka console and look in the Basic Information section.
After you enable the access control list (ACL) feature on a Professional Edition instance, sending a message without specifying an existing topic does not trigger automatic topic creation. Create the topic explicitly instead.
Enable automatic topic creation
Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.
On the Instances page, click the name of the instance that you want to manage.
On the Instances page, click the name of your instance.
In the Configuration Information section of the Instance Details page, turn on Automatic Topic Creation.
After you turn on the switch, sending a message to a nonexistent topic on a Standard Edition instance automatically creates that topic.
Topic naming rules
Auto-created topic names must meet all of the following requirements:
| Rule | Details |
|---|---|
| Length | 3 to 64 characters |
| Allowed characters | Letters, digits, hyphens (-), and underscores (_) |
| Content | At least one letter or digit |
A message sent to a topic name that violates these rules does not trigger topic creation.
Disable automatic topic creation after migration or testing
Once your migration or test cycle is complete, turn off Automatic Topic Creation in the Configuration Information section of the Instance Details page. Managing topics through the console or API prevents the risks described in Why not use this in production?.
Production risks
Why not use this in production?
We recommend that you do not enable the automatic topic creation feature in production environments. If you do enable the feature in production environments, do not use it in major production activities. Automatic topic creation introduces the following risks:
| Risk | Description |
|---|---|
| No fine-grained access control | Any producer can create topics implicitly, bypassing permission controls and resource auditing. |
| Resource sprawl | Typos or misconfigured clients silently create unintended topics, consuming partition and topic quotas. |
| Silent failures | Topics can still fail to be created even with auto-creation enabled. In production, these failures can disrupt your application without warning. See Troubleshoot auto-creation failures. |
When to use automatic topic creation
Use this feature only during cloud migration or testing. For example:
Cloud migration. Migrating a self-managed or cross-cloud Kafka deployment to ApsaraMQ for Kafka, where recreating hundreds of topics manually is impractical. See Migrate a self-managed or cross-cloud instance to an ApsaraMQ for Kafka instance.
Integration testing. Running tests where the test framework creates topics dynamically.
Troubleshoot auto-creation failures
If a moderate number of topics failed, create them manually in the ApsaraMQ for Kafka console or by calling the CreateTopic API.
If a large number of topics failed, check your application logs and review the following common causes:
| Cause | Details |
|---|---|
| Invalid topic name | The topic name does not follow the naming rules. For example, it is shorter than 3 characters, longer than 64, or contains unsupported characters. |
| Insufficient quotas | The instance has reached its partition or topic limit. Delete existing topics or increase the quota before creating more. |
| Wrong storage engine | The topic specifies a storage engine that is not compatible with the instance configuration. |
| ACL enabled | On a Professional Edition instance with ACL enabled, auto-creation is blocked. Create the topic explicitly instead. |
| Infrastructure issues | Network jitters, broker upgrades, console upgrades or maintenance, or Elastic Compute Service (ECS) instance failures can cause creation requests to time out. |
| Name conflicts | If the system tries to create a topic with a name that already exists, the data for both topics can become interleaved. This conflict is rarely detected automatically and can lead to irreversible losses. |
| Creation latency | Auto-creating a topic takes significantly longer than sending to an existing topic. The delay can cause temporary timeouts in your application. |
Related information
Migrate a self-managed or cross-cloud instance to an ApsaraMQ for Kafka instance
Step 3: Create resources (manual topic creation in the console)