Can topics be automatically created?
Yes. ApsaraMQ for Kafka supports automatic topic creation. However, automatic topic creation is disabled by default. In most cases, create topics manually through the ApsaraMQ for Kafka console or by calling API operations.
Reasons to keep automatic topic creation disabled in production:
Security risks: Fine-grained permission control and resource action audit are unavailable when automatic topic creation is enabled.
Resource management issues: Automatic topic creation can lead to excess resource usage, inefficient resource management, and system instability.
Creation failures: Topics may fail to be automatically created due to naming violations, quota limits, or transient infrastructure issues. For more information, see Troubleshoot automatic topic creation failures.
When to enable automatic topic creation
Automatic topic creation involves non-standard operations. Limit it to non-production scenarios:
Cloud migration: Temporarily enable the feature while migrating workloads from a self-managed Kafka cluster.
Testing: Enable the feature in development or testing environments for convenience.
After you enable the feature, use it only in the migration and testing scenarios. Disable the feature after migration or testing is complete.
How to enable automatic topic creation
Do not enable this feature for production workloads. Use it only for migration and testing.
For step-by-step instructions, see Automatically create topics.
Troubleshoot automatic topic creation failures
If a moderate number of topics failed to be created, create them manually in the ApsaraMQ for Kafka console or by calling the corresponding API operation.
If many topics failed, check the logs and investigate the following causes:
Invalid topic names
Topic names must follow the naming conventions. A topic fails to be created if its name contains invalid characters or exceeds the length limit.
Incorrect configuration or insufficient quota
Topics fail to be created if an improper engine is specified, or if the remaining topic or partition quota is insufficient.
Infrastructure issues
Transient events such as network jitters, server upgrades, console upgrades and maintenance, or Elastic Compute Service (ECS) instance failures can delay topic creation beyond the timeout threshold, causing failures.
Name conflicts
If the system tries to create a topic with the same name as an existing topic, the creation fails. In some cases, data from both topics may be recorded and the system may not detect the conflict. This type of conflict is rarely recoverable and can lead to irreversible losses.
First-access latency
Accessing an automatically created topic for the first time takes significantly longer than accessing an existing topic. This delay can cause temporary business interruptions.
Quota limits reached
No more topics can be created after the instance reaches its maximum number of topics or partitions.
Other reasons
Topics may also fail to be created for other reasons not listed above.
If the issue persists after investigation, contact ApsaraMQ for Kafka technical support.